Annotation of src/usr.bin/lex/flex.1, Revision 1.24
1.24 ! sobrado 1: .\" $OpenBSD: flex.1,v 1.23 2009/02/10 19:27:03 jmc Exp $
1.16 jmc 2: .\"
1.12 jmc 3: .\" Copyright (c) 1990 The Regents of the University of California.
4: .\" All rights reserved.
1.2 deraadt 5: .\"
1.12 jmc 6: .\" This code is derived from software contributed to Berkeley by
7: .\" Vern Paxson.
8: .\"
9: .\" The United States Government has rights in this work pursuant
10: .\" to contract no. DE-AC03-76SF00098 between the United States
11: .\" Department of Energy and the University of California.
12: .\"
13: .\" Redistribution and use in source and binary forms, with or without
1.13 millert 14: .\" modification, are permitted provided that the following conditions
15: .\" are met:
16: .\"
17: .\" 1. Redistributions of source code must retain the above copyright
18: .\" notice, this list of conditions and the following disclaimer.
19: .\" 2. Redistributions in binary form must reproduce the above copyright
20: .\" notice, this list of conditions and the following disclaimer in the
21: .\" documentation and/or other materials provided with the distribution.
22: .\"
23: .\" Neither the name of the University nor the names of its contributors
24: .\" may be used to endorse or promote products derived from this software
25: .\" without specific prior written permission.
26: .\"
27: .\" THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR
28: .\" IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED
29: .\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
30: .\" PURPOSE.
1.16 jmc 31: .\"
1.24 ! sobrado 32: .Dd $Mdocdate: February 10 2009 $
1.16 jmc 33: .Dt FLEX 1
34: .Os
35: .Sh NAME
36: .Nm flex
37: .Nd fast lexical analyzer generator
38: .Sh SYNOPSIS
39: .Nm
40: .Op Fl 78BbcdFfhIiLlnpsTtVvw+?
41: .Op Fl C Ns Op Cm aeFfmr
42: .Op Fl Fl help
43: .Op Fl Fl version
44: .Sm off
45: .Op Fl o Ar output
46: .Op Fl P Ar prefix
47: .Op Fl S Ar skeleton
48: .Op Ar filename ...
49: .Sm on
1.21 jmc 50: .Sh DESCRIPTION
51: .Nm
52: is a tool for generating
53: .Em scanners :
54: programs which recognize lexical patterns in text.
55: .Nm
56: reads the given input files, or its standard input if no file names are given,
57: for a description of a scanner to generate.
58: The description is in the form of pairs of regular expressions and C code,
59: called
60: .Em rules .
61: .Nm
62: generates as output a C source file,
63: .Pa lex.yy.c ,
64: which defines a routine
65: .Fn yylex .
66: This file is compiled and linked with the
67: .Fl lfl
68: library to produce an executable.
69: When the executable is run, it analyzes its input for occurrences
70: of the regular expressions.
71: Whenever it finds one, it executes the corresponding C code.
72: .Pp
1.16 jmc 73: The manual includes both tutorial and reference sections:
74: .Bl -ohang
75: .It Sy Some Simple Examples
76: .It Sy Format of the Input File
77: .It Sy Patterns
78: The extended regular expressions used by
79: .Nm .
80: .It Sy How the Input is Matched
81: The rules for determining what has been matched.
82: .It Sy Actions
83: How to specify what to do when a pattern is matched.
84: .It Sy The Generated Scanner
85: Details regarding the scanner that
86: .Nm
87: produces;
88: how to control the input source.
89: .It Sy Start Conditions
90: Introducing context into scanners, and managing
91: .Qq mini-scanners .
92: .It Sy Multiple Input Buffers
93: How to manipulate multiple input sources;
94: how to scan from strings instead of files.
95: .It Sy End-of-File Rules
96: Special rules for matching the end of the input.
97: .It Sy Miscellaneous Macros
98: A summary of macros available to the actions.
99: .It Sy Values Available to the User
100: A summary of values available to the actions.
101: .It Sy Interfacing with Yacc
102: Connecting flex scanners together with
103: .Xr yacc 1
104: parsers.
105: .It Sy Options
106: .Nm
107: command-line options, and the
108: .Dq %option
109: directive.
110: .It Sy Performance Considerations
111: How to make scanners go as fast as possible.
112: .It Sy Generating C++ Scanners
113: The
114: .Pq experimental
115: facility for generating C++ scanner classes.
116: .It Sy Incompatibilities with Lex and POSIX
117: How
118: .Nm
119: differs from AT&T lex and the
120: .Tn POSIX
121: lex standard.
122: .It Sy Files
123: Files used by
124: .Nm .
125: .It Sy Diagnostics
126: Those error messages produced by
127: .Nm
128: .Pq or scanners it generates
129: whose meanings might not be apparent.
130: .It Sy See Also
131: Other documentation, related tools.
132: .It Sy Authors
133: Includes contact information.
134: .It Sy Bugs
135: Known problems with
136: .Nm .
137: .El
138: .Sh SOME SIMPLE EXAMPLES
1.1 deraadt 139: First some simple examples to get the flavor of how one uses
1.16 jmc 140: .Nm .
1.1 deraadt 141: The following
1.16 jmc 142: .Nm
1.1 deraadt 143: input specifies a scanner which whenever it encounters the string
1.16 jmc 144: .Qq username
145: will replace it with the user's login name:
146: .Bd -literal -offset indent
147: %%
148: username printf("%s", getlogin());
149: .Ed
150: .Pp
1.1 deraadt 151: By default, any text not matched by a
1.16 jmc 152: .Nm
153: scanner is copied to the output, so the net effect of this scanner is
154: to copy its input file to its output with each occurrence of
155: .Qq username
156: expanded.
157: In this input, there is just one rule.
158: .Qq username
159: is the
160: .Em pattern
161: and the
162: .Qq printf
163: is the
164: .Em action .
165: The
166: .Qq %%
167: marks the beginning of the rules.
168: .Pp
1.1 deraadt 169: Here's another simple example:
1.16 jmc 170: .Bd -literal -offset indent
1.20 pvalchev 171: %{
1.16 jmc 172: int num_lines = 0, num_chars = 0;
1.20 pvalchev 173: %}
1.1 deraadt 174:
1.16 jmc 175: %%
176: \en ++num_lines; ++num_chars;
177: \&. ++num_chars;
178:
179: %%
180: main()
181: {
182: yylex();
183: printf("# of lines = %d, # of chars = %d\en",
184: num_lines, num_chars);
185: }
186: .Ed
187: .Pp
1.1 deraadt 188: This scanner counts the number of characters and the number
1.16 jmc 189: of lines in its input
190: (it produces no output other than the final report on the counts).
191: The first line declares two globals,
192: .Qq num_lines
193: and
194: .Qq num_chars ,
195: which are accessible both inside
196: .Fn yylex
1.1 deraadt 197: and in the
1.16 jmc 198: .Fn main
199: routine declared after the second
200: .Qq %% .
201: There are two rules, one which matches a newline
202: .Pq \&"\en\&"
203: and increments both the line count and the character count,
204: and one which matches any character other than a newline
205: (indicated by the
206: .Qq \&.
207: regular expression).
208: .Pp
1.1 deraadt 209: A somewhat more complicated example:
1.16 jmc 210: .Bd -literal -offset indent
211: /* scanner for a toy Pascal-like language */
1.1 deraadt 212:
1.16 jmc 213: %{
214: /* need this for the call to atof() below */
215: #include <math.h>
216: %}
1.1 deraadt 217:
1.16 jmc 218: DIGIT [0-9]
219: ID [a-z][a-z0-9]*
1.1 deraadt 220:
1.16 jmc 221: %%
1.1 deraadt 222:
1.16 jmc 223: {DIGIT}+ {
224: printf("An integer: %s (%d)\en", yytext,
225: atoi(yytext));
226: }
1.1 deraadt 227:
1.16 jmc 228: {DIGIT}+"."{DIGIT}* {
229: printf("A float: %s (%g)\en", yytext,
230: atof(yytext));
231: }
1.1 deraadt 232:
1.16 jmc 233: if|then|begin|end|procedure|function {
234: printf("A keyword: %s\en", yytext);
235: }
1.1 deraadt 236:
1.16 jmc 237: {ID} printf("An identifier: %s\en", yytext);
1.1 deraadt 238:
1.16 jmc 239: "+"|"-"|"*"|"/" printf("An operator: %s\en", yytext);
1.1 deraadt 240:
1.16 jmc 241: "{"[^}\en]*"}" /* eat up one-line comments */
1.1 deraadt 242:
1.16 jmc 243: [ \et\en]+ /* eat up whitespace */
1.1 deraadt 244:
1.16 jmc 245: \&. printf("Unrecognized character: %s\en", yytext);
1.1 deraadt 246:
1.16 jmc 247: %%
1.1 deraadt 248:
1.16 jmc 249: main(int argc, char *argv[])
250: {
251: ++argv; --argc; /* skip over program name */
252: if (argc > 0)
253: yyin = fopen(argv[0], "r");
1.1 deraadt 254: else
255: yyin = stdin;
1.7 aaron 256:
1.1 deraadt 257: yylex();
1.16 jmc 258: }
259: .Ed
260: .Pp
261: This is the beginnings of a simple scanner for a language like Pascal.
262: It identifies different types of
263: .Em tokens
1.1 deraadt 264: and reports on what it has seen.
1.16 jmc 265: .Pp
266: The details of this example will be explained in the following sections.
267: .Sh FORMAT OF THE INPUT FILE
1.1 deraadt 268: The
1.16 jmc 269: .Nm
1.1 deraadt 270: input file consists of three sections, separated by a line with just
1.16 jmc 271: .Qq %%
1.1 deraadt 272: in it:
1.16 jmc 273: .Bd -unfilled -offset indent
274: definitions
275: %%
276: rules
277: %%
278: user code
279: .Ed
280: .Pp
1.1 deraadt 281: The
1.16 jmc 282: .Em definitions
1.1 deraadt 283: section contains declarations of simple
1.16 jmc 284: .Em name
1.1 deraadt 285: definitions to simplify the scanner specification, and declarations of
1.16 jmc 286: .Em start conditions ,
1.1 deraadt 287: which are explained in a later section.
1.16 jmc 288: .Pp
1.1 deraadt 289: Name definitions have the form:
1.16 jmc 290: .Pp
291: .D1 name definition
292: .Pp
293: The
294: .Qq name
295: is a word beginning with a letter or an underscore
296: .Pq Sq _
297: followed by zero or more letters, digits,
298: .Sq _ ,
299: or
300: .Sq -
301: .Pq dash .
1.8 aaron 302: The definition is taken to begin at the first non-whitespace character
1.1 deraadt 303: following the name and continuing to the end of the line.
1.16 jmc 304: The definition can subsequently be referred to using
305: .Qq {name} ,
306: which will expand to
307: .Qq (definition) .
308: For example:
309: .Bd -literal -offset indent
310: DIGIT [0-9]
311: ID [a-z][a-z0-9]*
312: .Ed
313: .Pp
314: This defines
315: .Qq DIGIT
316: to be a regular expression which matches a single digit, and
317: .Qq ID
318: to be a regular expression which matches a letter
1.1 deraadt 319: followed by zero-or-more letters-or-digits.
320: A subsequent reference to
1.16 jmc 321: .Pp
322: .Dl {DIGIT}+"."{DIGIT}*
323: .Pp
1.1 deraadt 324: is identical to
1.16 jmc 325: .Pp
326: .Dl ([0-9])+"."([0-9])*
327: .Pp
328: and matches one-or-more digits followed by a
329: .Sq .\&
330: followed by zero-or-more digits.
331: .Pp
1.1 deraadt 332: The
1.16 jmc 333: .Em rules
1.1 deraadt 334: section of the
1.16 jmc 335: .Nm
1.1 deraadt 336: input contains a series of rules of the form:
1.16 jmc 337: .Pp
338: .D1 pattern action
339: .Pp
340: The pattern must be unindented and the action must begin
1.1 deraadt 341: on the same line.
1.16 jmc 342: .Pp
1.1 deraadt 343: See below for a further description of patterns and actions.
1.16 jmc 344: .Pp
1.1 deraadt 345: Finally, the user code section is simply copied to
1.16 jmc 346: .Pa lex.yy.c
1.1 deraadt 347: verbatim.
1.16 jmc 348: It is used for companion routines which call or are called by the scanner.
349: The presence of this section is optional;
1.1 deraadt 350: if it is missing, the second
1.16 jmc 351: .Qq %%
352: in the input file may be skipped too.
353: .Pp
354: In the definitions and rules sections, any indented text or text enclosed in
355: .Sq %{
1.1 deraadt 356: and
1.16 jmc 357: .Sq %}
358: is copied verbatim to the output
359: .Pq with the %{}'s removed .
1.1 deraadt 360: The %{}'s must appear unindented on lines by themselves.
1.16 jmc 361: .Pp
1.1 deraadt 362: In the rules section,
1.16 jmc 363: any indented or %{} text appearing before the first rule may be used to
364: declare variables which are local to the scanning routine and
365: .Pq after the declarations
1.1 deraadt 366: code which is to be executed whenever the scanning routine is entered.
367: Other indented or %{} text in the rule section is still copied to the output,
368: but its meaning is not well-defined and it may well cause compile-time
369: errors (this feature is present for
1.16 jmc 370: .Tn POSIX
1.1 deraadt 371: compliance; see below for other such features).
1.16 jmc 372: .Pp
373: In the definitions section
374: .Pq but not in the rules section ,
375: an unindented comment
376: (i.e., a line beginning with
377: .Qq /* )
378: is also copied verbatim to the output up to the next
379: .Qq */ .
380: .Sh PATTERNS
1.1 deraadt 381: The patterns in the input are written using an extended set of regular
1.16 jmc 382: expressions.
383: These are:
384: .Bl -tag -width "XXXXXXXX"
385: .It x
386: Match the character
387: .Sq x .
388: .It .\&
389: Any character
390: .Pq byte
391: except newline.
392: .It [xyz]
393: A
394: .Qq character class ;
395: in this case, the pattern matches either an
396: .Sq x ,
397: a
398: .Sq y ,
399: or a
400: .Sq z .
401: .It [abj-oZ]
402: A
403: .Qq character class
404: with a range in it; matches an
405: .Sq a ,
406: a
407: .Sq b ,
408: any letter from
409: .Sq j
410: through
411: .Sq o ,
412: or a
413: .Sq Z .
414: .It [^A-Z]
415: A
416: .Qq negated character class ,
417: i.e., any character but those in the class.
418: In this case, any character EXCEPT an uppercase letter.
419: .It [^A-Z\en]
420: Any character EXCEPT an uppercase letter or a newline.
421: .It r*
422: Zero or more r's, where
423: .Sq r
424: is any regular expression.
425: .It r+
426: One or more r's.
427: .It r?
428: Zero or one r's (that is,
429: .Qq an optional r ) .
430: .It r{2,5}
431: Anywhere from two to five r's.
432: .It r{2,}
433: Two or more r's.
434: .It r{4}
435: Exactly 4 r's.
436: .It {name}
437: The expansion of the
438: .Qq name
439: definition
440: .Pq see above .
441: .It \&"[xyz]\e\&"foo\&"
442: The literal string: [xyz]"foo.
443: .It \eX
444: If
445: .Sq X
446: is an
447: .Sq a ,
448: .Sq b ,
449: .Sq f ,
450: .Sq n ,
451: .Sq r ,
452: .Sq t ,
453: or
454: .Sq v ,
455: then the ANSI-C interpretation of
456: .Sq \eX .
457: Otherwise, a literal
458: .Sq X
459: (used to escape operators such as
460: .Sq * ) .
461: .It \e0
462: A NUL character
463: .Pq ASCII code 0 .
464: .It \e123
465: The character with octal value 123.
466: .It \ex2a
467: The character with hexadecimal value 2a.
468: .It (r)
469: Match an
470: .Sq r ;
471: parentheses are used to override precedence
472: .Pq see below .
473: .It rs
474: The regular expression
475: .Sq r
476: followed by the regular expression
477: .Sq s ;
478: called
479: .Qq concatenation .
480: .It r|s
481: Either an
482: .Sq r
483: or an
484: .Sq s .
485: .It r/s
486: An
487: .Sq r ,
488: but only if it is followed by an
489: .Sq s .
490: The text matched by
491: .Sq s
492: is included when determining whether this rule is the
493: .Qq longest match ,
494: but is then returned to the input before the action is executed.
495: So the action only sees the text matched by
496: .Sq r .
497: This type of pattern is called
498: .Qq trailing context .
499: (There are some combinations of r/s that
500: .Nm
501: cannot match correctly; see notes in the
502: .Sx BUGS
503: section below regarding
504: .Qq dangerous trailing context . )
505: .It ^r
506: An
507: .Sq r ,
508: but only at the beginning of a line
509: (i.e., just starting to scan, or right after a newline has been scanned).
510: .It r$
511: An
512: .Sq r ,
513: but only at the end of a line
514: .Pq i.e., just before a newline .
515: Equivalent to
516: .Qq r/\en .
517: .Pp
518: Note that
519: .Nm flex Ns 's
520: notion of
521: .Qq newline
522: is exactly whatever the C compiler used to compile
523: .Nm
524: interprets
525: .Sq \en
526: as.
527: .\" In particular, on some DOS systems you must either filter out \er's in the
528: .\" input yourself, or explicitly use r/\er\en for
529: .\" .Qq r$ .
530: .It <s>r
531: An
532: .Sq r ,
533: but only in start condition
534: .Sq s
535: .Pq see below for discussion of start conditions .
536: .It <s1,s2,s3>r
537: The same, but in any of start conditions s1, s2, or s3.
538: .It <*>r
539: An
540: .Sq r
541: in any start condition, even an exclusive one.
542: .It <<EOF>>
543: An end-of-file.
544: .It <s1,s2><<EOF>>
545: An end-of-file when in start condition s1 or s2.
546: .El
547: .Pp
1.1 deraadt 548: Note that inside of a character class, all regular expression operators
1.16 jmc 549: lose their special meaning except escape
550: .Pq Sq \e
551: and the character class operators,
552: .Sq - ,
553: .Sq ]\& ,
554: and, at the beginning of the class,
555: .Sq ^ .
556: .Pp
1.1 deraadt 557: The regular expressions listed above are grouped according to
558: precedence, from highest precedence at the top to lowest at the bottom.
1.16 jmc 559: Those grouped together have equal precedence.
560: For example,
561: .Pp
562: .D1 foo|bar*
563: .Pp
1.1 deraadt 564: is the same as
1.16 jmc 565: .Pp
566: .D1 (foo)|(ba(r*))
567: .Pp
568: since the
569: .Sq *
570: operator has higher precedence than concatenation,
571: and concatenation higher than alternation
572: .Pq Sq |\& .
573: This pattern therefore matches
574: .Em either
575: the string
576: .Qq foo
577: .Em or
578: the string
579: .Qq ba
580: followed by zero-or-more r's.
581: To match
582: .Qq foo
583: or zero-or-more "bar"'s,
584: use:
585: .Pp
586: .D1 foo|(bar)*
587: .Pp
1.1 deraadt 588: and to match zero-or-more "foo"'s-or-"bar"'s:
1.16 jmc 589: .Pp
590: .D1 (foo|bar)*
591: .Pp
1.1 deraadt 592: In addition to characters and ranges of characters, character classes
593: can also contain character class
1.16 jmc 594: .Em expressions .
1.1 deraadt 595: These are expressions enclosed inside
1.16 jmc 596: .Sq [:
597: and
598: .Sq :]
599: delimiters (which themselves must appear between the
600: .Sq [
1.1 deraadt 601: and
1.16 jmc 602: .Sq ]\&
603: of the
1.1 deraadt 604: character class; other elements may occur inside the character class, too).
605: The valid expressions are:
1.16 jmc 606: .Bd -unfilled -offset indent
607: [:alnum:] [:alpha:] [:blank:]
608: [:cntrl:] [:digit:] [:graph:]
609: [:lower:] [:print:] [:punct:]
610: [:space:] [:upper:] [:xdigit:]
611: .Ed
612: .Pp
1.1 deraadt 613: These expressions all designate a set of characters equivalent to
614: the corresponding standard C
1.16 jmc 615: .Fn isXXX
616: function.
617: For example, [:alnum:] designates those characters for which
618: .Xr isalnum 3
619: returns true \- i.e., any alphabetic or numeric.
1.1 deraadt 620: Some systems don't provide
1.16 jmc 621: .Xr isblank 3 ,
622: so
623: .Nm
624: defines [:blank:] as a blank or a tab.
625: .Pp
1.1 deraadt 626: For example, the following character classes are all equivalent:
1.16 jmc 627: .Bd -unfilled -offset indent
628: [[:alnum:]]
629: [[:alpha:][:digit:]]
630: [[:alpha:]0-9]
631: [a-zA-Z0-9]
632: .Ed
633: .Pp
634: If the scanner is case-insensitive (the
635: .Fl i
636: flag), then [:upper:] and [:lower:] are equivalent to [:alpha:].
637: .Pp
1.1 deraadt 638: Some notes on patterns:
1.16 jmc 639: .Bl -dash
640: .It
641: A negated character class such as the example
642: .Qq [^A-Z]
643: above will match a newline unless "\en"
644: .Pq or an equivalent escape sequence
645: is one of the characters explicitly present in the negated character class
646: (e.g.,
647: .Qq [^A-Z\en] ) .
648: This is unlike how many other regular expression tools treat negated character
649: classes, but unfortunately the inconsistency is historically entrenched.
650: Matching newlines means that a pattern like
651: .Qq [^"]*
652: can match the entire input unless there's another quote in the input.
653: .It
654: A rule can have at most one instance of trailing context
655: (the
656: .Sq /
657: operator or the
658: .Sq $
659: operator).
660: The start condition,
661: .Sq ^ ,
662: and
663: .Qq <<EOF>>
664: patterns can only occur at the beginning of a pattern, and, as well as with
665: .Sq /
666: and
667: .Sq $ ,
668: cannot be grouped inside parentheses.
669: A
670: .Sq ^
671: which does not occur at the beginning of a rule or a
672: .Sq $
673: which does not occur at the end of a rule loses its special properties
674: and is treated as a normal character.
675: .It
1.1 deraadt 676: The following are illegal:
1.16 jmc 677: .Bd -unfilled -offset indent
678: foo/bar$
679: <sc1>foo<sc2>bar
680: .Ed
681: .Pp
682: Note that the first of these, can be written
683: .Qq foo/bar\en .
684: .It
685: The following will result in
686: .Sq $
687: or
688: .Sq ^
689: being treated as a normal character:
690: .Bd -unfilled -offset indent
691: foo|(bar$)
692: foo|^bar
693: .Ed
694: .Pp
695: If what's wanted is a
696: .Qq foo
697: or a bar-followed-by-a-newline, the following could be used
698: (the special
699: .Sq |\&
700: action is explained below):
701: .Bd -unfilled -offset indent
702: foo |
703: bar$ /* action goes here */
704: .Ed
705: .Pp
1.1 deraadt 706: A similar trick will work for matching a foo or a
707: bar-at-the-beginning-of-a-line.
1.16 jmc 708: .El
709: .Sh HOW THE INPUT IS MATCHED
710: When the generated scanner is run,
711: it analyzes its input looking for strings which match any of its patterns.
712: If it finds more than one match,
713: it takes the one matching the most text
714: (for trailing context rules, this includes the length of the trailing part,
715: even though it will then be returned to the input).
716: If it finds two or more matches of the same length,
717: the rule listed first in the
718: .Nm
1.1 deraadt 719: input file is chosen.
1.16 jmc 720: .Pp
1.1 deraadt 721: Once the match is determined, the text corresponding to the match
722: (called the
1.16 jmc 723: .Em token )
1.1 deraadt 724: is made available in the global character pointer
1.16 jmc 725: .Fa yytext ,
1.1 deraadt 726: and its length in the global integer
1.16 jmc 727: .Fa yyleng .
1.1 deraadt 728: The
1.16 jmc 729: .Em action
730: corresponding to the matched pattern is then executed
731: .Pq a more detailed description of actions follows ,
732: and then the remaining input is scanned for another match.
733: .Pp
734: If no match is found, then the default rule is executed:
735: the next character in the input is considered matched and
736: copied to the standard output.
737: Thus, the simplest legal
738: .Nm
1.1 deraadt 739: input is:
1.16 jmc 740: .Pp
741: .D1 %%
742: .Pp
743: which generates a scanner that simply copies its input
744: .Pq one character at a time
745: to its output.
746: .Pp
1.1 deraadt 747: Note that
1.16 jmc 748: .Fa yytext
749: can be defined in two different ways:
750: either as a character pointer or as a character array.
751: Which definition
752: .Nm
753: uses can be controlled by including one of the special directives
754: .Dq %pointer
755: or
756: .Dq %array
757: in the first
758: .Pq definitions
759: section of flex input.
760: The default is
761: .Dq %pointer ,
762: unless the
763: .Fl l
764: lex compatibility option is used, in which case
765: .Fa yytext
1.1 deraadt 766: will be an array.
767: The advantage of using
1.16 jmc 768: .Dq %pointer
1.1 deraadt 769: is substantially faster scanning and no buffer overflow when matching
1.16 jmc 770: very large tokens
771: .Pq unless not enough dynamic memory is available .
772: The disadvantage is that actions are restricted in how they can modify
773: .Fa yytext
774: .Pq see the next section ,
775: and calls to the
776: .Fn unput
1.10 deraadt 777: function destroy the present contents of
1.16 jmc 778: .Fa yytext ,
1.1 deraadt 779: which can be a considerable porting headache when moving between different
1.16 jmc 780: .Nm lex
1.1 deraadt 781: versions.
1.16 jmc 782: .Pp
1.1 deraadt 783: The advantage of
1.16 jmc 784: .Dq %array
785: is that
786: .Fa yytext
787: can be modified as much as wanted, and calls to
788: .Fn unput
1.1 deraadt 789: do not destroy
1.16 jmc 790: .Fa yytext
791: .Pq see below .
792: Furthermore, existing
793: .Nm lex
1.1 deraadt 794: programs sometimes access
1.16 jmc 795: .Fa yytext
1.1 deraadt 796: externally using declarations of the form:
1.16 jmc 797: .Pp
798: .D1 extern char yytext[];
799: .Pp
1.1 deraadt 800: This definition is erroneous when used with
1.16 jmc 801: .Dq %pointer ,
1.1 deraadt 802: but correct for
1.16 jmc 803: .Dq %array .
804: .Pp
805: .Dq %array
1.1 deraadt 806: defines
1.16 jmc 807: .Fa yytext
1.1 deraadt 808: to be an array of
1.16 jmc 809: .Dv YYLMAX
810: characters, which defaults to a fairly large value.
811: The size can be changed by simply #define'ing
812: .Dv YYLMAX
813: to a different value in the first section of
814: .Nm
815: input.
816: As mentioned above, with
817: .Dq %pointer
818: yytext grows dynamically to accommodate large tokens.
819: While this means a
820: .Dq %pointer
821: scanner can accommodate very large tokens
822: .Pq such as matching entire blocks of comments ,
823: bear in mind that each time the scanner must resize
824: .Fa yytext
1.1 deraadt 825: it also must rescan the entire token from the beginning, so matching such
826: tokens can prove slow.
1.16 jmc 827: .Fa yytext
828: presently does not dynamically grow if a call to
829: .Fn unput
1.1 deraadt 830: results in too much text being pushed back; instead, a run-time error results.
1.16 jmc 831: .Pp
832: Also note that
833: .Dq %array
834: cannot be used with C++ scanner classes
835: .Pq the c++ option; see below .
836: .Sh ACTIONS
837: Each pattern in a rule has a corresponding action,
838: which can be any arbitrary C statement.
839: The pattern ends at the first non-escaped whitespace character;
840: the remainder of the line is its action.
841: If the action is empty,
842: then when the pattern is matched the input token is simply discarded.
843: For example, here is the specification for a program
844: which deletes all occurrences of
845: .Qq zap me
846: from its input:
847: .Bd -literal -offset indent
848: %%
849: "zap me"
850: .Ed
851: .Pp
1.1 deraadt 852: (It will copy all other characters in the input to the output since
853: they will be matched by the default rule.)
1.16 jmc 854: .Pp
1.1 deraadt 855: Here is a program which compresses multiple blanks and tabs down to
856: a single blank, and throws away whitespace found at the end of a line:
1.16 jmc 857: .Bd -literal -offset indent
858: %%
859: [ \et]+ putchar(' ');
860: [ \et]+$ /* ignore this token */
861: .Ed
862: .Pp
863: If the action contains a
864: .Sq { ,
865: then the action spans till the balancing
866: .Sq }
1.1 deraadt 867: is found, and the action may cross multiple lines.
1.16 jmc 868: .Nm
1.1 deraadt 869: knows about C strings and comments and won't be fooled by braces found
870: within them, but also allows actions to begin with
1.16 jmc 871: .Sq %{
1.1 deraadt 872: and will consider the action to be all the text up to the next
1.16 jmc 873: .Sq %}
874: .Pq regardless of ordinary braces inside the action .
875: .Pp
876: An action consisting solely of a vertical bar
877: .Pq Sq |\&
878: means
879: .Qq same as the action for the next rule .
880: See below for an illustration.
881: .Pp
882: Actions can include arbitrary C code,
883: including return statements to return a value to whatever routine called
884: .Fn yylex .
1.1 deraadt 885: Each time
1.16 jmc 886: .Fn yylex
887: is called, it continues processing tokens from where it last left off
888: until it either reaches the end of the file or executes a return.
889: .Pp
1.1 deraadt 890: Actions are free to modify
1.16 jmc 891: .Fa yytext
892: except for lengthening it
893: (adding characters to its end \- these will overwrite later characters in the
894: input stream).
895: This, however, does not apply when using
896: .Dq %array
897: .Pq see above ;
898: in that case,
899: .Fa yytext
1.1 deraadt 900: may be freely modified in any way.
1.16 jmc 901: .Pp
1.1 deraadt 902: Actions are free to modify
1.16 jmc 903: .Fa yyleng
1.1 deraadt 904: except they should not do so if the action also includes use of
1.16 jmc 905: .Fn yymore
906: .Pq see below .
907: .Pp
1.1 deraadt 908: There are a number of special directives which can be included within
909: an action:
1.16 jmc 910: .Bl -tag -width Ds
911: .It ECHO
912: Copies
913: .Fa yytext
914: to the scanner's output.
915: .It BEGIN
916: Followed by the name of a start condition, places the scanner in the
917: corresponding start condition
918: .Pq see below .
919: .It REJECT
920: Directs the scanner to proceed on to the
921: .Qq second best
922: rule which matched the input
923: .Pq or a prefix of the input .
924: The rule is chosen as described above in
925: .Sx HOW THE INPUT IS MATCHED ,
926: and
927: .Fa yytext
1.1 deraadt 928: and
1.16 jmc 929: .Fa yyleng
1.1 deraadt 930: set up appropriately.
931: It may either be one which matched as much text
932: as the originally chosen rule but came later in the
1.16 jmc 933: .Nm
1.1 deraadt 934: input file, or one which matched less text.
935: For example, the following will both count the
1.16 jmc 936: words in the input and call the routine
937: .Fn special
938: whenever
939: .Qq frob
940: is seen:
941: .Bd -literal -offset indent
942: int word_count = 0;
943: %%
944:
945: frob special(); REJECT;
946: [^ \et\en]+ ++word_count;
947: .Ed
948: .Pp
1.1 deraadt 949: Without the
1.16 jmc 950: .Em REJECT ,
951: any "frob"'s in the input would not be counted as words,
952: since the scanner normally executes only one action per token.
1.1 deraadt 953: Multiple
1.16 jmc 954: .Em REJECT Ns 's
955: are allowed,
956: each one finding the next best choice to the currently active rule.
957: For example, when the following scanner scans the token
958: .Qq abcd ,
959: it will write
960: .Qq abcdabcaba
961: to the output:
962: .Bd -literal -offset indent
963: %%
964: a |
965: ab |
966: abc |
967: abcd ECHO; REJECT;
968: \&.|\en /* eat up any unmatched character */
969: .Ed
970: .Pp
1.1 deraadt 971: (The first three rules share the fourth's action since they use
1.16 jmc 972: the special
973: .Sq |\&
974: action.)
975: .Em REJECT
1.1 deraadt 976: is a particularly expensive feature in terms of scanner performance;
1.16 jmc 977: if it is used in any of the scanner's actions it will slow down
978: all of the scanner's matching.
979: Furthermore,
980: .Em REJECT
1.1 deraadt 981: cannot be used with the
1.16 jmc 982: .Fl Cf
1.1 deraadt 983: or
1.16 jmc 984: .Fl CF
985: options
986: .Pq see below .
987: .Pp
1.1 deraadt 988: Note also that unlike the other special actions,
1.16 jmc 989: .Em REJECT
1.1 deraadt 990: is a
1.16 jmc 991: .Em branch ;
992: code immediately following it in the action will not be executed.
993: .It yymore()
994: Tells the scanner that the next time it matches a rule, the corresponding
995: token should be appended onto the current value of
996: .Fa yytext
997: rather than replacing it.
998: For example, given the input
999: .Qq mega-kludge
1000: the following will write
1001: .Qq mega-mega-kludge
1002: to the output:
1003: .Bd -literal -offset indent
1004: %%
1005: mega- ECHO; yymore();
1006: kludge ECHO;
1007: .Ed
1008: .Pp
1009: First
1010: .Qq mega-
1011: is matched and echoed to the output.
1012: Then
1013: .Qq kludge
1014: is matched, but the previous
1015: .Qq mega-
1016: is still hanging around at the beginning of
1017: .Fa yytext
1.1 deraadt 1018: so the
1.16 jmc 1019: .Em ECHO
1020: for the
1021: .Qq kludge
1022: rule will actually write
1023: .Qq mega-kludge .
1024: .Pp
1.1 deraadt 1025: Two notes regarding use of
1.16 jmc 1026: .Fn yymore :
1.1 deraadt 1027: First,
1.16 jmc 1028: .Fn yymore
1.1 deraadt 1029: depends on the value of
1.16 jmc 1030: .Fa yyleng
1031: correctly reflecting the size of the current token, so
1032: .Fa yyleng
1033: must not be modified when using
1034: .Fn yymore .
1.1 deraadt 1035: Second, the presence of
1.16 jmc 1036: .Fn yymore
1.1 deraadt 1037: in the scanner's action entails a minor performance penalty in the
1038: scanner's matching speed.
1.16 jmc 1039: .It yyless(n)
1040: Returns all but the first
1041: .Ar n
1.1 deraadt 1042: characters of the current token back to the input stream, where they
1043: will be rescanned when the scanner looks for the next match.
1.16 jmc 1044: .Fa yytext
1.1 deraadt 1045: and
1.16 jmc 1046: .Fa yyleng
1.1 deraadt 1047: are adjusted appropriately (e.g.,
1.16 jmc 1048: .Fa yyleng
1.1 deraadt 1049: will now be equal to
1.16 jmc 1050: .Ar n ) .
1051: For example, on the input
1052: .Qq foobar
1053: the following will write out
1054: .Qq foobarbar :
1055: .Bd -literal -offset indent
1056: %%
1057: foobar ECHO; yyless(3);
1058: [a-z]+ ECHO;
1059: .Ed
1060: .Pp
1.1 deraadt 1061: An argument of 0 to
1.16 jmc 1062: .Fa yyless
1063: will cause the entire current input string to be scanned again.
1064: Unless how the scanner will subsequently process its input has been changed
1065: (using
1066: .Em BEGIN ,
1067: for example),
1068: this will result in an endless loop.
1069: .Pp
1.1 deraadt 1070: Note that
1.16 jmc 1071: .Fa yyless
1072: is a macro and can only be used in the
1073: .Nm
1074: input file, not from other source files.
1075: .It unput(c)
1076: Puts the character
1077: .Ar c
1078: back into the input stream.
1079: It will be the next character scanned.
1.1 deraadt 1080: The following action will take the current token and cause it
1081: to be rescanned enclosed in parentheses.
1.16 jmc 1082: .Bd -literal -offset indent
1083: {
1084: int i;
1085: char *yycopy;
1086:
1087: /* Copy yytext because unput() trashes yytext */
1088: if ((yycopy = strdup(yytext)) == NULL)
1089: err(1, NULL);
1090: unput(')');
1091: for (i = yyleng - 1; i >= 0; --i)
1092: unput(yycopy[i]);
1093: unput('(');
1094: free(yycopy);
1095: }
1096: .Ed
1097: .Pp
1.1 deraadt 1098: Note that since each
1.16 jmc 1099: .Fn unput
1100: puts the given character back at the beginning of the input stream,
1101: pushing back strings must be done back-to-front.
1102: .Pp
1.1 deraadt 1103: An important potential problem when using
1.16 jmc 1104: .Fn unput
1105: is that if using
1106: .Dq %pointer
1107: .Pq the default ,
1108: a call to
1109: .Fn unput
1110: destroys the contents of
1111: .Fa yytext ,
1.1 deraadt 1112: starting with its rightmost character and devouring one character to
1.16 jmc 1113: the left with each call.
1114: If the value of
1115: .Fa yytext
1116: should be preserved after a call to
1117: .Fn unput
1118: .Pq as in the above example ,
1119: it must either first be copied elsewhere, or the scanner must be built using
1120: .Dq %array
1121: instead (see
1122: .Sx HOW THE INPUT IS MATCHED ) .
1123: .Pp
1124: Finally, note that EOF cannot be put back
1.1 deraadt 1125: to attempt to mark the input stream with an end-of-file.
1.16 jmc 1126: .It input()
1127: Reads the next character from the input stream.
1128: For example, the following is one way to eat up C comments:
1129: .Bd -literal -offset indent
1130: %%
1131: "/*" {
1132: int c;
1133:
1134: for (;;) {
1135: while ((c = input()) != '*' && c != EOF)
1136: ; /* eat up text of comment */
1137:
1138: if (c == '*') {
1139: while ((c = input()) == '*')
1140: ;
1141: if (c == '/')
1142: break; /* found the end */
1143: }
1144:
1145: if (c == EOF) {
1146: errx(1, "EOF in comment");
1.1 deraadt 1147: break;
1148: }
1.16 jmc 1149: }
1150: }
1151: .Ed
1152: .Pp
1153: (Note that if the scanner is compiled using C++, then
1154: .Fn input
1.1 deraadt 1155: is instead referred to as
1.16 jmc 1156: .Fn yyinput ,
1157: in order to avoid a name clash with the C++ stream by the name of input.)
1158: .It YY_FLUSH_BUFFER
1159: Flushes the scanner's internal buffer
1160: so that the next time the scanner attempts to match a token,
1161: it will first refill the buffer using
1162: .Dv YY_INPUT
1163: (see
1164: .Sx THE GENERATED SCANNER ,
1165: below).
1166: This action is a special case of the more general
1167: .Fn yy_flush_buffer
1168: function, described below in the section
1169: .Sx MULTIPLE INPUT BUFFERS .
1170: .It yyterminate()
1171: Can be used in lieu of a return statement in an action.
1172: It terminates the scanner and returns a 0 to the scanner's caller, indicating
1173: .Qq all done .
1.1 deraadt 1174: By default,
1.16 jmc 1175: .Fn yyterminate
1176: is also called when an end-of-file is encountered.
1177: It is a macro and may be redefined.
1178: .El
1179: .Sh THE GENERATED SCANNER
1.1 deraadt 1180: The output of
1.16 jmc 1181: .Nm
1.1 deraadt 1182: is the file
1.16 jmc 1183: .Pa lex.yy.c ,
1.1 deraadt 1184: which contains the scanning routine
1.16 jmc 1185: .Fn yylex ,
1186: a number of tables used by it for matching tokens,
1187: and a number of auxiliary routines and macros.
1188: By default,
1189: .Fn yylex
1.1 deraadt 1190: is declared as follows:
1.16 jmc 1191: .Bd -unfilled -offset indent
1192: int yylex()
1193: {
1194: ... various definitions and the actions in here ...
1195: }
1196: .Ed
1197: .Pp
1198: (If the environment supports function prototypes, then it will
1199: be "int yylex(void)".)
1200: This definition may be changed by defining the
1201: .Dv YY_DECL
1202: macro.
1203: For example:
1204: .Bd -literal -offset indent
1205: #define YY_DECL float lexscan(a, b) float a, b;
1206: .Ed
1207: .Pp
1208: would give the scanning routine the name
1209: .Em lexscan ,
1210: returning a float, and taking two floats as arguments.
1211: Note that if arguments are given to the scanning routine using a
1212: K&R-style/non-prototyped function declaration,
1213: the definition must be terminated with a semi-colon
1214: .Pq Sq ;\& .
1215: .Pp
1.1 deraadt 1216: Whenever
1.16 jmc 1217: .Fn yylex
1.1 deraadt 1218: is called, it scans tokens from the global input file
1.16 jmc 1219: .Pa yyin
1220: .Pq which defaults to stdin .
1221: It continues until it either reaches an end-of-file
1222: .Pq at which point it returns the value 0
1223: or one of its actions executes a
1224: .Em return
1.1 deraadt 1225: statement.
1.16 jmc 1226: .Pp
1.1 deraadt 1227: If the scanner reaches an end-of-file, subsequent calls are undefined
1228: unless either
1.16 jmc 1229: .Em yyin
1230: is pointed at a new input file
1231: .Pq in which case scanning continues from that file ,
1232: or
1233: .Fn yyrestart
1.1 deraadt 1234: is called.
1.16 jmc 1235: .Fn yyrestart
1.1 deraadt 1236: takes one argument, a
1.16 jmc 1237: .Fa FILE *
1238: pointer (which can be nil, if
1239: .Dv YY_INPUT
1240: has been set up to scan from a source other than
1241: .Em yyin ) ,
1.1 deraadt 1242: and initializes
1.16 jmc 1243: .Em yyin
1244: for scanning from that file.
1245: Essentially there is no difference between just assigning
1246: .Em yyin
1.1 deraadt 1247: to a new input file or using
1.16 jmc 1248: .Fn yyrestart
1249: to do so; the latter is available for compatibility with previous versions of
1250: .Nm ,
1.1 deraadt 1251: and because it can be used to switch input files in the middle of scanning.
1.16 jmc 1252: It can also be used to throw away the current input buffer,
1253: by calling it with an argument of
1254: .Em yyin ;
1.1 deraadt 1255: but better is to use
1.16 jmc 1256: .Dv YY_FLUSH_BUFFER
1257: .Pq see above .
1.1 deraadt 1258: Note that
1.16 jmc 1259: .Fn yyrestart
1260: does not reset the start condition to
1261: .Em INITIAL
1262: (see
1263: .Sx START CONDITIONS ,
1264: below).
1265: .Pp
1.1 deraadt 1266: If
1.16 jmc 1267: .Fn yylex
1.1 deraadt 1268: stops scanning due to executing a
1.16 jmc 1269: .Em return
1.1 deraadt 1270: statement in one of the actions, the scanner may then be called again and it
1271: will resume scanning where it left off.
1.16 jmc 1272: .Pp
1273: By default
1274: .Pq and for purposes of efficiency ,
1275: the scanner uses block-reads rather than simple
1276: .Xr getc 3
1.1 deraadt 1277: calls to read characters from
1.16 jmc 1278: .Em yyin .
1.1 deraadt 1279: The nature of how it gets its input can be controlled by defining the
1.16 jmc 1280: .Dv YY_INPUT
1.1 deraadt 1281: macro.
1.16 jmc 1282: .Dv YY_INPUT Ns 's
1283: calling sequence is
1284: .Qq YY_INPUT(buf,result,max_size) .
1285: Its action is to place up to
1286: .Dv max_size
1.1 deraadt 1287: characters in the character array
1.16 jmc 1288: .Em buf
1.1 deraadt 1289: and return in the integer variable
1.16 jmc 1290: .Em result
1291: either the number of characters read or the constant
1292: .Dv YY_NULL
1293: (0 on
1294: .Ux
1295: systems)
1296: to indicate
1297: .Dv EOF .
1298: The default
1299: .Dv YY_INPUT
1300: reads from the global file-pointer
1301: .Qq yyin .
1302: .Pp
1303: A sample definition of
1304: .Dv YY_INPUT
1305: .Pq in the definitions section of the input file :
1306: .Bd -unfilled -offset indent
1307: %{
1308: #define YY_INPUT(buf,result,max_size) \e
1309: { \e
1310: int c = getchar(); \e
1311: result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \e
1312: }
1313: %}
1314: .Ed
1315: .Pp
1.1 deraadt 1316: This definition will change the input processing to occur
1317: one character at a time.
1.16 jmc 1318: .Pp
1319: When the scanner receives an end-of-file indication from
1320: .Dv YY_INPUT ,
1.1 deraadt 1321: it then checks the
1.16 jmc 1322: .Fn yywrap
1323: function.
1324: If
1325: .Fn yywrap
1326: returns false
1327: .Pq zero ,
1328: then it is assumed that the function has gone ahead and set up
1329: .Em yyin
1330: to point to another input file, and scanning continues.
1331: If it returns true
1332: .Pq non-zero ,
1333: then the scanner terminates, returning 0 to its caller.
1334: Note that in either case, the start condition remains unchanged;
1335: it does not revert to
1336: .Em INITIAL .
1337: .Pp
1.1 deraadt 1338: If you do not supply your own version of
1.16 jmc 1339: .Fn yywrap ,
1.1 deraadt 1340: then you must either use
1.16 jmc 1341: .Dq %option noyywrap
1.1 deraadt 1342: (in which case the scanner behaves as though
1.16 jmc 1343: .Fn yywrap
1.1 deraadt 1344: returned 1), or you must link with
1.16 jmc 1345: .Fl lfl
1.1 deraadt 1346: to obtain the default version of the routine, which always returns 1.
1.16 jmc 1347: .Pp
1.1 deraadt 1348: Three routines are available for scanning from in-memory buffers rather
1349: than files:
1.16 jmc 1350: .Fn yy_scan_string ,
1351: .Fn yy_scan_bytes ,
1.1 deraadt 1352: and
1.16 jmc 1353: .Fn yy_scan_buffer .
1354: See the discussion of them below in the section
1355: .Sx MULTIPLE INPUT BUFFERS .
1356: .Pp
1.1 deraadt 1357: The scanner writes its
1.16 jmc 1358: .Em ECHO
1.1 deraadt 1359: output to the
1.16 jmc 1360: .Em yyout
1361: global
1362: .Pq default, stdout ,
1363: which may be redefined by the user simply by assigning it to some other
1364: .Va FILE
1.1 deraadt 1365: pointer.
1.16 jmc 1366: .Sh START CONDITIONS
1367: .Nm
1368: provides a mechanism for conditionally activating rules.
1369: Any rule whose pattern is prefixed with
1370: .Qq Aq sc
1371: will only be active when the scanner is in the start condition named
1372: .Qq sc .
1373: For example,
1374: .Bd -literal -offset indent
1375: <STRING>[^"]* { /* eat up the string body ... */
1376: ...
1377: }
1378: .Ed
1379: .Pp
1380: will be active only when the scanner is in the
1381: .Qq STRING
1382: start condition, and
1383: .Bd -literal -offset indent
1384: <INITIAL,STRING,QUOTE>\e. { /* handle an escape ... */
1385: ...
1386: }
1387: .Ed
1388: .Pp
1389: will be active only when the current start condition is either
1390: .Qq INITIAL ,
1391: .Qq STRING ,
1392: or
1393: .Qq QUOTE .
1394: .Pp
1395: Start conditions are declared in the definitions
1396: .Pq first
1397: section of the input using unindented lines beginning with either
1398: .Sq %s
1.1 deraadt 1399: or
1.16 jmc 1400: .Sq %x
1.1 deraadt 1401: followed by a list of names.
1402: The former declares
1.16 jmc 1403: .Em inclusive
1.1 deraadt 1404: start conditions, the latter
1.16 jmc 1405: .Em exclusive
1406: start conditions.
1407: A start condition is activated using the
1408: .Em BEGIN
1409: action.
1410: Until the next
1411: .Em BEGIN
1412: action is executed, rules with the given start condition will be active and
1.1 deraadt 1413: rules with other start conditions will be inactive.
1.16 jmc 1414: If the start condition is inclusive,
1.1 deraadt 1415: then rules with no start conditions at all will also be active.
1.16 jmc 1416: If it is exclusive,
1417: then only rules qualified with the start condition will be active.
1.1 deraadt 1418: A set of rules contingent on the same exclusive start condition
1419: describe a scanner which is independent of any of the other rules in the
1.16 jmc 1420: .Nm
1421: input.
1422: Because of this, exclusive start conditions make it easy to specify
1423: .Qq mini-scanners
1.1 deraadt 1424: which scan portions of the input that are syntactically different
1.16 jmc 1425: from the rest
1426: .Pq e.g., comments .
1427: .Pp
1.1 deraadt 1428: If the distinction between inclusive and exclusive start conditions
1429: is still a little vague, here's a simple example illustrating the
1.16 jmc 1430: connection between the two.
1431: The set of rules:
1432: .Bd -literal -offset indent
1433: %s example
1434: %%
1435:
1436: <example>foo do_something();
1437:
1438: bar something_else();
1439: .Ed
1440: .Pp
1.1 deraadt 1441: is equivalent to
1.16 jmc 1442: .Bd -literal -offset indent
1443: %x example
1444: %%
1445:
1446: <example>foo do_something();
1447:
1448: <INITIAL,example>bar something_else();
1449: .Ed
1450: .Pp
1.1 deraadt 1451: Without the
1.16 jmc 1452: .Aq INITIAL,example
1.1 deraadt 1453: qualifier, the
1.16 jmc 1454: .Dq bar
1455: pattern in the second example wouldn't be active
1456: .Pq i.e., couldn't match
1.1 deraadt 1457: when in start condition
1.16 jmc 1458: .Dq example .
1.1 deraadt 1459: If we just used
1.16 jmc 1460: .Aq example
1.1 deraadt 1461: to qualify
1.16 jmc 1462: .Dq bar ,
1.1 deraadt 1463: though, then it would only be active in
1.16 jmc 1464: .Dq example
1.1 deraadt 1465: and not in
1.16 jmc 1466: .Em INITIAL ,
1467: while in the first example it's active in both,
1468: because in the first example the
1469: .Dq example
1470: start condition is an inclusive
1471: .Pq Sq %s
1.1 deraadt 1472: start condition.
1.16 jmc 1473: .Pp
1.1 deraadt 1474: Also note that the special start-condition specifier
1.16 jmc 1475: .Sq Aq *
1476: matches every start condition.
1477: Thus, the above example could also have been written:
1478: .Bd -literal -offset indent
1479: %x example
1480: %%
1481:
1482: <example>foo do_something();
1483:
1484: <*>bar something_else();
1485: .Ed
1486: .Pp
1.1 deraadt 1487: The default rule (to
1.16 jmc 1488: .Em ECHO
1489: any unmatched character) remains active in start conditions.
1490: It is equivalent to:
1491: .Bd -literal -offset indent
1492: <*>.|\en ECHO;
1493: .Ed
1494: .Pp
1495: .Dq BEGIN(0)
1.1 deraadt 1496: returns to the original state where only the rules with
1.16 jmc 1497: no start conditions are active.
1498: This state can also be referred to as the start-condition
1499: .Em INITIAL ,
1500: so
1501: .Dq BEGIN(INITIAL)
1.1 deraadt 1502: is equivalent to
1.16 jmc 1503: .Dq BEGIN(0) .
1.1 deraadt 1504: (The parentheses around the start condition name are not required but
1505: are considered good style.)
1.16 jmc 1506: .Pp
1507: .Em BEGIN
1.1 deraadt 1508: actions can also be given as indented code at the beginning
1.16 jmc 1509: of the rules section.
1510: For example, the following will cause the scanner to enter the
1511: .Qq SPECIAL
1512: start condition whenever
1513: .Fn yylex
1.1 deraadt 1514: is called and the global variable
1.16 jmc 1515: .Fa enter_special
1.1 deraadt 1516: is true:
1.16 jmc 1517: .Bd -literal -offset indent
1518: int enter_special;
1.1 deraadt 1519:
1.16 jmc 1520: %x SPECIAL
1521: %%
1522: if (enter_special)
1.1 deraadt 1523: BEGIN(SPECIAL);
1524:
1.16 jmc 1525: <SPECIAL>blahblahblah
1526: \&...more rules follow...
1527: .Ed
1528: .Pp
1.1 deraadt 1529: To illustrate the uses of start conditions,
1530: here is a scanner which provides two different interpretations
1.16 jmc 1531: of a string like
1532: .Qq 123.456 .
1533: By default it will treat it as three tokens: the integer
1534: .Qq 123 ,
1535: a dot
1536: .Pq Sq .\& ,
1537: and the integer
1538: .Qq 456 .
1.1 deraadt 1539: But if the string is preceded earlier in the line by the string
1.16 jmc 1540: .Qq expect-floats
1541: it will treat it as a single token, the floating-point number 123.456:
1542: .Bd -literal -offset indent
1543: %{
1544: #include <math.h>
1545: %}
1546: %s expect
1547:
1548: %%
1549: expect-floats BEGIN(expect);
1550:
1551: <expect>[0-9]+"."[0-9]+ {
1552: printf("found a float, = %f\en",
1553: atof(yytext));
1554: }
1555: <expect>\en {
1556: /*
1557: * That's the end of the line, so
1558: * we need another "expect-number"
1559: * before we'll recognize any more
1560: * numbers.
1561: */
1562: BEGIN(INITIAL);
1563: }
1564:
1565: [0-9]+ {
1566: printf("found an integer, = %d\en",
1567: atoi(yytext));
1568: }
1569:
1570: "." printf("found a dot\en");
1571: .Ed
1572: .Pp
1573: Here is a scanner which recognizes
1574: .Pq and discards
1575: C comments while maintaining a count of the current input line:
1576: .Bd -literal -offset indent
1577: %x comment
1578: %%
1579: int line_num = 1;
1580:
1581: "/*" BEGIN(comment);
1582:
1583: <comment>[^*\en]* /* eat anything that's not a '*' */
1584: <comment>"*"+[^*/\en]* /* eat up '*'s not followed by '/'s */
1585: <comment>\en ++line_num;
1586: <comment>"*"+"/" BEGIN(INITIAL);
1587: .Ed
1588: .Pp
1.1 deraadt 1589: This scanner goes to a bit of trouble to match as much
1.16 jmc 1590: text as possible with each rule.
1591: In general, when attempting to write a high-speed scanner
1592: try to match as much as possible in each rule, as it's a big win.
1593: .Pp
1.10 deraadt 1594: Note that start-condition names are really integer values and
1.16 jmc 1595: can be stored as such.
1596: Thus, the above could be extended in the following fashion:
1597: .Bd -literal -offset indent
1598: %x comment foo
1599: %%
1600: int line_num = 1;
1601: int comment_caller;
1602:
1603: "/*" {
1604: comment_caller = INITIAL;
1605: BEGIN(comment);
1606: }
1607:
1608: \&...
1609:
1610: <foo>"/*" {
1611: comment_caller = foo;
1612: BEGIN(comment);
1613: }
1614:
1615: <comment>[^*\en]* /* eat anything that's not a '*' */
1616: <comment>"*"+[^*/\en]* /* eat up '*'s not followed by '/'s */
1617: <comment>\en ++line_num;
1618: <comment>"*"+"/" BEGIN(comment_caller);
1619: .Ed
1620: .Pp
1621: Furthermore, the current start condition can be accessed by using
1.1 deraadt 1622: the integer-valued
1.16 jmc 1623: .Dv YY_START
1624: macro.
1625: For example, the above assignments to
1626: .Em comment_caller
1.1 deraadt 1627: could instead be written
1.16 jmc 1628: .Pp
1629: .Dl comment_caller = YY_START;
1630: .Pp
1.1 deraadt 1631: Flex provides
1.16 jmc 1632: .Dv YYSTATE
1.1 deraadt 1633: as an alias for
1.16 jmc 1634: .Dv YY_START
1.1 deraadt 1635: (since that is what's used by AT&T
1.16 jmc 1636: .Nm lex ) .
1637: .Pp
1638: Note that start conditions do not have their own name-space;
1639: %s's and %x's declare names in the same fashion as #define's.
1640: .Pp
1.1 deraadt 1641: Finally, here's an example of how to match C-style quoted strings using
1.16 jmc 1642: exclusive start conditions, including expanded escape sequences
1643: (but not including checking for a string that's too long):
1644: .Bd -literal -offset indent
1645: %x str
1646:
1647: %%
1648: #define MAX_STR_CONST 1024
1649: char string_buf[MAX_STR_CONST];
1650: char *string_buf_ptr;
1651:
1652: \e" string_buf_ptr = string_buf; BEGIN(str);
1653:
1654: <str>\e" { /* saw closing quote - all done */
1655: BEGIN(INITIAL);
1656: *string_buf_ptr = '\e0';
1657: /*
1658: * return string constant token type and
1659: * value to parser
1660: */
1661: }
1662:
1663: <str>\en {
1664: /* error - unterminated string constant */
1665: /* generate error message */
1666: }
1667:
1668: <str>\e\e[0-7]{1,3} {
1669: /* octal escape sequence */
1670: int result;
1671:
1672: (void) sscanf(yytext + 1, "%o", &result);
1673:
1674: if (result > 0xff) {
1675: /* error, constant is out-of-bounds */
1676: } else
1677: *string_buf_ptr++ = result;
1678: }
1679:
1680: <str>\e\e[0-9]+ {
1681: /*
1682: * generate error - bad escape sequence; something
1683: * like '\e48' or '\e0777777'
1684: */
1685: }
1686:
1687: <str>\e\en *string_buf_ptr++ = '\en';
1688: <str>\e\et *string_buf_ptr++ = '\et';
1689: <str>\e\er *string_buf_ptr++ = '\er';
1690: <str>\e\eb *string_buf_ptr++ = '\eb';
1691: <str>\e\ef *string_buf_ptr++ = '\ef';
1692:
1693: <str>\e\e(.|\en) *string_buf_ptr++ = yytext[1];
1694:
1695: <str>[^\e\e\en\e"]+ {
1696: char *yptr = yytext;
1697:
1698: while (*yptr)
1699: *string_buf_ptr++ = *yptr++;
1700: }
1701: .Ed
1702: .Pp
1703: Often, such as in some of the examples above,
1704: a whole bunch of rules are all preceded by the same start condition(s).
1705: .Nm
1.1 deraadt 1706: makes this a little easier and cleaner by introducing a notion of
1707: start condition
1.16 jmc 1708: .Em scope .
1.1 deraadt 1709: A start condition scope is begun with:
1.16 jmc 1710: .Pp
1711: .Dl <SCs>{
1712: .Pp
1.1 deraadt 1713: where
1.16 jmc 1714: .Dq SCs
1715: is a list of one or more start conditions.
1716: Inside the start condition scope, every rule automatically has the prefix
1717: .Aq SCs
1.1 deraadt 1718: applied to it, until a
1.16 jmc 1719: .Sq }
1.1 deraadt 1720: which matches the initial
1.16 jmc 1721: .Sq { .
1.1 deraadt 1722: So, for example,
1.16 jmc 1723: .Bd -literal -offset indent
1724: <ESC>{
1725: "\e\en" return '\en';
1726: "\e\er" return '\er';
1727: "\e\ef" return '\ef';
1728: "\e\e0" return '\e0';
1729: }
1730: .Ed
1731: .Pp
1.1 deraadt 1732: is equivalent to:
1.16 jmc 1733: .Bd -literal -offset indent
1734: <ESC>"\e\en" return '\en';
1735: <ESC>"\e\er" return '\er';
1736: <ESC>"\e\ef" return '\ef';
1737: <ESC>"\e\e0" return '\e0';
1738: .Ed
1739: .Pp
1.1 deraadt 1740: Start condition scopes may be nested.
1.16 jmc 1741: .Pp
1.1 deraadt 1742: Three routines are available for manipulating stacks of start conditions:
1.16 jmc 1743: .Bl -tag -width Ds
1744: .It void yy_push_state(int new_state)
1745: Pushes the current start condition onto the top of the start condition
1.1 deraadt 1746: stack and switches to
1.16 jmc 1747: .Fa new_state
1748: as though
1749: .Dq BEGIN new_state
1750: had been used
1751: .Pq recall that start condition names are also integers .
1752: .It void yy_pop_state()
1753: Pops the top of the stack and switches to it via
1754: .Em BEGIN .
1755: .It int yy_top_state()
1756: Returns the top of the stack without altering the stack's contents.
1757: .El
1758: .Pp
1.1 deraadt 1759: The start condition stack grows dynamically and so has no built-in
1.16 jmc 1760: size limitation.
1761: If memory is exhausted, program execution aborts.
1762: .Pp
1763: To use start condition stacks, scanners must include a
1764: .Dq %option stack
1765: directive (see
1766: .Sx OPTIONS
1767: below).
1768: .Sh MULTIPLE INPUT BUFFERS
1769: Some scanners
1770: (such as those which support
1771: .Qq include
1772: files)
1773: require reading from several input streams.
1774: As
1775: .Nm
1.1 deraadt 1776: scanners do a large amount of buffering, one cannot control
1777: where the next input will be read from by simply writing a
1.16 jmc 1778: .Dv YY_INPUT
1.1 deraadt 1779: which is sensitive to the scanning context.
1.16 jmc 1780: .Dv YY_INPUT
1.1 deraadt 1781: is only called when the scanner reaches the end of its buffer, which
1.16 jmc 1782: may be a long time after scanning a statement such as an
1783: .Qq include
1.1 deraadt 1784: which requires switching the input source.
1.16 jmc 1785: .Pp
1.1 deraadt 1786: To negotiate these sorts of problems,
1.16 jmc 1787: .Nm
1.1 deraadt 1788: provides a mechanism for creating and switching between multiple
1.16 jmc 1789: input buffers.
1790: An input buffer is created by using:
1791: .Pp
1792: .D1 YY_BUFFER_STATE yy_create_buffer(FILE *file, int size)
1793: .Pp
1.1 deraadt 1794: which takes a
1.16 jmc 1795: .Fa FILE
1796: pointer and a
1797: .Fa size
1798: and creates a buffer associated with the given file and large enough to hold
1799: .Fa size
1.1 deraadt 1800: characters (when in doubt, use
1.16 jmc 1801: .Dv YY_BUF_SIZE
1802: for the size).
1803: It returns a
1804: .Dv YY_BUFFER_STATE
1805: handle, which may then be passed to other routines
1806: .Pq see below .
1807: The
1808: .Dv YY_BUFFER_STATE
1.1 deraadt 1809: type is a pointer to an opaque
1.16 jmc 1810: .Dq struct yy_buffer_state
1811: structure, so
1812: .Dv YY_BUFFER_STATE
1813: variables may be safely initialized to
1814: .Dq ((YY_BUFFER_STATE) 0)
1815: if desired, and the opaque structure can also be referred to in order to
1816: correctly declare input buffers in source files other than that of scanners.
1817: Note that the
1818: .Fa FILE
1.1 deraadt 1819: pointer in the call to
1.16 jmc 1820: .Fn yy_create_buffer
1.1 deraadt 1821: is only used as the value of
1.16 jmc 1822: .Fa yyin
1.1 deraadt 1823: seen by
1.16 jmc 1824: .Dv YY_INPUT ;
1825: if
1826: .Dv YY_INPUT
1827: is redefined so that it no longer uses
1828: .Fa yyin ,
1829: then a nil
1830: .Fa FILE
1831: pointer can safely be passed to
1832: .Fn yy_create_buffer .
1833: To select a particular buffer to scan:
1834: .Pp
1835: .D1 void yy_switch_to_buffer(YY_BUFFER_STATE new_buffer)
1836: .Pp
1837: It switches the scanner's input buffer so subsequent tokens will
1.1 deraadt 1838: come from
1.16 jmc 1839: .Fa new_buffer .
1.1 deraadt 1840: Note that
1.16 jmc 1841: .Fn yy_switch_to_buffer
1842: may be used by
1843: .Fn yywrap
1844: to set things up for continued scanning,
1845: instead of opening a new file and pointing
1846: .Fa yyin
1847: at it.
1848: Note also that switching input sources via either
1849: .Fn yy_switch_to_buffer
1850: or
1851: .Fn yywrap
1852: does not change the start condition.
1853: .Pp
1854: .D1 void yy_delete_buffer(YY_BUFFER_STATE buffer)
1855: .Pp
1856: is used to reclaim the storage associated with a buffer.
1857: .Pf ( Fa buffer
1.1 deraadt 1858: can be nil, in which case the routine does nothing.)
1.16 jmc 1859: To clear the current contents of a buffer:
1860: .Pp
1861: .D1 void yy_flush_buffer(YY_BUFFER_STATE buffer)
1862: .Pp
1.1 deraadt 1863: This function discards the buffer's contents,
1.16 jmc 1864: so the next time the scanner attempts to match a token from the buffer,
1865: it will first fill the buffer anew using
1866: .Dv YY_INPUT .
1867: .Pp
1868: .Fn yy_new_buffer
1.1 deraadt 1869: is an alias for
1.16 jmc 1870: .Fn yy_create_buffer ,
1.1 deraadt 1871: provided for compatibility with the C++ use of
1.16 jmc 1872: .Em new
1.1 deraadt 1873: and
1.16 jmc 1874: .Em delete
1.1 deraadt 1875: for creating and destroying dynamic objects.
1.16 jmc 1876: .Pp
1.1 deraadt 1877: Finally, the
1.16 jmc 1878: .Dv YY_CURRENT_BUFFER
1.1 deraadt 1879: macro returns a
1.16 jmc 1880: .Dv YY_BUFFER_STATE
1.1 deraadt 1881: handle to the current buffer.
1.16 jmc 1882: .Pp
1.1 deraadt 1883: Here is an example of using these features for writing a scanner
1884: which expands include files (the
1.16 jmc 1885: .Aq Aq EOF
1.1 deraadt 1886: feature is discussed below):
1.16 jmc 1887: .Bd -literal -offset indent
1888: /*
1889: * the "incl" state is used for picking up the name
1890: * of an include file
1891: */
1892: %x incl
1893:
1894: %{
1895: #define MAX_INCLUDE_DEPTH 10
1896: YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
1897: int include_stack_ptr = 0;
1898: %}
1899:
1900: %%
1901: include BEGIN(incl);
1902:
1903: [a-z]+ ECHO;
1904: [^a-z\en]*\en? ECHO;
1905:
1906: <incl>[ \et]* /* eat the whitespace */
1907: <incl>[^ \et\en]+ { /* got the include file name */
1908: if (include_stack_ptr >= MAX_INCLUDE_DEPTH)
1909: errx(1, "Includes nested too deeply");
1910:
1911: include_stack[include_stack_ptr++] =
1912: YY_CURRENT_BUFFER;
1913:
1914: yyin = fopen(yytext, "r");
1915:
1916: if (yyin == NULL)
1917: err(1, NULL);
1.1 deraadt 1918:
1.16 jmc 1919: yy_switch_to_buffer(
1920: yy_create_buffer(yyin, YY_BUF_SIZE));
1.1 deraadt 1921:
1.16 jmc 1922: BEGIN(INITIAL);
1923: }
1.1 deraadt 1924:
1.16 jmc 1925: <<EOF>> {
1926: if (--include_stack_ptr < 0)
1.1 deraadt 1927: yyterminate();
1.16 jmc 1928: else {
1929: yy_delete_buffer(YY_CURRENT_BUFFER);
1.1 deraadt 1930: yy_switch_to_buffer(
1.16 jmc 1931: include_stack[include_stack_ptr]);
1932: }
1933: }
1934: .Ed
1935: .Pp
1.1 deraadt 1936: Three routines are available for setting up input buffers for
1.16 jmc 1937: scanning in-memory strings instead of files.
1938: All of them create a new input buffer for scanning the string,
1939: and return a corresponding
1940: .Dv YY_BUFFER_STATE
1941: handle (which should be deleted afterwards using
1942: .Fn yy_delete_buffer ) .
1943: They also switch to the new buffer using
1944: .Fn yy_switch_to_buffer ,
1.1 deraadt 1945: so the next call to
1.16 jmc 1946: .Fn yylex
1.1 deraadt 1947: will start scanning the string.
1.16 jmc 1948: .Bl -tag -width Ds
1949: .It yy_scan_string(const char *str)
1950: Scans a NUL-terminated string.
1951: .It yy_scan_bytes(const char *bytes, int len)
1952: Scans
1953: .Fa len
1954: bytes
1955: .Pq including possibly NUL's
1.1 deraadt 1956: starting at location
1.16 jmc 1957: .Fa bytes .
1958: .El
1959: .Pp
1960: Note that both of these functions create and scan a copy
1961: of the string or bytes.
1962: (This may be desirable, since
1963: .Fn yylex
1964: modifies the contents of the buffer it is scanning.)
1965: The copy can be avoided by using:
1966: .Bl -tag -width Ds
1967: .It yy_scan_buffer(char *base, yy_size_t size)
1968: Which scans the buffer starting at
1969: .Fa base ,
1.1 deraadt 1970: consisting of
1.16 jmc 1971: .Fa size
1972: bytes, the last two bytes of which must be
1973: .Dv YY_END_OF_BUFFER_CHAR
1974: .Pq ASCII NUL .
1975: These last two bytes are not scanned; thus, scanning consists of
1976: base[0] through base[size-2], inclusive.
1977: .Pp
1978: If
1979: .Fa base
1980: is not set up in this manner
1981: (i.e., forget the final two
1982: .Dv YY_END_OF_BUFFER_CHAR
1.1 deraadt 1983: bytes), then
1.16 jmc 1984: .Fn yy_scan_buffer
1.1 deraadt 1985: returns a nil pointer instead of creating a new input buffer.
1.16 jmc 1986: .Pp
1.1 deraadt 1987: The type
1.16 jmc 1988: .Fa yy_size_t
1989: is an integral type which can be cast to an integer expression
1.1 deraadt 1990: reflecting the size of the buffer.
1.16 jmc 1991: .El
1992: .Sh END-OF-FILE RULES
1993: The special rule
1994: .Qq Aq Aq EOF
1995: indicates actions which are to be taken when an end-of-file is encountered and
1996: .Fn yywrap
1997: returns non-zero
1998: .Pq i.e., indicates no further files to process .
1999: The action must finish by doing one of four things:
2000: .Bl -dash
2001: .It
2002: Assigning
2003: .Em yyin
2004: to a new input file
2005: (in previous versions of
2006: .Nm ,
2007: after doing the assignment, it was necessary to call the special action
2008: .Dv YY_NEW_FILE ;
2009: this is no longer necessary).
2010: .It
2011: Executing a
2012: .Em return
2013: statement.
2014: .It
2015: Executing the special
2016: .Fn yyterminate
2017: action.
2018: .It
2019: Switching to a new buffer using
2020: .Fn yy_switch_to_buffer
1.1 deraadt 2021: as shown in the example above.
1.16 jmc 2022: .El
2023: .Pp
2024: .Aq Aq EOF
2025: rules may not be used with other patterns;
2026: they may only be qualified with a list of start conditions.
2027: If an unqualified
2028: .Aq Aq EOF
2029: rule is given, it applies to all start conditions which do not already have
2030: .Aq Aq EOF
2031: actions.
2032: To specify an
2033: .Aq Aq EOF
2034: rule for only the initial start condition, use
2035: .Pp
2036: .Dl <INITIAL><<EOF>>
2037: .Pp
1.1 deraadt 2038: These rules are useful for catching things like unclosed comments.
2039: An example:
1.16 jmc 2040: .Bd -literal -offset indent
2041: %x quote
2042: %%
2043:
2044: \&...other rules for dealing with quotes...
2045:
2046: <quote><<EOF>> {
2047: error("unterminated quote");
2048: yyterminate();
2049: }
2050: <<EOF>> {
2051: if (*++filelist)
2052: yyin = fopen(*filelist, "r");
2053: else
2054: yyterminate();
2055: }
2056: .Ed
2057: .Sh MISCELLANEOUS MACROS
1.1 deraadt 2058: The macro
1.16 jmc 2059: .Dv YY_USER_ACTION
1.1 deraadt 2060: can be defined to provide an action
1.16 jmc 2061: which is always executed prior to the matched rule's action.
2062: For example,
1.1 deraadt 2063: it could be #define'd to call a routine to convert yytext to lower-case.
2064: When
1.16 jmc 2065: .Dv YY_USER_ACTION
1.1 deraadt 2066: is invoked, the variable
1.16 jmc 2067: .Fa yy_act
2068: gives the number of the matched rule
2069: .Pq rules are numbered starting with 1 .
2070: For example, to profile how often each rule is matched,
2071: the following would do the trick:
2072: .Pp
2073: .Dl #define YY_USER_ACTION ++ctr[yy_act]
2074: .Pp
1.1 deraadt 2075: where
1.16 jmc 2076: .Fa ctr
2077: is an array to hold the counts for the different rules.
2078: Note that the macro
2079: .Dv YY_NUM_RULES
2080: gives the total number of rules
2081: (including the default rule, even if
2082: .Fl s
2083: is used),
1.1 deraadt 2084: so a correct declaration for
1.16 jmc 2085: .Fa ctr
1.1 deraadt 2086: is:
1.16 jmc 2087: .Pp
2088: .Dl int ctr[YY_NUM_RULES];
2089: .Pp
1.1 deraadt 2090: The macro
1.16 jmc 2091: .Dv YY_USER_INIT
1.1 deraadt 2092: may be defined to provide an action which is always executed before
1.16 jmc 2093: the first scan
2094: .Pq and before the scanner's internal initializations are done .
1.1 deraadt 2095: For example, it could be used to call a routine to read
2096: in a data table or open a logging file.
1.16 jmc 2097: .Pp
1.1 deraadt 2098: The macro
1.16 jmc 2099: .Dv yy_set_interactive(is_interactive)
1.1 deraadt 2100: can be used to control whether the current buffer is considered
1.16 jmc 2101: .Em interactive .
1.1 deraadt 2102: An interactive buffer is processed more slowly,
2103: but must be used when the scanner's input source is indeed
2104: interactive to avoid problems due to waiting to fill buffers
2105: (see the discussion of the
1.16 jmc 2106: .Fl I
2107: flag below).
2108: A non-zero value in the macro invocation marks the buffer as interactive,
2109: a zero value as non-interactive.
2110: Note that use of this macro overrides
2111: .Dq %option always-interactive
2112: or
2113: .Dq %option never-interactive
2114: (see
2115: .Sx OPTIONS
2116: below).
2117: .Fn yy_set_interactive
1.1 deraadt 2118: must be invoked prior to beginning to scan the buffer that is
1.16 jmc 2119: .Pq or is not
2120: to be considered interactive.
2121: .Pp
1.1 deraadt 2122: The macro
1.16 jmc 2123: .Dv yy_set_bol(at_bol)
1.1 deraadt 2124: can be used to control whether the current buffer's scanning
2125: context for the next token match is done as though at the
1.16 jmc 2126: beginning of a line.
2127: A non-zero macro argument makes rules anchored with
2128: .Sq ^
2129: active, while a zero argument makes
2130: .Sq ^
2131: rules inactive.
2132: .Pp
1.1 deraadt 2133: The macro
1.16 jmc 2134: .Dv YY_AT_BOL
2135: returns true if the next token scanned from the current buffer will have
2136: .Sq ^
2137: rules active, false otherwise.
2138: .Pp
1.1 deraadt 2139: In the generated scanner, the actions are all gathered in one large
2140: switch statement and separated using
1.16 jmc 2141: .Dv YY_BREAK ,
2142: which may be redefined.
2143: By default, it is simply a
2144: .Qq break ,
2145: to separate each rule's action from the following rules.
1.1 deraadt 2146: Redefining
1.16 jmc 2147: .Dv YY_BREAK
1.1 deraadt 2148: allows, for example, C++ users to
1.16 jmc 2149: .Dq #define YY_BREAK
2150: to do nothing
2151: (while being very careful that every rule ends with a
2152: .Qq break
2153: or a
2154: .Qq return ! )
2155: to avoid suffering from unreachable statement warnings where because a rule's
2156: action ends with
2157: .Dq return ,
2158: the
2159: .Dv YY_BREAK
1.1 deraadt 2160: is inaccessible.
1.16 jmc 2161: .Sh VALUES AVAILABLE TO THE USER
1.1 deraadt 2162: This section summarizes the various values available to the user
2163: in the rule actions.
1.16 jmc 2164: .Bl -tag -width Ds
2165: .It char *yytext
2166: Holds the text of the current token.
2167: It may be modified but not lengthened
2168: .Pq characters cannot be appended to the end .
2169: .Pp
1.1 deraadt 2170: If the special directive
1.16 jmc 2171: .Dq %array
1.1 deraadt 2172: appears in the first section of the scanner description, then
1.16 jmc 2173: .Fa yytext
1.1 deraadt 2174: is instead declared
1.16 jmc 2175: .Dq char yytext[YYLMAX] ,
1.1 deraadt 2176: where
1.16 jmc 2177: .Dv YYLMAX
2178: is a macro definition that can be redefined in the first section
2179: to change the default value
2180: .Pq generally 8KB .
2181: Using
2182: .Dq %array
1.1 deraadt 2183: results in somewhat slower scanners, but the value of
1.16 jmc 2184: .Fa yytext
1.1 deraadt 2185: becomes immune to calls to
1.16 jmc 2186: .Fn input
1.1 deraadt 2187: and
1.16 jmc 2188: .Fn unput ,
1.1 deraadt 2189: which potentially destroy its value when
1.16 jmc 2190: .Fa yytext
2191: is a character pointer.
2192: The opposite of
2193: .Dq %array
1.1 deraadt 2194: is
1.16 jmc 2195: .Dq %pointer ,
1.1 deraadt 2196: which is the default.
1.16 jmc 2197: .Pp
2198: .Dq %array
2199: cannot be used when generating C++ scanner classes
1.1 deraadt 2200: (the
1.16 jmc 2201: .Fl +
1.1 deraadt 2202: flag).
1.16 jmc 2203: .It int yyleng
2204: Holds the length of the current token.
2205: .It FILE *yyin
2206: Is the file which by default
2207: .Nm
2208: reads from.
2209: It may be redefined, but doing so only makes sense before
2210: scanning begins or after an
2211: .Dv EOF
2212: has been encountered.
2213: Changing it in the midst of scanning will have unexpected results since
2214: .Nm
1.1 deraadt 2215: buffers its input; use
1.16 jmc 2216: .Fn yyrestart
1.1 deraadt 2217: instead.
2218: Once scanning terminates because an end-of-file
1.16 jmc 2219: has been seen,
2220: .Fa yyin
2221: can be assigned as the new input file
2222: and the scanner can be called again to continue scanning.
2223: .It void yyrestart(FILE *new_file)
2224: May be called to point
2225: .Fa yyin
2226: at the new input file.
2227: The switch-over to the new file is immediate
2228: .Pq any previously buffered-up input is lost .
2229: Note that calling
2230: .Fn yyrestart
1.1 deraadt 2231: with
1.16 jmc 2232: .Fa yyin
1.1 deraadt 2233: as an argument thus throws away the current input buffer and continues
2234: scanning the same input file.
1.16 jmc 2235: .It FILE *yyout
2236: Is the file to which
2237: .Em ECHO
2238: actions are done.
2239: It can be reassigned by the user.
2240: .It YY_CURRENT_BUFFER
2241: Returns a
2242: .Dv YY_BUFFER_STATE
1.1 deraadt 2243: handle to the current buffer.
1.16 jmc 2244: .It YY_START
2245: Returns an integer value corresponding to the current start condition.
2246: This value can subsequently be used with
2247: .Em BEGIN
1.1 deraadt 2248: to return to that start condition.
1.16 jmc 2249: .El
2250: .Sh INTERFACING WITH YACC
1.1 deraadt 2251: One of the main uses of
1.16 jmc 2252: .Nm
1.1 deraadt 2253: is as a companion to the
1.16 jmc 2254: .Xr yacc 1
1.1 deraadt 2255: parser-generator.
1.16 jmc 2256: yacc parsers expect to call a routine named
2257: .Fn yylex
2258: to find the next input token.
2259: The routine is supposed to return the type of the next token
2260: as well as putting any associated value in the global
1.17 jmc 2261: .Fa yylval ,
2262: which is defined externally,
2263: and can be a union or any other complex data structure.
1.1 deraadt 2264: To use
1.16 jmc 2265: .Nm
2266: with yacc, one specifies the
2267: .Fl d
2268: option to yacc to instruct it to generate the file
2269: .Pa y.tab.h
1.1 deraadt 2270: containing definitions of all the
1.16 jmc 2271: .Dq %tokens
2272: appearing in the yacc input.
2273: This file is then included in the
2274: .Nm
2275: scanner.
2276: For example, if one of the tokens is
2277: .Qq TOK_NUMBER ,
1.1 deraadt 2278: part of the scanner might look like:
1.16 jmc 2279: .Bd -literal -offset indent
2280: %{
2281: #include "y.tab.h"
2282: %}
2283:
2284: %%
2285:
2286: [0-9]+ yylval = atoi(yytext); return TOK_NUMBER;
2287: .Ed
2288: .Sh OPTIONS
2289: .Nm
1.1 deraadt 2290: has the following options:
1.16 jmc 2291: .Bl -tag -width Ds
2292: .It Fl 7
2293: Instructs
2294: .Nm
2295: to generate a 7-bit scanner, i.e., one which can only recognize 7-bit
2296: characters in its input.
2297: The advantage of using
2298: .Fl 7
1.1 deraadt 2299: is that the scanner's tables can be up to half the size of those generated
2300: using the
1.16 jmc 2301: .Fl 8
2302: option
2303: .Pq see below .
2304: The disadvantage is that such scanners often hang
1.1 deraadt 2305: or crash if their input contains an 8-bit character.
1.16 jmc 2306: .Pp
2307: Note, however, that unless generating a scanner using the
2308: .Fl Cf
1.1 deraadt 2309: or
1.16 jmc 2310: .Fl CF
1.1 deraadt 2311: table compression options, use of
1.16 jmc 2312: .Fl 7
2313: will save only a small amount of table space,
2314: and make the scanner considerably less portable.
2315: .Nm flex Ns 's
2316: default behavior is to generate an 8-bit scanner unless
2317: .Fl Cf
2318: or
2319: .Fl CF
2320: is specified, in which case
2321: .Nm
2322: defaults to generating 7-bit scanners unless it was
2323: configured to generate 8-bit scanners
2324: (as will often be the case with non-USA sites).
2325: It is possible tell whether
2326: .Nm
2327: generated a 7-bit or an 8-bit scanner by inspecting the flag summary in the
2328: .Fl v
2329: output as described below.
2330: .Pp
2331: Note that if
2332: .Fl Cfe
2333: or
2334: .Fl CFe
2335: are used
2336: (the table compression options, but also using equivalence classes as
2337: discussed below),
2338: .Nm
2339: still defaults to generating an 8-bit scanner,
2340: since usually with these compression options full 8-bit tables
1.1 deraadt 2341: are not much more expensive than 7-bit tables.
1.16 jmc 2342: .It Fl 8
2343: Instructs
2344: .Nm
1.1 deraadt 2345: to generate an 8-bit scanner, i.e., one which can recognize 8-bit
1.16 jmc 2346: characters.
2347: This flag is only needed for scanners generated using
2348: .Fl Cf
1.1 deraadt 2349: or
1.16 jmc 2350: .Fl CF ,
2351: as otherwise
2352: .Nm
2353: defaults to generating an 8-bit scanner anyway.
2354: .Pp
1.1 deraadt 2355: See the discussion of
1.16 jmc 2356: .Fl 7
2357: above for
2358: .Nm flex Ns 's
2359: default behavior and the tradeoffs between 7-bit and 8-bit scanners.
2360: .It Fl B
2361: Instructs
2362: .Nm
2363: to generate a
2364: .Em batch
2365: scanner, the opposite of
2366: .Em interactive
2367: scanners generated by
2368: .Fl I
2369: .Pq see below .
2370: In general,
2371: .Fl B
2372: is used when the scanner will never be used interactively,
2373: and you want to squeeze a little more performance out of it.
2374: If the aim is instead to squeeze out a lot more performance,
2375: use the
2376: .Fl Cf
2377: or
2378: .Fl CF
2379: options
2380: .Pq discussed below ,
2381: which turn on
2382: .Fl B
2383: automatically anyway.
2384: .It Fl b
2385: Generate backing-up information to
2386: .Pa lex.backup .
2387: This is a list of scanner states which require backing up
2388: and the input characters on which they do so.
2389: By adding rules one can remove backing-up states.
2390: If all backing-up states are eliminated and
2391: .Fl Cf
2392: or
2393: .Fl CF
2394: is used, the generated scanner will run faster (see the
2395: .Fl p
2396: flag).
2397: Only users who wish to squeeze every last cycle out of their
2398: scanners need worry about this option.
2399: (See the section on
2400: .Sx PERFORMANCE CONSIDERATIONS
2401: below.)
2402: .It Fl C Ns Op Cm aeFfmr
2403: Controls the degree of table compression and, more generally, trade-offs
1.1 deraadt 2404: between small scanners and fast scanners.
1.16 jmc 2405: .Bl -tag -width Ds
2406: .It Fl Ca
2407: Instructs
2408: .Nm
2409: to trade off larger tables in the generated scanner for faster performance
2410: because the elements of the tables are better aligned for memory access
2411: and computation.
2412: On some
2413: .Tn RISC
2414: architectures, fetching and manipulating longwords is more efficient
2415: than with smaller-sized units such as shortwords.
2416: This option can double the size of the tables used by the scanner.
2417: .It Fl Ce
2418: Directs
2419: .Nm
1.1 deraadt 2420: to construct
1.16 jmc 2421: .Em equivalence classes ,
2422: i.e., sets of characters which have identical lexical properties
2423: (for example, if the only appearance of digits in the
2424: .Nm
1.1 deraadt 2425: input is in the character class
1.16 jmc 2426: .Qq [0-9]
2427: then the digits
2428: .Sq 0 ,
2429: .Sq 1 ,
2430: .Sq ... ,
2431: .Sq 9
2432: will all be put in the same equivalence class).
2433: Equivalence classes usually give dramatic reductions in the final
2434: table/object file sizes
2435: .Pq typically a factor of 2\-5
2436: and are pretty cheap performance-wise
2437: .Pq one array look-up per character scanned .
2438: .It Fl CF
2439: Specifies that the alternate fast scanner representation
2440: (described below under the
2441: .Fl F
2442: option)
2443: should be used.
2444: This option cannot be used with
2445: .Fl + .
2446: .It Fl Cf
2447: Specifies that the
2448: .Em full
2449: scanner tables should be generated \-
2450: .Nm
2451: should not compress the tables by taking advantage of
2452: similar transition functions for different states.
2453: .It Fl \&Cm
2454: Directs
2455: .Nm
1.1 deraadt 2456: to construct
1.16 jmc 2457: .Em meta-equivalence classes ,
2458: which are sets of equivalence classes
2459: (or characters, if equivalence classes are not being used)
2460: that are commonly used together.
2461: Meta-equivalence classes are often a big win when using compressed tables,
2462: but they have a moderate performance impact
2463: (one or two
2464: .Qq if
2465: tests and one array look-up per character scanned).
2466: .It Fl Cr
2467: Causes the generated scanner to
2468: .Em bypass
2469: use of the standard I/O library
2470: .Pq stdio
2471: for input.
2472: Instead of calling
2473: .Xr fread 3
1.1 deraadt 2474: or
1.16 jmc 2475: .Xr getc 3 ,
1.1 deraadt 2476: the scanner will use the
1.16 jmc 2477: .Xr read 2
2478: system call,
2479: resulting in a performance gain which varies from system to system,
2480: but in general is probably negligible unless
2481: .Fl Cf
1.1 deraadt 2482: or
1.16 jmc 2483: .Fl CF
2484: are being used.
1.1 deraadt 2485: Using
1.16 jmc 2486: .Fl Cr
2487: can cause strange behavior if, for example, reading from
2488: .Fa yyin
2489: using stdio prior to calling the scanner
2490: (because the scanner will miss whatever text previous reads left
2491: in the stdio input buffer).
2492: .Pp
2493: .Fl Cr
2494: has no effect if
2495: .Dv YY_INPUT
2496: is defined
2497: (see
2498: .Sx THE GENERATED SCANNER
2499: above).
2500: .El
2501: .Pp
1.1 deraadt 2502: A lone
1.16 jmc 2503: .Fl C
1.1 deraadt 2504: specifies that the scanner tables should be compressed but neither
2505: equivalence classes nor meta-equivalence classes should be used.
1.16 jmc 2506: .Pp
1.1 deraadt 2507: The options
1.16 jmc 2508: .Fl Cf
1.1 deraadt 2509: or
1.16 jmc 2510: .Fl CF
1.1 deraadt 2511: and
1.16 jmc 2512: .Fl \&Cm
2513: do not make sense together \- there is no opportunity for meta-equivalence
2514: classes if the table is not being compressed.
2515: Otherwise the options may be freely mixed, and are cumulative.
2516: .Pp
1.1 deraadt 2517: The default setting is
1.16 jmc 2518: .Fl Cem
1.1 deraadt 2519: which specifies that
1.16 jmc 2520: .Nm
2521: should generate equivalence classes and meta-equivalence classes.
2522: This setting provides the highest degree of table compression.
2523: It is possible to trade off faster-executing scanners at the cost of
2524: larger tables with the following generally being true:
2525: .Bd -unfilled -offset indent
2526: slowest & smallest
2527: -Cem
2528: -Cm
2529: -Ce
2530: -C
2531: -C{f,F}e
2532: -C{f,F}
2533: -C{f,F}a
2534: fastest & largest
2535: .Ed
2536: .Pp
1.1 deraadt 2537: Note that scanners with the smallest tables are usually generated and
1.16 jmc 2538: compiled the quickest,
2539: so during development the default is usually best,
2540: maximal compression.
2541: .Pp
2542: .Fl Cfe
2543: is often a good compromise between speed and size for production scanners.
2544: .It Fl c
2545: A do-nothing, deprecated option included for
2546: .Tn POSIX
2547: compliance.
2548: .It Fl d
2549: Makes the generated scanner run in debug mode.
2550: Whenever a pattern is recognized and the global
2551: .Fa yy_flex_debug
2552: is non-zero
2553: .Pq which is the default ,
2554: the scanner will write to stderr a line of the form:
2555: .Pp
2556: .D1 --accepting rule at line 53 ("the matched text")
2557: .Pp
2558: The line number refers to the location of the rule in the file
2559: defining the scanner
2560: (i.e., the file that was fed to
2561: .Nm ) .
2562: Messages are also generated when the scanner backs up,
2563: accepts the default rule,
2564: reaches the end of its input buffer
2565: (or encounters a NUL;
2566: at this point, the two look the same as far as the scanner's concerned),
2567: or reaches an end-of-file.
2568: .It Fl F
2569: Specifies that the fast scanner table representation should be used
2570: .Pq and stdio bypassed .
2571: This representation is about as fast as the full table representation
2572: .Pq Fl f ,
2573: and for some sets of patterns will be considerably smaller
2574: .Pq and for others, larger .
2575: In general, if the pattern set contains both
2576: .Qq keywords
2577: and a catch-all,
2578: .Qq identifier
2579: rule, such as in the set:
2580: .Bd -unfilled -offset indent
2581: "case" return TOK_CASE;
2582: "switch" return TOK_SWITCH;
2583: \&...
2584: "default" return TOK_DEFAULT;
2585: [a-z]+ return TOK_ID;
2586: .Ed
2587: .Pp
2588: then it's better to use the full table representation.
2589: If only the
2590: .Qq identifier
2591: rule is present and a hash table or some such is used to detect the keywords,
2592: it's better to use
2593: .Fl F .
2594: .Pp
2595: This option is equivalent to
2596: .Fl CFr
2597: .Pq see above .
2598: It cannot be used with
2599: .Fl + .
2600: .It Fl f
2601: Specifies
2602: .Em fast scanner .
2603: No table compression is done and stdio is bypassed.
2604: The result is large but fast.
2605: This option is equivalent to
2606: .Fl Cfr
2607: .Pq see above .
2608: .It Fl h
2609: Generates a help summary of
2610: .Nm flex Ns 's
2611: options to stdout and then exits.
2612: .Fl ?\&
2613: and
2614: .Fl Fl help
2615: are synonyms for
2616: .Fl h .
2617: .It Fl I
2618: Instructs
2619: .Nm
2620: to generate an
2621: .Em interactive
2622: scanner.
2623: An interactive scanner is one that only looks ahead to decide
2624: what token has been matched if it absolutely must.
2625: It turns out that always looking one extra character ahead,
2626: even if the scanner has already seen enough text
2627: to disambiguate the current token, is a bit faster than
2628: only looking ahead when necessary.
2629: But scanners that always look ahead give dreadful interactive performance;
2630: for example, when a user types a newline,
2631: it is not recognized as a newline token until they enter
2632: .Em another
2633: token, which often means typing in another whole line.
2634: .Pp
2635: .Nm
2636: scanners default to
2637: .Em interactive
2638: unless
2639: .Fl Cf
2640: or
2641: .Fl CF
2642: table-compression options are specified
2643: .Pq see above .
2644: That's because if high-performance is most important,
2645: one of these options should be used,
2646: so if they weren't,
2647: .Nm
1.24 ! sobrado 2648: assumes it is preferable to trade off a bit of run-time performance for
1.16 jmc 2649: intuitive interactive behavior.
2650: Note also that
2651: .Fl I
2652: cannot be used in conjunction with
2653: .Fl Cf
2654: or
2655: .Fl CF .
2656: Thus, this option is not really needed; it is on by default for all those
2657: cases in which it is allowed.
2658: .Pp
2659: A scanner can be forced to not be interactive by using
2660: .Fl B
2661: .Pq see above .
2662: .It Fl i
2663: Instructs
2664: .Nm
2665: to generate a case-insensitive scanner.
2666: The case of letters given in the
2667: .Nm
2668: input patterns will be ignored,
2669: and tokens in the input will be matched regardless of case.
2670: The matched text given in
2671: .Fa yytext
2672: will have the preserved case
2673: .Pq i.e., it will not be folded .
2674: .It Fl L
2675: Instructs
2676: .Nm
2677: not to generate
2678: .Dq #line
2679: directives.
2680: Without this option,
2681: .Nm
2682: peppers the generated scanner with #line directives so error messages
2683: in the actions will be correctly located with respect to either the original
2684: .Nm
2685: input file
2686: (if the errors are due to code in the input file),
2687: or
2688: .Pa lex.yy.c
2689: (if the errors are
2690: .Nm flex Ns 's
2691: fault \- these sorts of errors should be reported to the email address
2692: given below).
2693: .It Fl l
2694: Turns on maximum compatibility with the original AT&T
2695: .Nm lex
2696: implementation.
2697: Note that this does not mean full compatibility.
2698: Use of this option costs a considerable amount of performance,
2699: and it cannot be used with the
2700: .Fl + , f , F , Cf ,
2701: or
2702: .Fl CF
2703: options.
2704: For details on the compatibilities it provides, see the section
2705: .Sx INCOMPATIBILITIES WITH LEX AND POSIX
2706: below.
2707: This option also results in the name
2708: .Dv YY_FLEX_LEX_COMPAT
2709: being #define'd in the generated scanner.
2710: .It Fl n
2711: Another do-nothing, deprecated option included only for
2712: .Tn POSIX
2713: compliance.
2714: .It Fl o Ns Ar output
2715: Directs
2716: .Nm
2717: to write the scanner to the file
2718: .Ar output
1.1 deraadt 2719: instead of
1.16 jmc 2720: .Pa lex.yy.c .
2721: If
2722: .Fl o
2723: is combined with the
2724: .Fl t
2725: option, then the scanner is written to stdout but its
2726: .Dq #line
2727: directives
2728: (see the
2729: .Fl L
2730: option above)
2731: refer to the file
2732: .Ar output .
2733: .It Fl P Ns Ar prefix
2734: Changes the default
2735: .Qq yy
1.1 deraadt 2736: prefix used by
1.16 jmc 2737: .Nm
1.6 aaron 2738: for all globally visible variable and function names to instead be
1.16 jmc 2739: .Ar prefix .
1.1 deraadt 2740: For example,
1.16 jmc 2741: .Fl P Ns Ar foo
1.1 deraadt 2742: changes the name of
1.16 jmc 2743: .Fa yytext
1.1 deraadt 2744: to
1.16 jmc 2745: .Fa footext .
1.1 deraadt 2746: It also changes the name of the default output file from
1.16 jmc 2747: .Pa lex.yy.c
1.1 deraadt 2748: to
1.16 jmc 2749: .Pa lex.foo.c .
1.1 deraadt 2750: Here are all of the names affected:
1.16 jmc 2751: .Bd -unfilled -offset indent
2752: yy_create_buffer
2753: yy_delete_buffer
2754: yy_flex_debug
2755: yy_init_buffer
2756: yy_flush_buffer
2757: yy_load_buffer_state
2758: yy_switch_to_buffer
2759: yyin
2760: yyleng
2761: yylex
2762: yylineno
2763: yyout
2764: yyrestart
2765: yytext
2766: yywrap
2767: .Ed
2768: .Pp
2769: (If using a C++ scanner, then only
2770: .Fa yywrap
1.1 deraadt 2771: and
1.16 jmc 2772: .Fa yyFlexLexer
1.1 deraadt 2773: are affected.)
1.16 jmc 2774: Within the scanner itself, it is still possible to refer to the global variables
1.1 deraadt 2775: and functions using either version of their name; but externally, they
2776: have the modified name.
1.16 jmc 2777: .Pp
2778: This option allows multiple
2779: .Nm
2780: programs to be easily linked together into the same executable.
2781: Note, though, that using this option also renames
2782: .Fn yywrap ,
2783: so now either an
2784: .Pq appropriately named
2785: version of the routine for the scanner must be supplied, or
2786: .Dq %option noyywrap
2787: must be used, as linking with
2788: .Fl lfl
2789: no longer provides one by default.
2790: .It Fl p
2791: Generates a performance report to stderr.
2792: The report consists of comments regarding features of the
2793: .Nm
2794: input file which will cause a serious loss of performance in the resulting
2795: scanner.
2796: If the flag is specified twice,
2797: comments regarding features that lead to minor performance losses
2798: will also be reported>
2799: .Pp
2800: Note that the use of
2801: .Em REJECT ,
2802: .Dq %option yylineno ,
2803: and variable trailing context
2804: (see the
2805: .Sx BUGS
2806: section below)
2807: entails a substantial performance penalty; use of
2808: .Fn yymore ,
2809: the
2810: .Sq ^
2811: operator, and the
2812: .Fl I
2813: flag entail minor performance penalties.
2814: .It Fl S Ns Ar skeleton
2815: Overrides the default skeleton file from which
2816: .Nm
2817: constructs its scanners.
2818: This option is needed only for
2819: .Nm
1.1 deraadt 2820: maintenance or development.
1.16 jmc 2821: .It Fl s
2822: Causes the default rule
2823: .Pq that unmatched scanner input is echoed to stdout
2824: to be suppressed.
2825: If the scanner encounters input that does not
2826: match any of its rules, it aborts with an error.
2827: This option is useful for finding holes in a scanner's rule set.
2828: .It Fl T
2829: Makes
2830: .Nm
2831: run in
2832: .Em trace
2833: mode.
2834: It will generate a lot of messages to stderr concerning
2835: the form of the input and the resultant non-deterministic and deterministic
2836: finite automata.
2837: This option is mostly for use in maintaining
2838: .Nm .
2839: .It Fl t
2840: Instructs
2841: .Nm
2842: to write the scanner it generates to standard output instead of
2843: .Pa lex.yy.c .
2844: .It Fl V
2845: Prints the version number to stdout and exits.
2846: .Fl Fl version
2847: is a synonym for
2848: .Fl V .
2849: .It Fl v
2850: Specifies that
2851: .Nm
2852: should write to stderr
2853: a summary of statistics regarding the scanner it generates.
2854: Most of the statistics are meaningless to the casual
2855: .Nm
2856: user, but the first line identifies the version of
2857: .Nm
2858: (same as reported by
2859: .Fl V ) ,
2860: and the next line the flags used when generating the scanner,
2861: including those that are on by default.
2862: .It Fl w
2863: Suppresses warning messages.
2864: .It Fl +
2865: Specifies that
2866: .Nm
2867: should generate a C++ scanner class.
2868: See the section on
2869: .Sx GENERATING C++ SCANNERS
2870: below for details.
2871: .El
2872: .Pp
2873: .Nm
1.1 deraadt 2874: also provides a mechanism for controlling options within the
1.16 jmc 2875: scanner specification itself, rather than from the
2876: .Nm
2877: command-line.
1.1 deraadt 2878: This is done by including
1.16 jmc 2879: .Dq %option
1.1 deraadt 2880: directives in the first section of the scanner specification.
1.16 jmc 2881: Multiple options can be specified with a single
2882: .Dq %option
2883: directive, and multiple directives in the first section of the
2884: .Nm
2885: input file.
2886: .Pp
2887: Most options are given simply as names, optionally preceded by the word
2888: .Qq no
2889: .Pq with no intervening whitespace
2890: to negate their meaning.
2891: A number are equivalent to
2892: .Nm
2893: flags or their negation:
2894: .Bd -unfilled -offset indent
2895: 7bit -7 option
2896: 8bit -8 option
2897: align -Ca option
2898: backup -b option
2899: batch -B option
2900: c++ -+ option
2901:
2902: caseful or
2903: case-sensitive opposite of -i (default)
2904:
2905: case-insensitive or
2906: caseless -i option
2907:
2908: debug -d option
2909: default opposite of -s option
2910: ecs -Ce option
2911: fast -F option
2912: full -f option
2913: interactive -I option
2914: lex-compat -l option
2915: meta-ecs -Cm option
2916: perf-report -p option
2917: read -Cr option
2918: stdout -t option
2919: verbose -v option
2920: warn opposite of -w option
2921: (use "%option nowarn" for -w)
2922:
2923: array equivalent to "%array"
2924: pointer equivalent to "%pointer" (default)
2925: .Ed
2926: .Pp
2927: Some %option's provide features otherwise not available:
2928: .Bl -tag -width Ds
2929: .It always-interactive
2930: Instructs
2931: .Nm
2932: to generate a scanner which always considers its input
2933: .Qq interactive .
2934: Normally, on each new input file the scanner calls
2935: .Fn isatty
2936: in an attempt to determine whether the scanner's input source is interactive
2937: and thus should be read a character at a time.
2938: When this option is used, however, no such call is made.
2939: .It main
2940: Directs
2941: .Nm
2942: to provide a default
2943: .Fn main
1.1 deraadt 2944: program for the scanner, which simply calls
1.16 jmc 2945: .Fn yylex .
1.1 deraadt 2946: This option implies
1.16 jmc 2947: .Dq noyywrap
2948: .Pq see below .
2949: .It never-interactive
2950: Instructs
2951: .Nm
2952: to generate a scanner which never considers its input
2953: .Qq interactive
2954: (again, no call made to
2955: .Fn isatty ) .
1.1 deraadt 2956: This is the opposite of
1.16 jmc 2957: .Dq always-interactive .
2958: .It stack
2959: Enables the use of start condition stacks
2960: (see
2961: .Sx START CONDITIONS
2962: above).
2963: .It stdinit
2964: If set (i.e.,
2965: .Dq %option stdinit ) ,
1.1 deraadt 2966: initializes
1.16 jmc 2967: .Fa yyin
1.1 deraadt 2968: and
1.16 jmc 2969: .Fa yyout
2970: to stdin and stdout, instead of the default of
2971: .Dq nil .
1.1 deraadt 2972: Some existing
1.16 jmc 2973: .Nm lex
2974: programs depend on this behavior, even though it is not compliant with ANSI C,
2975: which does not require stdin and stdout to be compile-time constant.
2976: .It yylineno
2977: Directs
2978: .Nm
1.1 deraadt 2979: to generate a scanner that maintains the number of the current line
2980: read from its input in the global variable
1.16 jmc 2981: .Fa yylineno .
1.1 deraadt 2982: This option is implied by
1.16 jmc 2983: .Dq %option lex-compat .
2984: .It yywrap
2985: If unset (i.e.,
2986: .Dq %option noyywrap ) ,
1.1 deraadt 2987: makes the scanner not call
1.16 jmc 2988: .Fn yywrap
2989: upon an end-of-file, but simply assume that there are no more files to scan
2990: (until the user points
2991: .Fa yyin
1.1 deraadt 2992: at a new file and calls
1.16 jmc 2993: .Fn yylex
1.1 deraadt 2994: again).
1.16 jmc 2995: .El
2996: .Pp
2997: .Nm
2998: scans rule actions to determine whether the
2999: .Em REJECT
3000: or
3001: .Fn yymore
3002: features are being used.
3003: The
3004: .Dq reject
1.1 deraadt 3005: and
1.16 jmc 3006: .Dq yymore
3007: options are available to override its decision as to whether to use the
1.1 deraadt 3008: options, either by setting them (e.g.,
1.16 jmc 3009: .Dq %option reject )
3010: to indicate the feature is indeed used,
3011: or unsetting them to indicate it actually is not used
1.1 deraadt 3012: (e.g.,
1.16 jmc 3013: .Dq %option noyymore ) .
3014: .Pp
3015: Three options take string-delimited values, offset with
3016: .Sq = :
3017: .Pp
3018: .D1 %option outfile="ABC"
3019: .Pp
1.1 deraadt 3020: is equivalent to
1.16 jmc 3021: .Fl o Ns Ar ABC ,
1.1 deraadt 3022: and
1.16 jmc 3023: .Pp
3024: .D1 %option prefix="XYZ"
3025: .Pp
1.1 deraadt 3026: is equivalent to
1.16 jmc 3027: .Fl P Ns Ar XYZ .
1.1 deraadt 3028: Finally,
1.16 jmc 3029: .Pp
3030: .D1 %option yyclass="foo"
3031: .Pp
3032: only applies when generating a C++ scanner
3033: .Pf ( Fl +
3034: option).
3035: It informs
3036: .Nm
3037: that
3038: .Dq foo
3039: has been derived as a subclass of yyFlexLexer, so
3040: .Nm
3041: will place actions in the member function
3042: .Dq foo::yylex()
1.1 deraadt 3043: instead of
1.16 jmc 3044: .Dq yyFlexLexer::yylex() .
1.1 deraadt 3045: It also generates a
1.16 jmc 3046: .Dq yyFlexLexer::yylex()
1.1 deraadt 3047: member function that emits a run-time error (by invoking
1.16 jmc 3048: .Dq yyFlexLexer::LexerError() )
1.1 deraadt 3049: if called.
1.16 jmc 3050: See
3051: .Sx GENERATING C++ SCANNERS ,
3052: below, for additional information.
3053: .Pp
3054: A number of options are available for
3055: .Xr lint 1
3056: purists who want to suppress the appearance of unneeded routines
3057: in the generated scanner.
3058: Each of the following, if unset
1.1 deraadt 3059: (e.g.,
1.16 jmc 3060: .Dq %option nounput ) ,
3061: results in the corresponding routine not appearing in the generated scanner:
3062: .Bd -unfilled -offset indent
3063: input, unput
3064: yy_push_state, yy_pop_state, yy_top_state
3065: yy_scan_buffer, yy_scan_bytes, yy_scan_string
3066: .Ed
3067: .Pp
1.1 deraadt 3068: (though
1.16 jmc 3069: .Fn yy_push_state
3070: and friends won't appear anyway unless
3071: .Dq %option stack
3072: is being used).
3073: .Sh PERFORMANCE CONSIDERATIONS
1.1 deraadt 3074: The main design goal of
1.16 jmc 3075: .Nm
3076: is that it generate high-performance scanners.
3077: It has been optimized for dealing well with large sets of rules.
3078: Aside from the effects on scanner speed of the table compression
3079: .Fl C
1.1 deraadt 3080: options outlined above,
1.16 jmc 3081: there are a number of options/actions which degrade performance.
3082: These are, from most expensive to least:
3083: .Bd -unfilled -offset indent
3084: REJECT
3085: %option yylineno
3086: arbitrary trailing context
3087:
3088: pattern sets that require backing up
3089: %array
3090: %option interactive
3091: %option always-interactive
3092:
3093: \&'^' beginning-of-line operator
3094: yymore()
3095: .Ed
3096: .Pp
3097: with the first three all being quite expensive
3098: and the last two being quite cheap.
3099: Note also that
3100: .Fn unput
3101: is implemented as a routine call that potentially does quite a bit of work,
3102: while
3103: .Fn yyless
3104: is a quite-cheap macro; so if just putting back some excess text,
3105: use
3106: .Fn yyless .
3107: .Pp
3108: .Em REJECT
1.1 deraadt 3109: should be avoided at all costs when performance is important.
3110: It is a particularly expensive option.
1.16 jmc 3111: .Pp
1.1 deraadt 3112: Getting rid of backing up is messy and often may be an enormous
1.16 jmc 3113: amount of work for a complicated scanner.
3114: In principal, one begins by using the
3115: .Fl b
1.1 deraadt 3116: flag to generate a
1.16 jmc 3117: .Pa lex.backup
3118: file.
3119: For example, on the input
3120: .Bd -literal -offset indent
3121: %%
3122: foo return TOK_KEYWORD;
3123: foobar return TOK_KEYWORD;
3124: .Ed
3125: .Pp
1.1 deraadt 3126: the file looks like:
1.16 jmc 3127: .Bd -literal -offset indent
3128: State #6 is non-accepting -
3129: associated rule line numbers:
3130: 2 3
3131: out-transitions: [ o ]
3132: jam-transitions: EOF [ \e001-n p-\e177 ]
3133:
3134: State #8 is non-accepting -
3135: associated rule line numbers:
3136: 3
3137: out-transitions: [ a ]
3138: jam-transitions: EOF [ \e001-` b-\e177 ]
3139:
3140: State #9 is non-accepting -
3141: associated rule line numbers:
3142: 3
3143: out-transitions: [ r ]
3144: jam-transitions: EOF [ \e001-q s-\e177 ]
3145:
3146: Compressed tables always back up.
3147: .Ed
3148: .Pp
1.1 deraadt 3149: The first few lines tell us that there's a scanner state in
1.16 jmc 3150: which it can make a transition on an
3151: .Sq o
3152: but not on any other character,
3153: and that in that state the currently scanned text does not match any rule.
3154: The state occurs when trying to match the rules found
1.1 deraadt 3155: at lines 2 and 3 in the input file.
1.16 jmc 3156: If the scanner is in that state and then reads something other than an
3157: .Sq o ,
3158: it will have to back up to find a rule which is matched.
3159: With a bit of headscratching one can see that this must be the
3160: state it's in when it has seen
3161: .Sq fo .
3162: When this has happened, if anything other than another
3163: .Sq o
3164: is seen, the scanner will have to back up to simply match the
3165: .Sq f
3166: .Pq by the default rule .
3167: .Pp
3168: The comment regarding State #8 indicates there's a problem when
3169: .Qq foob
3170: has been scanned.
3171: Indeed, on any character other than an
3172: .Sq a ,
3173: the scanner will have to back up to accept
3174: .Qq foo .
3175: Similarly, the comment for State #9 concerns when
3176: .Qq fooba
3177: has been scanned and an
3178: .Sq r
3179: does not follow.
3180: .Pp
1.1 deraadt 3181: The final comment reminds us that there's no point going to
1.16 jmc 3182: all the trouble of removing backing up from the rules unless we're using
3183: .Fl Cf
1.1 deraadt 3184: or
1.16 jmc 3185: .Fl CF ,
1.1 deraadt 3186: since there's no performance gain doing so with compressed scanners.
1.16 jmc 3187: .Pp
3188: The way to remove the backing up is to add
3189: .Qq error
3190: rules:
3191: .Bd -literal -offset indent
3192: %%
3193: foo return TOK_KEYWORD;
3194: foobar return TOK_KEYWORD;
3195:
3196: fooba |
3197: foob |
3198: fo {
3199: /* false alarm, not really a keyword */
3200: return TOK_ID;
3201: }
3202: .Ed
3203: .Pp
3204: Eliminating backing up among a list of keywords can also be done using a
3205: .Qq catch-all
3206: rule:
3207: .Bd -literal -offset indent
3208: %%
3209: foo return TOK_KEYWORD;
3210: foobar return TOK_KEYWORD;
3211:
3212: [a-z]+ return TOK_ID;
3213: .Ed
3214: .Pp
1.1 deraadt 3215: This is usually the best solution when appropriate.
1.16 jmc 3216: .Pp
1.1 deraadt 3217: Backing up messages tend to cascade.
1.16 jmc 3218: With a complicated set of rules it's not uncommon to get hundreds of messages.
3219: If one can decipher them, though,
3220: it often only takes a dozen or so rules to eliminate the backing up
3221: (though it's easy to make a mistake and have an error rule accidentally match
3222: a valid token; a possible future
3223: .Nm
1.1 deraadt 3224: feature will be to automatically add rules to eliminate backing up).
1.16 jmc 3225: .Pp
3226: It's important to keep in mind that the benefits of eliminating
3227: backing up are gained only if
3228: .Em every
3229: instance of backing up is eliminated.
3230: Leaving just one gains nothing.
3231: .Pp
3232: .Em Variable
3233: trailing context
3234: (where both the leading and trailing parts do not have a fixed length)
3235: entails almost the same performance loss as
3236: .Em REJECT
3237: .Pq i.e., substantial .
3238: So when possible a rule like:
3239: .Bd -literal -offset indent
3240: %%
3241: mouse|rat/(cat|dog) run();
3242: .Ed
3243: .Pp
1.1 deraadt 3244: is better written:
1.16 jmc 3245: .Bd -literal -offset indent
3246: %%
3247: mouse/cat|dog run();
3248: rat/cat|dog run();
3249: .Ed
3250: .Pp
1.1 deraadt 3251: or as
1.16 jmc 3252: .Bd -literal -offset indent
3253: %%
3254: mouse|rat/cat run();
3255: mouse|rat/dog run();
3256: .Ed
3257: .Pp
3258: Note that here the special
3259: .Sq |\&
3260: action does not provide any savings, and can even make things worse (see
3261: .Sx BUGS
3262: below).
3263: .Pp
1.1 deraadt 3264: Another area where the user can increase a scanner's performance
1.16 jmc 3265: .Pq and one that's easier to implement
3266: arises from the fact that the longer the tokens matched,
3267: the faster the scanner will run.
1.1 deraadt 3268: This is because with long tokens the processing of most input
1.16 jmc 3269: characters takes place in the
3270: .Pq short
3271: inner scanning loop, and does not often have to go through the additional work
3272: of setting up the scanning environment (e.g.,
3273: .Fa yytext )
3274: for the action.
3275: Recall the scanner for C comments:
3276: .Bd -literal -offset indent
3277: %x comment
3278: %%
3279: int line_num = 1;
3280:
3281: "/*" BEGIN(comment);
3282:
3283: <comment>[^*\en]*
3284: <comment>"*"+[^*/\en]*
3285: <comment>\en ++line_num;
3286: <comment>"*"+"/" BEGIN(INITIAL);
3287: .Ed
3288: .Pp
1.1 deraadt 3289: This could be sped up by writing it as:
1.16 jmc 3290: .Bd -literal -offset indent
3291: %x comment
3292: %%
3293: int line_num = 1;
3294:
3295: "/*" BEGIN(comment);
3296:
3297: <comment>[^*\en]*
3298: <comment>[^*\en]*\en ++line_num;
3299: <comment>"*"+[^*/\en]*
3300: <comment>"*"+[^*/\en]*\en ++line_num;
3301: <comment>"*"+"/" BEGIN(INITIAL);
3302: .Ed
3303: .Pp
3304: Now instead of each newline requiring the processing of another action,
3305: recognizing the newlines is
3306: .Qq distributed
3307: over the other rules to keep the matched text as long as possible.
3308: Note that adding rules does
3309: .Em not
3310: slow down the scanner!
3311: The speed of the scanner is independent of the number of rules or
3312: (modulo the considerations given at the beginning of this section)
3313: how complicated the rules are with regard to operators such as
3314: .Sq *
3315: and
3316: .Sq |\& .
3317: .Pp
3318: A final example in speeding up a scanner:
3319: scan through a file containing identifiers and keywords, one per line
3320: and with no other extraneous characters, and recognize all the keywords.
3321: A natural first approach is:
3322: .Bd -literal -offset indent
3323: %%
3324: asm |
3325: auto |
3326: break |
3327: \&... etc ...
3328: volatile |
3329: while /* it's a keyword */
3330:
3331: \&.|\en /* it's not a keyword */
3332: .Ed
3333: .Pp
1.1 deraadt 3334: To eliminate the back-tracking, introduce a catch-all rule:
1.16 jmc 3335: .Bd -literal -offset indent
3336: %%
3337: asm |
3338: auto |
3339: break |
3340: \&... etc ...
3341: volatile |
3342: while /* it's a keyword */
3343:
3344: [a-z]+ |
3345: \&.|\en /* it's not a keyword */
3346: .Ed
3347: .Pp
1.1 deraadt 3348: Now, if it's guaranteed that there's exactly one word per line,
3349: then we can reduce the total number of matches by a half by
1.16 jmc 3350: merging in the recognition of newlines with that of the other tokens:
3351: .Bd -literal -offset indent
3352: %%
3353: asm\en |
3354: auto\en |
3355: break\en |
3356: \&... etc ...
3357: volatile\en |
3358: while\en /* it's a keyword */
3359:
3360: [a-z]+\en |
3361: \&.|\en /* it's not a keyword */
3362: .Ed
3363: .Pp
3364: One has to be careful here,
3365: as we have now reintroduced backing up into the scanner.
3366: In particular, while we know that there will never be any characters
3367: in the input stream other than letters or newlines,
3368: .Nm
1.1 deraadt 3369: can't figure this out, and it will plan for possibly needing to back up
1.16 jmc 3370: when it has scanned a token like
3371: .Qq auto
3372: and then the next character is something other than a newline or a letter.
3373: Previously it would then just match the
3374: .Qq auto
3375: rule and be done, but now it has no
3376: .Qq auto
3377: rule, only an
3378: .Qq auto\en
3379: rule.
3380: To eliminate the possibility of backing up,
1.1 deraadt 3381: we could either duplicate all rules but without final newlines, or,
3382: since we never expect to encounter such an input and therefore don't
1.16 jmc 3383: how it's classified, we can introduce one more catch-all rule,
3384: this one which doesn't include a newline:
3385: .Bd -literal -offset indent
3386: %%
3387: asm\en |
3388: auto\en |
3389: break\en |
3390: \&... etc ...
3391: volatile\en |
3392: while\en /* it's a keyword */
3393:
3394: [a-z]+\en |
3395: [a-z]+ |
3396: \&.|\en /* it's not a keyword */
3397: .Ed
3398: .Pp
1.1 deraadt 3399: Compiled with
1.16 jmc 3400: .Fl Cf ,
1.1 deraadt 3401: this is about as fast as one can get a
1.16 jmc 3402: .Nm
1.1 deraadt 3403: scanner to go for this particular problem.
1.16 jmc 3404: .Pp
1.1 deraadt 3405: A final note:
1.16 jmc 3406: .Nm
3407: is slow when matching NUL's,
3408: particularly when a token contains multiple NUL's.
3409: It's best to write rules which match short
1.1 deraadt 3410: amounts of text if it's anticipated that the text will often include NUL's.
1.16 jmc 3411: .Pp
1.1 deraadt 3412: Another final note regarding performance: as mentioned above in the section
1.16 jmc 3413: .Sx HOW THE INPUT IS MATCHED ,
3414: dynamically resizing
3415: .Fa yytext
1.1 deraadt 3416: to accommodate huge tokens is a slow process because it presently requires that
1.16 jmc 3417: the
3418: .Pq huge
3419: token be rescanned from the beginning.
3420: Thus if performance is vital, it is better to attempt to match
3421: .Qq large
3422: quantities of text but not
3423: .Qq huge
3424: quantities, where the cutoff between the two is at about 8K characters/token.
3425: .Sh GENERATING C++ SCANNERS
3426: .Nm
3427: provides two different ways to generate scanners for use with C++.
3428: The first way is to simply compile a scanner generated by
3429: .Nm
3430: using a C++ compiler instead of a C compiler.
3431: This should not generate any compilation errors
3432: (please report any found to the email address given in the
3433: .Sx AUTHORS
3434: section below).
3435: C++ code can then be used in rule actions instead of C code.
3436: Note that the default input source for scanners remains
3437: .Fa yyin ,
1.1 deraadt 3438: and default echoing is still done to
1.16 jmc 3439: .Fa yyout .
1.1 deraadt 3440: Both of these remain
1.16 jmc 3441: .Fa FILE *
3442: variables and not C++ streams.
3443: .Pp
3444: .Nm
3445: can also be used to generate a C++ scanner class, using the
3446: .Fl +
1.1 deraadt 3447: option (or, equivalently,
1.16 jmc 3448: .Dq %option c++ ) ,
3449: which is automatically specified if the name of the flex executable ends in a
3450: .Sq + ,
3451: such as
3452: .Nm flex++ .
3453: When using this option,
3454: .Nm
3455: defaults to generating the scanner to the file
3456: .Pa lex.yy.cc
1.1 deraadt 3457: instead of
1.16 jmc 3458: .Pa lex.yy.c .
1.1 deraadt 3459: The generated scanner includes the header file
1.16 jmc 3460: .Aq Pa g++/FlexLexer.h ,
1.1 deraadt 3461: which defines the interface to two C++ classes.
1.16 jmc 3462: .Pp
1.1 deraadt 3463: The first class,
1.16 jmc 3464: .Em FlexLexer ,
3465: provides an abstract base class defining the general scanner class interface.
3466: It provides the following member functions:
3467: .Bl -tag -width Ds
3468: .It const char* YYText()
3469: Returns the text of the most recently matched token, the equivalent of
3470: .Fa yytext .
3471: .It int YYLeng()
3472: Returns the length of the most recently matched token, the equivalent of
3473: .Fa yyleng .
3474: .It int lineno() const
3475: Returns the current input line number
1.1 deraadt 3476: (see
1.16 jmc 3477: .Dq %option yylineno ) ,
3478: or 1 if
3479: .Dq %option yylineno
1.1 deraadt 3480: was not used.
1.16 jmc 3481: .It void set_debug(int flag)
3482: Sets the debugging flag for the scanner, equivalent to assigning to
3483: .Fa yy_flex_debug
3484: (see the
3485: .Sx OPTIONS
3486: section above).
3487: Note that the scanner must be built using
3488: .Dq %option debug
1.1 deraadt 3489: to include debugging information in it.
1.16 jmc 3490: .It int debug() const
3491: Returns the current setting of the debugging flag.
3492: .El
3493: .Pp
1.1 deraadt 3494: Also provided are member functions equivalent to
1.16 jmc 3495: .Fn yy_switch_to_buffer ,
3496: .Fn yy_create_buffer
1.1 deraadt 3497: (though the first argument is an
1.18 espie 3498: .Fa std::istream*
1.1 deraadt 3499: object pointer and not a
1.16 jmc 3500: .Fa FILE* ) ,
3501: .Fn yy_flush_buffer ,
3502: .Fn yy_delete_buffer ,
1.1 deraadt 3503: and
1.16 jmc 3504: .Fn yyrestart
1.10 deraadt 3505: (again, the first argument is an
1.18 espie 3506: .Fa std::istream*
1.1 deraadt 3507: object pointer).
1.16 jmc 3508: .Pp
1.1 deraadt 3509: The second class defined in
1.16 jmc 3510: .Aq Pa g++/FlexLexer.h
1.1 deraadt 3511: is
1.16 jmc 3512: .Fa yyFlexLexer ,
1.1 deraadt 3513: which is derived from
1.16 jmc 3514: .Fa FlexLexer .
1.1 deraadt 3515: It defines the following additional member functions:
1.16 jmc 3516: .Bl -tag -width Ds
1.18 espie 3517: .It "yyFlexLexer(std::istream* arg_yyin = 0, std::ostream* arg_yyout = 0)"
1.16 jmc 3518: Constructs a
3519: .Fa yyFlexLexer
3520: object using the given streams for input and output.
3521: If not specified, the streams default to
3522: .Fa cin
1.1 deraadt 3523: and
1.16 jmc 3524: .Fa cout ,
1.1 deraadt 3525: respectively.
1.16 jmc 3526: .It virtual int yylex()
3527: Performs the same role as
3528: .Fn yylex
1.1 deraadt 3529: does for ordinary flex scanners: it scans the input stream, consuming
1.16 jmc 3530: tokens, until a rule's action returns a value.
3531: If subclass
3532: .Sq S
3533: is derived from
3534: .Fa yyFlexLexer ,
3535: in order to access the member functions and variables of
3536: .Sq S
1.1 deraadt 3537: inside
1.16 jmc 3538: .Fn yylex ,
3539: use
3540: .Dq %option yyclass="S"
1.1 deraadt 3541: to inform
1.16 jmc 3542: .Nm
3543: that the
3544: .Sq S
3545: subclass will be used instead of
3546: .Fa yyFlexLexer .
1.1 deraadt 3547: In this case, rather than generating
1.16 jmc 3548: .Dq yyFlexLexer::yylex() ,
3549: .Nm
1.1 deraadt 3550: generates
1.16 jmc 3551: .Dq S::yylex()
1.1 deraadt 3552: (and also generates a dummy
1.16 jmc 3553: .Dq yyFlexLexer::yylex()
1.1 deraadt 3554: that calls
1.16 jmc 3555: .Dq yyFlexLexer::LexerError()
1.1 deraadt 3556: if called).
1.18 espie 3557: .It "virtual void switch_streams(std::istream* new_in = 0, std::ostream* new_out = 0)"
1.16 jmc 3558: Reassigns
3559: .Fa yyin
1.1 deraadt 3560: to
1.16 jmc 3561: .Fa new_in
3562: .Pq if non-nil
1.1 deraadt 3563: and
1.16 jmc 3564: .Fa yyout
1.1 deraadt 3565: to
1.16 jmc 3566: .Fa new_out
3567: .Pq ditto ,
3568: deleting the previous input buffer if
3569: .Fa yyin
1.1 deraadt 3570: is reassigned.
1.18 espie 3571: .It int yylex(std::istream* new_in, std::ostream* new_out = 0)
1.16 jmc 3572: First switches the input streams via
3573: .Dq switch_streams(new_in, new_out)
1.1 deraadt 3574: and then returns the value of
1.16 jmc 3575: .Fn yylex .
3576: .El
3577: .Pp
1.1 deraadt 3578: In addition,
1.16 jmc 3579: .Fa yyFlexLexer
3580: defines the following protected virtual functions which can be redefined
1.1 deraadt 3581: in derived classes to tailor the scanner:
1.16 jmc 3582: .Bl -tag -width Ds
3583: .It virtual int LexerInput(char* buf, int max_size)
3584: Reads up to
3585: .Fa max_size
1.1 deraadt 3586: characters into
1.16 jmc 3587: .Fa buf
3588: and returns the number of characters read.
3589: To indicate end-of-input, return 0 characters.
3590: Note that
3591: .Qq interactive
3592: scanners (see the
3593: .Fl B
1.1 deraadt 3594: and
1.16 jmc 3595: .Fl I
1.1 deraadt 3596: flags) define the macro
1.16 jmc 3597: .Dv YY_INTERACTIVE .
3598: If
3599: .Fn LexerInput
3600: has been redefined, and it's necessary to take different actions depending on
3601: whether or not the scanner might be scanning an interactive input source,
3602: it's possible to test for the presence of this name via
3603: .Dq #ifdef .
3604: .It virtual void LexerOutput(const char* buf, int size)
3605: Writes out
3606: .Fa size
1.1 deraadt 3607: characters from the buffer
1.16 jmc 3608: .Fa buf ,
3609: which, while NUL-terminated, may also contain
3610: .Qq internal
3611: NUL's if the scanner's rules can match text with NUL's in them.
3612: .It virtual void LexerError(const char* msg)
3613: Reports a fatal error message.
3614: The default version of this function writes the message to the stream
3615: .Fa cerr
1.1 deraadt 3616: and exits.
1.16 jmc 3617: .El
3618: .Pp
1.1 deraadt 3619: Note that a
1.16 jmc 3620: .Fa yyFlexLexer
3621: object contains its entire scanning state.
3622: Thus such objects can be used to create reentrant scanners.
3623: Multiple instances of the same
3624: .Fa yyFlexLexer
3625: class can be instantiated, and multiple C++ scanner classes can be combined
1.1 deraadt 3626: in the same program using the
1.16 jmc 3627: .Fl P
1.1 deraadt 3628: option discussed above.
1.16 jmc 3629: .Pp
1.1 deraadt 3630: Finally, note that the
1.16 jmc 3631: .Dq %array
3632: feature is not available to C++ scanner classes;
3633: .Dq %pointer
3634: must be used
3635: .Pq the default .
3636: .Pp
1.1 deraadt 3637: Here is an example of a simple C++ scanner:
1.16 jmc 3638: .Bd -literal -offset indent
3639: // An example of using the flex C++ scanner class.
1.1 deraadt 3640:
1.16 jmc 3641: %{
3642: #include <errno.h>
3643: int mylineno = 0;
3644: %}
1.1 deraadt 3645:
1.16 jmc 3646: string \e"[^\en"]+\e"
1.1 deraadt 3647:
1.16 jmc 3648: ws [ \et]+
1.1 deraadt 3649:
1.16 jmc 3650: alpha [A-Za-z]
3651: dig [0-9]
3652: name ({alpha}|{dig}|\e$)({alpha}|{dig}|[_.\e-/$])*
3653: num1 [-+]?{dig}+\e.?([eE][-+]?{dig}+)?
3654: num2 [-+]?{dig}*\e.{dig}+([eE][-+]?{dig}+)?
3655: number {num1}|{num2}
1.1 deraadt 3656:
1.16 jmc 3657: %%
1.1 deraadt 3658:
1.16 jmc 3659: {ws} /* skip blanks and tabs */
1.1 deraadt 3660:
1.16 jmc 3661: "/*" {
3662: int c;
1.1 deraadt 3663:
1.16 jmc 3664: while ((c = yyinput()) != 0) {
3665: if(c == '\en')
1.1 deraadt 3666: ++mylineno;
1.16 jmc 3667: else if(c == '*') {
3668: if ((c = yyinput()) == '/')
1.1 deraadt 3669: break;
3670: else
3671: unput(c);
3672: }
1.16 jmc 3673: }
3674: }
1.1 deraadt 3675:
1.16 jmc 3676: {number} cout << "number " << YYText() << '\en';
1.1 deraadt 3677:
1.16 jmc 3678: \en mylineno++;
1.1 deraadt 3679:
1.16 jmc 3680: {name} cout << "name " << YYText() << '\en';
1.1 deraadt 3681:
1.16 jmc 3682: {string} cout << "string " << YYText() << '\en';
3683:
3684: %%
3685:
3686: int main(int /* argc */, char** /* argv */)
3687: {
3688: FlexLexer* lexer = new yyFlexLexer;
3689: while(lexer->yylex() != 0)
3690: ;
3691: return 0;
3692: }
3693: .Ed
3694: .Pp
3695: To create multiple
3696: .Pq different
3697: lexer classes, use the
3698: .Fl P
3699: flag
3700: (or the
3701: .Dq prefix=
3702: option)
3703: to rename each
3704: .Fa yyFlexLexer
1.1 deraadt 3705: to some other
1.16 jmc 3706: .Fa xxFlexLexer .
3707: .Aq Pa g++/FlexLexer.h
3708: can then be included in other sources once per lexer class, first renaming
3709: .Fa yyFlexLexer
1.1 deraadt 3710: as follows:
1.16 jmc 3711: .Bd -literal -offset indent
3712: #undef yyFlexLexer
3713: #define yyFlexLexer xxFlexLexer
3714: #include <g++/FlexLexer.h>
3715:
3716: #undef yyFlexLexer
3717: #define yyFlexLexer zzFlexLexer
3718: #include <g++/FlexLexer.h>
3719: .Ed
3720: .Pp
3721: If, for example,
3722: .Dq %option prefix="xx"
3723: is used for one scanner and
3724: .Dq %option prefix="zz"
3725: is used for the other.
3726: .Pp
3727: .Sy IMPORTANT :
3728: the present form of the scanning class is experimental
1.7 aaron 3729: and may change considerably between major releases.
1.16 jmc 3730: .Sh INCOMPATIBILITIES WITH LEX AND POSIX
3731: .Nm
1.1 deraadt 3732: is a rewrite of the AT&T Unix
1.16 jmc 3733: .Nm lex
3734: tool
3735: (the two implementations do not share any code, though),
3736: with some extensions and incompatibilities, both of which are of concern
3737: to those who wish to write scanners acceptable to either implementation.
3738: .Nm
3739: is fully compliant with the
3740: .Tn POSIX
3741: .Nm lex
1.1 deraadt 3742: specification, except that when using
1.16 jmc 3743: .Dq %pointer
3744: .Pq the default ,
3745: a call to
3746: .Fn unput
1.1 deraadt 3747: destroys the contents of
1.16 jmc 3748: .Fa yytext ,
3749: which is counter to the
3750: .Tn POSIX
3751: specification.
3752: .Pp
3753: In this section we discuss all of the known areas of incompatibility between
3754: .Nm ,
3755: AT&T
3756: .Nm lex ,
3757: and the
3758: .Tn POSIX
3759: specification.
3760: .Pp
3761: .Nm flex Ns 's
3762: .Fl l
1.1 deraadt 3763: option turns on maximum compatibility with the original AT&T
1.16 jmc 3764: .Nm lex
1.1 deraadt 3765: implementation, at the cost of a major loss in the generated scanner's
1.16 jmc 3766: performance.
3767: We note below which incompatibilities can be overcome using the
3768: .Fl l
1.1 deraadt 3769: option.
1.16 jmc 3770: .Pp
3771: .Nm
1.1 deraadt 3772: is fully compatible with
1.16 jmc 3773: .Nm lex
1.1 deraadt 3774: with the following exceptions:
1.16 jmc 3775: .Bl -dash
3776: .It
1.1 deraadt 3777: The undocumented
1.16 jmc 3778: .Nm lex
1.1 deraadt 3779: scanner internal variable
1.16 jmc 3780: .Fa yylineno
1.1 deraadt 3781: is not supported unless
1.16 jmc 3782: .Fl l
1.1 deraadt 3783: or
1.16 jmc 3784: .Dq %option yylineno
1.1 deraadt 3785: is used.
1.16 jmc 3786: .Pp
3787: .Fa yylineno
1.1 deraadt 3788: should be maintained on a per-buffer basis, rather than a per-scanner
1.16 jmc 3789: .Pq single global variable
3790: basis.
3791: .Pp
3792: .Fa yylineno
3793: is not part of the
3794: .Tn POSIX
3795: specification.
3796: .It
1.1 deraadt 3797: The
1.16 jmc 3798: .Fn input
1.1 deraadt 3799: routine is not redefinable, though it may be called to read characters
1.16 jmc 3800: following whatever has been matched by a rule.
3801: If
3802: .Fn input
3803: encounters an end-of-file, the normal
3804: .Fn yywrap
3805: processing is done.
3806: A
3807: .Dq real
3808: end-of-file is returned by
3809: .Fn input
1.1 deraadt 3810: as
1.16 jmc 3811: .Dv EOF .
3812: .Pp
1.1 deraadt 3813: Input is instead controlled by defining the
1.16 jmc 3814: .Dv YY_INPUT
1.1 deraadt 3815: macro.
1.16 jmc 3816: .Pp
1.1 deraadt 3817: The
1.16 jmc 3818: .Nm
1.1 deraadt 3819: restriction that
1.16 jmc 3820: .Fn input
3821: cannot be redefined is in accordance with the
3822: .Tn POSIX
3823: specification, which simply does not specify any way of controlling the
1.1 deraadt 3824: scanner's input other than by making an initial assignment to
1.16 jmc 3825: .Fa yyin .
3826: .It
1.1 deraadt 3827: The
1.16 jmc 3828: .Fn unput
3829: routine is not redefinable.
3830: This restriction is in accordance with
3831: .Tn POSIX .
3832: .It
3833: .Nm
1.1 deraadt 3834: scanners are not as reentrant as
1.16 jmc 3835: .Nm lex
3836: scanners.
3837: In particular, if a scanner is interactive and
3838: an interrupt handler long-jumps out of the scanner,
3839: and the scanner is subsequently called again,
3840: the following error message may be displayed:
3841: .Pp
3842: .D1 fatal flex scanner internal error--end of buffer missed
3843: .Pp
1.1 deraadt 3844: To reenter the scanner, first use
1.16 jmc 3845: .Pp
3846: .Dl yyrestart(yyin);
3847: .Pp
3848: Note that this call will throw away any buffered input;
3849: usually this isn't a problem with an interactive scanner.
3850: .Pp
3851: Also note that flex C++ scanner classes are reentrant,
3852: so if using C++ is an option , they should be used instead.
3853: See
3854: .Sx GENERATING C++ SCANNERS
3855: above for details.
3856: .It
3857: .Fn output
1.1 deraadt 3858: is not supported.
3859: Output from the
1.16 jmc 3860: .Em ECHO
1.1 deraadt 3861: macro is done to the file-pointer
1.16 jmc 3862: .Fa yyout
3863: .Pq default stdout .
3864: .Pp
3865: .Fn output
3866: is not part of the
3867: .Tn POSIX
3868: specification.
3869: .It
3870: .Nm lex
3871: does not support exclusive start conditions
3872: .Pq %x ,
3873: though they are in the
3874: .Tn POSIX
3875: specification.
3876: .It
1.1 deraadt 3877: When definitions are expanded,
1.16 jmc 3878: .Nm
1.1 deraadt 3879: encloses them in parentheses.
1.16 jmc 3880: With
3881: .Nm lex ,
3882: the following:
3883: .Bd -literal -offset indent
3884: NAME [A-Z][A-Z0-9]*
3885: %%
3886: foo{NAME}? printf("Found it\en");
3887: %%
3888: .Ed
3889: .Pp
3890: will not match the string
3891: .Qq foo
3892: because when the macro is expanded the rule is equivalent to
3893: .Qq foo[A-Z][A-Z0-9]*?
3894: and the precedence is such that the
3895: .Sq ?\&
3896: is associated with
3897: .Qq [A-Z0-9]* .
3898: With
3899: .Nm ,
1.1 deraadt 3900: the rule will be expanded to
1.16 jmc 3901: .Qq foo([A-Z][A-Z0-9]*)?
3902: and so the string
3903: .Qq foo
3904: will match.
3905: .Pp
1.1 deraadt 3906: Note that if the definition begins with
1.16 jmc 3907: .Sq ^
1.1 deraadt 3908: or ends with
1.16 jmc 3909: .Sq $
3910: then it is not expanded with parentheses, to allow these operators to appear in
3911: definitions without losing their special meanings.
3912: But the
3913: .Sq Aq s ,
3914: .Sq / ,
1.1 deraadt 3915: and
1.16 jmc 3916: .Aq Aq EOF
1.1 deraadt 3917: operators cannot be used in a
1.16 jmc 3918: .Nm
1.1 deraadt 3919: definition.
1.16 jmc 3920: .Pp
1.1 deraadt 3921: Using
1.16 jmc 3922: .Fl l
1.1 deraadt 3923: results in the
1.16 jmc 3924: .Nm lex
1.1 deraadt 3925: behavior of no parentheses around the definition.
1.16 jmc 3926: .Pp
3927: The
3928: .Tn POSIX
3929: specification is that the definition be enclosed in parentheses.
3930: .It
1.1 deraadt 3931: Some implementations of
1.16 jmc 3932: .Nm lex
3933: allow a rule's action to begin on a separate line,
3934: if the rule's pattern has trailing whitespace:
3935: .Bd -literal -offset indent
3936: %%
3937: foo|bar<space here>
3938: { foobar_action(); }
3939: .Ed
3940: .Pp
3941: .Nm
1.1 deraadt 3942: does not support this feature.
1.16 jmc 3943: .It
1.1 deraadt 3944: The
1.16 jmc 3945: .Nm lex
3946: .Sq %r
3947: .Pq generate a Ratfor scanner
3948: option is not supported.
3949: It is not part of the
3950: .Tn POSIX
3951: specification.
3952: .It
1.1 deraadt 3953: After a call to
1.16 jmc 3954: .Fn unput ,
3955: .Fa yytext
3956: is undefined until the next token is matched,
3957: unless the scanner was built using
3958: .Dq %array .
1.1 deraadt 3959: This is not the case with
1.16 jmc 3960: .Nm lex
3961: or the
3962: .Tn POSIX
3963: specification.
3964: The
3965: .Fl l
1.1 deraadt 3966: option does away with this incompatibility.
1.16 jmc 3967: .It
1.1 deraadt 3968: The precedence of the
1.16 jmc 3969: .Sq {}
3970: .Pq numeric range
3971: operator is different.
3972: .Nm lex
3973: interprets
3974: .Qq abc{1,3}
3975: as match one, two, or three occurrences of
3976: .Sq abc ,
3977: whereas
3978: .Nm
3979: interprets it as match
3980: .Sq ab
3981: followed by one, two, or three occurrences of
3982: .Sq c .
3983: The latter is in agreement with the
3984: .Tn POSIX
3985: specification.
3986: .It
1.1 deraadt 3987: The precedence of the
1.16 jmc 3988: .Sq ^
1.1 deraadt 3989: operator is different.
1.16 jmc 3990: .Nm lex
3991: interprets
3992: .Qq ^foo|bar
3993: as match either
3994: .Sq foo
3995: at the beginning of a line, or
3996: .Sq bar
3997: anywhere, whereas
3998: .Nm
3999: interprets it as match either
4000: .Sq foo
4001: or
4002: .Sq bar
4003: if they come at the beginning of a line.
4004: The latter is in agreement with the
4005: .Tn POSIX
4006: specification.
4007: .It
1.1 deraadt 4008: The special table-size declarations such as
1.16 jmc 4009: .Sq %a
1.1 deraadt 4010: supported by
1.16 jmc 4011: .Nm lex
1.1 deraadt 4012: are not required by
1.16 jmc 4013: .Nm
1.1 deraadt 4014: scanners;
1.16 jmc 4015: .Nm
1.1 deraadt 4016: ignores them.
1.16 jmc 4017: .It
1.1 deraadt 4018: The name
1.16 jmc 4019: .Dv FLEX_SCANNER
1.1 deraadt 4020: is #define'd so scanners may be written for use with either
1.16 jmc 4021: .Nm
1.1 deraadt 4022: or
1.16 jmc 4023: .Nm lex .
1.1 deraadt 4024: Scanners also include
1.16 jmc 4025: .Dv YY_FLEX_MAJOR_VERSION
1.1 deraadt 4026: and
1.16 jmc 4027: .Dv YY_FLEX_MINOR_VERSION
1.1 deraadt 4028: indicating which version of
1.16 jmc 4029: .Nm
1.1 deraadt 4030: generated the scanner
1.16 jmc 4031: (for example, for the 2.5 release, these defines would be 2 and 5,
1.1 deraadt 4032: respectively).
1.16 jmc 4033: .El
4034: .Pp
1.1 deraadt 4035: The following
1.16 jmc 4036: .Nm
1.1 deraadt 4037: features are not included in
1.16 jmc 4038: .Nm lex
4039: or the
4040: .Tn POSIX
4041: specification:
4042: .Bd -unfilled -offset indent
4043: C++ scanners
4044: %option
4045: start condition scopes
4046: start condition stacks
4047: interactive/non-interactive scanners
4048: yy_scan_string() and friends
4049: yyterminate()
4050: yy_set_interactive()
4051: yy_set_bol()
4052: YY_AT_BOL()
4053: <<EOF>>
4054: <*>
4055: YY_DECL
4056: YY_START
4057: YY_USER_ACTION
4058: YY_USER_INIT
4059: #line directives
4060: %{}'s around actions
4061: multiple actions on a line
4062: .Ed
4063: .Pp
4064: plus almost all of the
4065: .Nm
4066: flags.
1.1 deraadt 4067: The last feature in the list refers to the fact that with
1.16 jmc 4068: .Nm
4069: Multiple actions ican be placed on the same line,
4070: separated with semi-colons, while with
4071: .Nm lex ,
1.1 deraadt 4072: the following
1.16 jmc 4073: .Pp
4074: .Dl foo handle_foo(); ++num_foos_seen;
4075: .Pp
4076: is
4077: .Pq rather surprisingly
4078: truncated to
4079: .Pp
4080: .Dl foo handle_foo();
4081: .Pp
4082: .Nm
4083: does not truncate the action.
4084: Actions that are not enclosed in braces
4085: are simply terminated at the end of the line.
4086: .Sh FILES
4087: .Bl -tag -width "<g++/FlexLexer.h>"
4088: .It flex.skl
4089: Skeleton scanner.
4090: This file is only used when building flex, not when
4091: .Nm
4092: executes.
4093: .It lex.backup
4094: Backing-up information for the
4095: .Fl b
4096: flag (called
4097: .Pa lex.bck
4098: on some systems).
4099: .It lex.yy.c
4100: Generated scanner
4101: (called
4102: .Pa lexyy.c
4103: on some systems).
4104: .It lex.yy.cc
4105: Generated C++ scanner class, when using
4106: .Fl + .
4107: .It Aq g++/FlexLexer.h
4108: Header file defining the C++ scanner base class,
4109: .Fa FlexLexer ,
4110: and its derived class,
4111: .Fa yyFlexLexer .
4112: .It /usr/lib/libl.*
4113: .Nm
4114: libraries.
4115: The
4116: .Pa /usr/lib/libfl.*\&
4117: libraries are links to these.
4118: Scanners must be linked using either
4119: .Fl \&ll
4120: or
4121: .Fl lfl .
4122: .El
4123: .Sh DIAGNOSTICS
4124: .Bl -diag
4125: .It warning, rule cannot be matched
4126: Indicates that the given rule cannot be matched because it follows other rules
4127: that will always match the same text as it.
4128: For example, in the following
4129: .Dq foo
4130: cannot be matched because it comes after an identifier
4131: .Qq catch-all
4132: rule:
4133: .Bd -literal -offset indent
4134: [a-z]+ got_identifier();
4135: foo got_foo();
4136: .Ed
4137: .Pp
1.1 deraadt 4138: Using
1.16 jmc 4139: .Em REJECT
1.1 deraadt 4140: in a scanner suppresses this warning.
1.16 jmc 4141: .It "warning, \-s option given but default rule can be matched"
4142: Means that it is possible
4143: .Pq perhaps only in a particular start condition
4144: that the default rule
4145: .Pq match any single character
4146: is the only one that will match a particular input.
4147: Since
4148: .Fl s
1.1 deraadt 4149: was given, presumably this is not intended.
1.16 jmc 4150: .It reject_used_but_not_detected undefined
4151: .It yymore_used_but_not_detected undefined
4152: These errors can occur at compile time.
4153: They indicate that the scanner uses
4154: .Em REJECT
1.1 deraadt 4155: or
1.16 jmc 4156: .Fn yymore
1.1 deraadt 4157: but that
1.16 jmc 4158: .Nm
1.1 deraadt 4159: failed to notice the fact, meaning that
1.16 jmc 4160: .Nm
1.1 deraadt 4161: scanned the first two sections looking for occurrences of these actions
1.16 jmc 4162: and failed to find any, but somehow they snuck in
4163: .Pq via an #include file, for example .
4164: Use
4165: .Dq %option reject
4166: or
4167: .Dq %option yymore
4168: to indicate to
4169: .Nm
4170: that these features are really needed.
4171: .It flex scanner jammed
4172: A scanner compiled with
4173: .Fl s
4174: has encountered an input string which wasn't matched by any of its rules.
4175: This error can also occur due to internal problems.
4176: .It token too large, exceeds YYLMAX
4177: The scanner uses
4178: .Dq %array
1.1 deraadt 4179: and one of its rules matched a string longer than the
1.16 jmc 4180: .Dv YYLMAX
4181: constant
4182: .Pq 8K bytes by default .
4183: The value can be increased by #define'ing
4184: .Dv YYLMAX
4185: in the definitions section of
4186: .Nm
1.1 deraadt 4187: input.
1.16 jmc 4188: .It "scanner requires \-8 flag to use the character 'x'"
4189: The scanner specification includes recognizing the 8-bit character
4190: .Sq x
4191: and the
4192: .Fl 8
4193: flag was not specified, and defaulted to 7-bit because the
4194: .Fl Cf
4195: or
4196: .Fl CF
4197: table compression options were used.
4198: See the discussion of the
4199: .Fl 7
1.1 deraadt 4200: flag for details.
1.16 jmc 4201: .It flex scanner push-back overflow
4202: unput() was used to push back so much text that the scanner's buffer
4203: could not hold both the pushed-back text and the current token in
4204: .Fa yytext .
4205: Ideally the scanner should dynamically resize the buffer in this case,
4206: but at present it does not.
4207: .It "input buffer overflow, can't enlarge buffer because scanner uses REJECT"
4208: The scanner was working on matching an extremely large token and needed
4209: to expand the input buffer.
4210: This doesn't work with scanners that use
4211: .Em REJECT .
4212: .It "fatal flex scanner internal error--end of buffer missed"
1.1 deraadt 4213: This can occur in an scanner which is reentered after a long-jump
1.16 jmc 4214: has jumped out
4215: .Pq or over
4216: the scanner's activation frame.
4217: Before reentering the scanner, use:
4218: .Pp
4219: .Dl yyrestart(yyin);
4220: .Pp
1.1 deraadt 4221: or, as noted above, switch to using the C++ scanner class.
1.16 jmc 4222: .It "too many start conditions in <> construct!"
4223: More start conditions than exist were listed in a <> construct
4224: (so at least one of them must have been listed twice).
4225: .El
4226: .Sh SEE ALSO
4227: .Xr awk 1 ,
4228: .Xr lex 1 ,
4229: .Xr sed 1 ,
4230: .Xr yacc 1
4231: .Pp
1.19 jmc 4232: "Lex \- A Lexical Analyzer Generator",
4233: .Pa /usr/share/doc/psd/16.lex/ .
1.16 jmc 4234: .Rs
4235: .%A John Levine
4236: .%A Tony Mason
4237: .%A Doug Brown
4238: .%B Lex & Yacc
4239: .%I O'Reilly and Associates
4240: .%N 2nd edition
4241: .Re
4242: .Rs
4243: .%A Alfred Aho
4244: .%A Ravi Sethi
4245: .%A Jeffrey Ullman
4246: .%B Compilers: Principles, Techniques and Tools
4247: .%I Addison-Wesley
4248: .%D 1986
4249: .%O "Describes the pattern-matching techniques used by flex (deterministic finite automata)"
4250: .Re
1.23 jmc 4251: .Sh STANDARDS
4252: The
4253: .Nm lex
4254: utility is compliant with the
4255: .St -p1003.1-2008
4256: specification,
4257: though its presence is optional.
4258: .Pp
4259: The flags
4260: .Op Fl 78BbCcdFfhIiLloPpSsTVw+? ,
4261: .Op Fl -help ,
4262: and
4263: .Op Fl -version
4264: are extensions to that specification.
1.16 jmc 4265: .Sh AUTHORS
1.1 deraadt 4266: Vern Paxson, with the help of many ideas and much inspiration from
1.16 jmc 4267: Van Jacobson.
4268: Original version by Jef Poskanzer.
4269: The fast table representation is a partial implementation of a design done by
4270: Van Jacobson.
4271: The implementation was done by Kevin Gong and Vern Paxson.
4272: .Pp
1.1 deraadt 4273: Thanks to the many
1.16 jmc 4274: .Nm
1.1 deraadt 4275: beta-testers, feedbackers, and contributors, especially Francois Pinard,
4276: Casey Leedom,
4277: Robert Abramovitz,
4278: Stan Adermann, Terry Allen, David Barker-Plummer, John Basrai,
4279: Neal Becker, Nelson H.F. Beebe, benson@odi.com,
4280: Karl Berry, Peter A. Bigot, Simon Blanchard,
4281: Keith Bostic, Frederic Brehm, Ian Brockbank, Kin Cho, Nick Christopher,
4282: Brian Clapper, J.T. Conklin,
4283: Jason Coughlin, Bill Cox, Nick Cropper, Dave Curtis, Scott David
1.11 deraadt 4284: Daniels, Chris G. Demetriou, Theo de Raadt,
1.1 deraadt 4285: Mike Donahue, Chuck Doucette, Tom Epperly, Leo Eskin,
4286: Chris Faylor, Chris Flatters, Jon Forrest, Jeffrey Friedl,
4287: Joe Gayda, Kaveh R. Ghazi, Wolfgang Glunz,
4288: Eric Goldman, Christopher M. Gould, Ulrich Grepel, Peer Griebel,
4289: Jan Hajic, Charles Hemphill, NORO Hideo,
4290: Jarkko Hietaniemi, Scott Hofmann,
4291: Jeff Honig, Dana Hudes, Eric Hughes, John Interrante,
4292: Ceriel Jacobs, Michal Jaegermann, Sakari Jalovaara, Jeffrey R. Jones,
4293: Henry Juengst, Klaus Kaempf, Jonathan I. Kamens, Terrence O Kane,
4294: Amir Katz, ken@ken.hilco.com, Kevin B. Kenny,
4295: Steve Kirsch, Winfried Koenig, Marq Kole, Ronald Lamprecht,
4296: Greg Lee, Rohan Lenard, Craig Leres, John Levine, Steve Liddle,
4297: David Loffredo, Mike Long,
4298: Mohamed el Lozy, Brian Madsen, Malte, Joe Marshall,
4299: Bengt Martensson, Chris Metcalf,
4300: Luke Mewburn, Jim Meyering, R. Alexander Milowski, Erik Naggum,
4301: G.T. Nicol, Landon Noll, James Nordby, Marc Nozell,
4302: Richard Ohnemus, Karsten Pahnke,
1.16 jmc 4303: Sven Panne, Roland Pesch, Walter Pelissero, Gaumond Pierre,
4304: Esmond Pitt, Jef Poskanzer, Joe Rahmeh, Jarmo Raiha,
1.1 deraadt 4305: Frederic Raimbault, Pat Rankin, Rick Richardson,
4306: Kevin Rodgers, Kai Uwe Rommel, Jim Roskind, Alberto Santini,
4307: Andreas Scherer, Darrell Schiebel, Raf Schietekat,
4308: Doug Schmidt, Philippe Schnoebelen, Andreas Schwab,
4309: Larry Schwimmer, Alex Siegel, Eckehard Stolz, Jan-Erik Strvmquist,
4310: Mike Stump, Paul Stuart, Dave Tallman, Ian Lance Taylor,
4311: Chris Thewalt, Richard M. Timoney, Jodi Tsai,
1.16 jmc 4312: Paul Tuinenga, Gary Weik, Frank Whaley, Gerhard Wilhelms, Kent Williams,
4313: Ken Yap, Ron Zellar, Nathan Zelle, David Zuhn,
4314: and those whose names have slipped my marginal mail-archiving skills
4315: but whose contributions are appreciated all the
1.1 deraadt 4316: same.
1.16 jmc 4317: .Pp
1.1 deraadt 4318: Thanks to Keith Bostic, Jon Forrest, Noah Friedman,
4319: John Gilmore, Craig Leres, John Levine, Bob Mulcahy, G.T.
4320: Nicol, Francois Pinard, Rich Salz, and Richard Stallman for help with various
4321: distribution headaches.
1.16 jmc 4322: .Pp
4323: Thanks to Esmond Pitt and Earle Horton for 8-bit character support;
4324: to Benson Margulies and Fred Burke for C++ support;
4325: to Kent Williams and Tom Epperly for C++ class support;
4326: to Ove Ewerlid for support of NUL's;
4327: and to Eric Hughes for support of multiple buffers.
4328: .Pp
1.1 deraadt 4329: This work was primarily done when I was with the Real Time Systems Group
1.16 jmc 4330: at the Lawrence Berkeley Laboratory in Berkeley, CA.
4331: Many thanks to all there for the support I received.
4332: .Pp
4333: Send comments to
4334: .Aq vern@ee.lbl.gov .
4335: .Sh BUGS
4336: Some trailing context patterns cannot be properly matched and generate
4337: warning messages
4338: .Pq "dangerous trailing context" .
4339: These are patterns where the ending of the first part of the rule
4340: matches the beginning of the second part, such as
4341: .Qq zx*/xy* ,
4342: where the
4343: .Sq x*
4344: matches the
4345: .Sq x
4346: at the beginning of the trailing context.
4347: (Note that the POSIX draft states that the text matched by such patterns
4348: is undefined.)
4349: .Pp
4350: For some trailing context rules, parts which are actually fixed-length are
4351: not recognized as such, leading to the above mentioned performance loss.
4352: In particular, parts using
4353: .Sq |\&
4354: or
4355: .Sq {n}
4356: (such as
4357: .Qq foo{3} )
4358: are always considered variable-length.
4359: .Pp
4360: Combining trailing context with the special
4361: .Sq |\&
4362: action can result in fixed trailing context being turned into
4363: the more expensive variable trailing context.
4364: For example, in the following:
4365: .Bd -literal -offset indent
4366: %%
4367: abc |
4368: xyz/def
4369: .Ed
4370: .Pp
4371: Use of
4372: .Fn unput
4373: invalidates yytext and yyleng, unless the
4374: .Dq %array
4375: directive
4376: or the
4377: .Fl l
4378: option has been used.
4379: .Pp
4380: Pattern-matching of NUL's is substantially slower than matching other
4381: characters.
4382: .Pp
4383: Dynamic resizing of the input buffer is slow, as it entails rescanning
4384: all the text matched so far by the current
4385: .Pq generally huge
4386: token.
4387: .Pp
4388: Due to both buffering of input and read-ahead,
4389: it is not possible to intermix calls to
4390: .Aq Pa stdio.h
4391: routines, such as, for example,
4392: .Fn getchar ,
4393: with
4394: .Nm
4395: rules and expect it to work.
4396: Call
4397: .Fn input
4398: instead.
4399: .Pp
4400: The total table entries listed by the
4401: .Fl v
4402: flag excludes the number of table entries needed to determine
4403: what rule has been matched.
4404: The number of entries is equal to the number of DFA states
4405: if the scanner does not use
4406: .Em REJECT ,
4407: and somewhat greater than the number of states if it does.
4408: .Pp
4409: .Em REJECT
4410: cannot be used with the
4411: .Fl f
4412: or
4413: .Fl F
4414: options.
4415: .Pp
4416: The
4417: .Nm
4418: internal algorithms need documentation.