Annotation of src/usr.bin/lex/flex.1, Revision 1.1
1.1 ! deraadt 1: .TH FLEX 1 "April 1995" "Version 2.5"
! 2: .SH NAME
! 3: flex \- fast lexical analyzer generator
! 4: .SH SYNOPSIS
! 5: .B flex
! 6: .B [\-bcdfhilnpstvwBFILTV78+? \-C[aefFmr] \-ooutput \-Pprefix \-Sskeleton]
! 7: .B [\-\-help \-\-version]
! 8: .I [filename ...]
! 9: .SH OVERVIEW
! 10: This manual describes
! 11: .I flex,
! 12: a tool for generating programs that perform pattern-matching on text. The
! 13: manual includes both tutorial and reference sections:
! 14: .nf
! 15:
! 16: Description
! 17: a brief overview of the tool
! 18:
! 19: Some Simple Examples
! 20:
! 21: Format Of The Input File
! 22:
! 23: Patterns
! 24: the extended regular expressions used by flex
! 25:
! 26: How The Input Is Matched
! 27: the rules for determining what has been matched
! 28:
! 29: Actions
! 30: how to specify what to do when a pattern is matched
! 31:
! 32: The Generated Scanner
! 33: details regarding the scanner that flex produces;
! 34: how to control the input source
! 35:
! 36: Start Conditions
! 37: introducing context into your scanners, and
! 38: managing "mini-scanners"
! 39:
! 40: Multiple Input Buffers
! 41: how to manipulate multiple input sources; how to
! 42: scan from strings instead of files
! 43:
! 44: End-of-file Rules
! 45: special rules for matching the end of the input
! 46:
! 47: Miscellaneous Macros
! 48: a summary of macros available to the actions
! 49:
! 50: Values Available To The User
! 51: a summary of values available to the actions
! 52:
! 53: Interfacing With Yacc
! 54: connecting flex scanners together with yacc parsers
! 55:
! 56: Options
! 57: flex command-line options, and the "%option"
! 58: directive
! 59:
! 60: Performance Considerations
! 61: how to make your scanner go as fast as possible
! 62:
! 63: Generating C++ Scanners
! 64: the (experimental) facility for generating C++
! 65: scanner classes
! 66:
! 67: Incompatibilities With Lex And POSIX
! 68: how flex differs from AT&T lex and the POSIX lex
! 69: standard
! 70:
! 71: Diagnostics
! 72: those error messages produced by flex (or scanners
! 73: it generates) whose meanings might not be apparent
! 74:
! 75: Files
! 76: files used by flex
! 77:
! 78: Deficiencies / Bugs
! 79: known problems with flex
! 80:
! 81: See Also
! 82: other documentation, related tools
! 83:
! 84: Author
! 85: includes contact information
! 86:
! 87: .fi
! 88: .SH DESCRIPTION
! 89: .I flex
! 90: is a tool for generating
! 91: .I scanners:
! 92: programs which recognized lexical patterns in text.
! 93: .I flex
! 94: reads
! 95: the given input files, or its standard input if no file names are given,
! 96: for a description of a scanner to generate. The description is in
! 97: the form of pairs
! 98: of regular expressions and C code, called
! 99: .I rules. flex
! 100: generates as output a C source file,
! 101: .B lex.yy.c,
! 102: which defines a routine
! 103: .B yylex().
! 104: This file is compiled and linked with the
! 105: .B \-lfl
! 106: library to produce an executable. When the executable is run,
! 107: it analyzes its input for occurrences
! 108: of the regular expressions. Whenever it finds one, it executes
! 109: the corresponding C code.
! 110: .SH SOME SIMPLE EXAMPLES
! 111: .PP
! 112: First some simple examples to get the flavor of how one uses
! 113: .I flex.
! 114: The following
! 115: .I flex
! 116: input specifies a scanner which whenever it encounters the string
! 117: "username" will replace it with the user's login name:
! 118: .nf
! 119:
! 120: %%
! 121: username printf( "%s", getlogin() );
! 122:
! 123: .fi
! 124: By default, any text not matched by a
! 125: .I flex
! 126: scanner
! 127: is copied to the output, so the net effect of this scanner is
! 128: to copy its input file to its output with each occurrence
! 129: of "username" expanded.
! 130: In this input, there is just one rule. "username" is the
! 131: .I pattern
! 132: and the "printf" is the
! 133: .I action.
! 134: The "%%" marks the beginning of the rules.
! 135: .PP
! 136: Here's another simple example:
! 137: .nf
! 138:
! 139: int num_lines = 0, num_chars = 0;
! 140:
! 141: %%
! 142: \\n ++num_lines; ++num_chars;
! 143: . ++num_chars;
! 144:
! 145: %%
! 146: main()
! 147: {
! 148: yylex();
! 149: printf( "# of lines = %d, # of chars = %d\\n",
! 150: num_lines, num_chars );
! 151: }
! 152:
! 153: .fi
! 154: This scanner counts the number of characters and the number
! 155: of lines in its input (it produces no output other than the
! 156: final report on the counts). The first line
! 157: declares two globals, "num_lines" and "num_chars", which are accessible
! 158: both inside
! 159: .B yylex()
! 160: and in the
! 161: .B main()
! 162: routine declared after the second "%%". There are two rules, one
! 163: which matches a newline ("\\n") and increments both the line count and
! 164: the character count, and one which matches any character other than
! 165: a newline (indicated by the "." regular expression).
! 166: .PP
! 167: A somewhat more complicated example:
! 168: .nf
! 169:
! 170: /* scanner for a toy Pascal-like language */
! 171:
! 172: %{
! 173: /* need this for the call to atof() below */
! 174: #include <math.h>
! 175: %}
! 176:
! 177: DIGIT [0-9]
! 178: ID [a-z][a-z0-9]*
! 179:
! 180: %%
! 181:
! 182: {DIGIT}+ {
! 183: printf( "An integer: %s (%d)\\n", yytext,
! 184: atoi( yytext ) );
! 185: }
! 186:
! 187: {DIGIT}+"."{DIGIT}* {
! 188: printf( "A float: %s (%g)\\n", yytext,
! 189: atof( yytext ) );
! 190: }
! 191:
! 192: if|then|begin|end|procedure|function {
! 193: printf( "A keyword: %s\\n", yytext );
! 194: }
! 195:
! 196: {ID} printf( "An identifier: %s\\n", yytext );
! 197:
! 198: "+"|"-"|"*"|"/" printf( "An operator: %s\\n", yytext );
! 199:
! 200: "{"[^}\\n]*"}" /* eat up one-line comments */
! 201:
! 202: [ \\t\\n]+ /* eat up whitespace */
! 203:
! 204: . printf( "Unrecognized character: %s\\n", yytext );
! 205:
! 206: %%
! 207:
! 208: main( argc, argv )
! 209: int argc;
! 210: char **argv;
! 211: {
! 212: ++argv, --argc; /* skip over program name */
! 213: if ( argc > 0 )
! 214: yyin = fopen( argv[0], "r" );
! 215: else
! 216: yyin = stdin;
! 217:
! 218: yylex();
! 219: }
! 220:
! 221: .fi
! 222: This is the beginnings of a simple scanner for a language like
! 223: Pascal. It identifies different types of
! 224: .I tokens
! 225: and reports on what it has seen.
! 226: .PP
! 227: The details of this example will be explained in the following
! 228: sections.
! 229: .SH FORMAT OF THE INPUT FILE
! 230: The
! 231: .I flex
! 232: input file consists of three sections, separated by a line with just
! 233: .B %%
! 234: in it:
! 235: .nf
! 236:
! 237: definitions
! 238: %%
! 239: rules
! 240: %%
! 241: user code
! 242:
! 243: .fi
! 244: The
! 245: .I definitions
! 246: section contains declarations of simple
! 247: .I name
! 248: definitions to simplify the scanner specification, and declarations of
! 249: .I start conditions,
! 250: which are explained in a later section.
! 251: .PP
! 252: Name definitions have the form:
! 253: .nf
! 254:
! 255: name definition
! 256:
! 257: .fi
! 258: The "name" is a word beginning with a letter or an underscore ('_')
! 259: followed by zero or more letters, digits, '_', or '-' (dash).
! 260: The definition is taken to begin at the first non-white-space character
! 261: following the name and continuing to the end of the line.
! 262: The definition can subsequently be referred to using "{name}", which
! 263: will expand to "(definition)". For example,
! 264: .nf
! 265:
! 266: DIGIT [0-9]
! 267: ID [a-z][a-z0-9]*
! 268:
! 269: .fi
! 270: defines "DIGIT" to be a regular expression which matches a
! 271: single digit, and
! 272: "ID" to be a regular expression which matches a letter
! 273: followed by zero-or-more letters-or-digits.
! 274: A subsequent reference to
! 275: .nf
! 276:
! 277: {DIGIT}+"."{DIGIT}*
! 278:
! 279: .fi
! 280: is identical to
! 281: .nf
! 282:
! 283: ([0-9])+"."([0-9])*
! 284:
! 285: .fi
! 286: and matches one-or-more digits followed by a '.' followed
! 287: by zero-or-more digits.
! 288: .PP
! 289: The
! 290: .I rules
! 291: section of the
! 292: .I flex
! 293: input contains a series of rules of the form:
! 294: .nf
! 295:
! 296: pattern action
! 297:
! 298: .fi
! 299: where the pattern must be unindented and the action must begin
! 300: on the same line.
! 301: .PP
! 302: See below for a further description of patterns and actions.
! 303: .PP
! 304: Finally, the user code section is simply copied to
! 305: .B lex.yy.c
! 306: verbatim.
! 307: It is used for companion routines which call or are called
! 308: by the scanner. The presence of this section is optional;
! 309: if it is missing, the second
! 310: .B %%
! 311: in the input file may be skipped, too.
! 312: .PP
! 313: In the definitions and rules sections, any
! 314: .I indented
! 315: text or text enclosed in
! 316: .B %{
! 317: and
! 318: .B %}
! 319: is copied verbatim to the output (with the %{}'s removed).
! 320: The %{}'s must appear unindented on lines by themselves.
! 321: .PP
! 322: In the rules section,
! 323: any indented or %{} text appearing before the
! 324: first rule may be used to declare variables
! 325: which are local to the scanning routine and (after the declarations)
! 326: code which is to be executed whenever the scanning routine is entered.
! 327: Other indented or %{} text in the rule section is still copied to the output,
! 328: but its meaning is not well-defined and it may well cause compile-time
! 329: errors (this feature is present for
! 330: .I POSIX
! 331: compliance; see below for other such features).
! 332: .PP
! 333: In the definitions section (but not in the rules section),
! 334: an unindented comment (i.e., a line
! 335: beginning with "/*") is also copied verbatim to the output up
! 336: to the next "*/".
! 337: .SH PATTERNS
! 338: The patterns in the input are written using an extended set of regular
! 339: expressions. These are:
! 340: .nf
! 341:
! 342: x match the character 'x'
! 343: . any character (byte) except newline
! 344: [xyz] a "character class"; in this case, the pattern
! 345: matches either an 'x', a 'y', or a 'z'
! 346: [abj-oZ] a "character class" with a range in it; matches
! 347: an 'a', a 'b', any letter from 'j' through 'o',
! 348: or a 'Z'
! 349: [^A-Z] a "negated character class", i.e., any character
! 350: but those in the class. In this case, any
! 351: character EXCEPT an uppercase letter.
! 352: [^A-Z\\n] any character EXCEPT an uppercase letter or
! 353: a newline
! 354: r* zero or more r's, where r is any regular expression
! 355: r+ one or more r's
! 356: r? zero or one r's (that is, "an optional r")
! 357: r{2,5} anywhere from two to five r's
! 358: r{2,} two or more r's
! 359: r{4} exactly 4 r's
! 360: {name} the expansion of the "name" definition
! 361: (see above)
! 362: "[xyz]\\"foo"
! 363: the literal string: [xyz]"foo
! 364: \\X if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
! 365: then the ANSI-C interpretation of \\x.
! 366: Otherwise, a literal 'X' (used to escape
! 367: operators such as '*')
! 368: \\0 a NUL character (ASCII code 0)
! 369: \\123 the character with octal value 123
! 370: \\x2a the character with hexadecimal value 2a
! 371: (r) match an r; parentheses are used to override
! 372: precedence (see below)
! 373:
! 374:
! 375: rs the regular expression r followed by the
! 376: regular expression s; called "concatenation"
! 377:
! 378:
! 379: r|s either an r or an s
! 380:
! 381:
! 382: r/s an r but only if it is followed by an s. The
! 383: text matched by s is included when determining
! 384: whether this rule is the "longest match",
! 385: but is then returned to the input before
! 386: the action is executed. So the action only
! 387: sees the text matched by r. This type
! 388: of pattern is called trailing context".
! 389: (There are some combinations of r/s that flex
! 390: cannot match correctly; see notes in the
! 391: Deficiencies / Bugs section below regarding
! 392: "dangerous trailing context".)
! 393: ^r an r, but only at the beginning of a line (i.e.,
! 394: which just starting to scan, or right after a
! 395: newline has been scanned).
! 396: r$ an r, but only at the end of a line (i.e., just
! 397: before a newline). Equivalent to "r/\\n".
! 398:
! 399: Note that flex's notion of "newline" is exactly
! 400: whatever the C compiler used to compile flex
! 401: interprets '\\n' as; in particular, on some DOS
! 402: systems you must either filter out \\r's in the
! 403: input yourself, or explicitly use r/\\r\\n for "r$".
! 404:
! 405:
! 406: <s>r an r, but only in start condition s (see
! 407: below for discussion of start conditions)
! 408: <s1,s2,s3>r
! 409: same, but in any of start conditions s1,
! 410: s2, or s3
! 411: <*>r an r in any start condition, even an exclusive one.
! 412:
! 413:
! 414: <<EOF>> an end-of-file
! 415: <s1,s2><<EOF>>
! 416: an end-of-file when in start condition s1 or s2
! 417:
! 418: .fi
! 419: Note that inside of a character class, all regular expression operators
! 420: lose their special meaning except escape ('\\') and the character class
! 421: operators, '-', ']', and, at the beginning of the class, '^'.
! 422: .PP
! 423: The regular expressions listed above are grouped according to
! 424: precedence, from highest precedence at the top to lowest at the bottom.
! 425: Those grouped together have equal precedence. For example,
! 426: .nf
! 427:
! 428: foo|bar*
! 429:
! 430: .fi
! 431: is the same as
! 432: .nf
! 433:
! 434: (foo)|(ba(r*))
! 435:
! 436: .fi
! 437: since the '*' operator has higher precedence than concatenation,
! 438: and concatenation higher than alternation ('|'). This pattern
! 439: therefore matches
! 440: .I either
! 441: the string "foo"
! 442: .I or
! 443: the string "ba" followed by zero-or-more r's.
! 444: To match "foo" or zero-or-more "bar"'s, use:
! 445: .nf
! 446:
! 447: foo|(bar)*
! 448:
! 449: .fi
! 450: and to match zero-or-more "foo"'s-or-"bar"'s:
! 451: .nf
! 452:
! 453: (foo|bar)*
! 454:
! 455: .fi
! 456: .PP
! 457: In addition to characters and ranges of characters, character classes
! 458: can also contain character class
! 459: .I expressions.
! 460: These are expressions enclosed inside
! 461: .B [:
! 462: and
! 463: .B :]
! 464: delimiters (which themselves must appear between the '[' and ']' of the
! 465: character class; other elements may occur inside the character class, too).
! 466: The valid expressions are:
! 467: .nf
! 468:
! 469: [:alnum:] [:alpha:] [:blank:]
! 470: [:cntrl:] [:digit:] [:graph:]
! 471: [:lower:] [:print:] [:punct:]
! 472: [:space:] [:upper:] [:xdigit:]
! 473:
! 474: .fi
! 475: These expressions all designate a set of characters equivalent to
! 476: the corresponding standard C
! 477: .B isXXX
! 478: function. For example,
! 479: .B [:alnum:]
! 480: designates those characters for which
! 481: .B isalnum()
! 482: returns true - i.e., any alphabetic or numeric.
! 483: Some systems don't provide
! 484: .B isblank(),
! 485: so flex defines
! 486: .B [:blank:]
! 487: as a blank or a tab.
! 488: .PP
! 489: For example, the following character classes are all equivalent:
! 490: .nf
! 491:
! 492: [[:alnum:]]
! 493: [[:alpha:][:digit:]
! 494: [[:alpha:]0-9]
! 495: [a-zA-Z0-9]
! 496:
! 497: .fi
! 498: If your scanner is case-insensitive (the
! 499: .B \-i
! 500: flag), then
! 501: .B [:upper:]
! 502: and
! 503: .B [:lower:]
! 504: are equivalent to
! 505: .B [:alpha:].
! 506: .PP
! 507: Some notes on patterns:
! 508: .IP -
! 509: A negated character class such as the example "[^A-Z]"
! 510: above
! 511: .I will match a newline
! 512: unless "\\n" (or an equivalent escape sequence) is one of the
! 513: characters explicitly present in the negated character class
! 514: (e.g., "[^A-Z\\n]"). This is unlike how many other regular
! 515: expression tools treat negated character classes, but unfortunately
! 516: the inconsistency is historically entrenched.
! 517: Matching newlines means that a pattern like [^"]* can match the entire
! 518: input unless there's another quote in the input.
! 519: .IP -
! 520: A rule can have at most one instance of trailing context (the '/' operator
! 521: or the '$' operator). The start condition, '^', and "<<EOF>>" patterns
! 522: can only occur at the beginning of a pattern, and, as well as with '/' and '$',
! 523: cannot be grouped inside parentheses. A '^' which does not occur at
! 524: the beginning of a rule or a '$' which does not occur at the end of
! 525: a rule loses its special properties and is treated as a normal character.
! 526: .IP
! 527: The following are illegal:
! 528: .nf
! 529:
! 530: foo/bar$
! 531: <sc1>foo<sc2>bar
! 532:
! 533: .fi
! 534: Note that the first of these, can be written "foo/bar\\n".
! 535: .IP
! 536: The following will result in '$' or '^' being treated as a normal character:
! 537: .nf
! 538:
! 539: foo|(bar$)
! 540: foo|^bar
! 541:
! 542: .fi
! 543: If what's wanted is a "foo" or a bar-followed-by-a-newline, the following
! 544: could be used (the special '|' action is explained below):
! 545: .nf
! 546:
! 547: foo |
! 548: bar$ /* action goes here */
! 549:
! 550: .fi
! 551: A similar trick will work for matching a foo or a
! 552: bar-at-the-beginning-of-a-line.
! 553: .SH HOW THE INPUT IS MATCHED
! 554: When the generated scanner is run, it analyzes its input looking
! 555: for strings which match any of its patterns. If it finds more than
! 556: one match, it takes the one matching the most text (for trailing
! 557: context rules, this includes the length of the trailing part, even
! 558: though it will then be returned to the input). If it finds two
! 559: or more matches of the same length, the
! 560: rule listed first in the
! 561: .I flex
! 562: input file is chosen.
! 563: .PP
! 564: Once the match is determined, the text corresponding to the match
! 565: (called the
! 566: .I token)
! 567: is made available in the global character pointer
! 568: .B yytext,
! 569: and its length in the global integer
! 570: .B yyleng.
! 571: The
! 572: .I action
! 573: corresponding to the matched pattern is then executed (a more
! 574: detailed description of actions follows), and then the remaining
! 575: input is scanned for another match.
! 576: .PP
! 577: If no match is found, then the
! 578: .I default rule
! 579: is executed: the next character in the input is considered matched and
! 580: copied to the standard output. Thus, the simplest legal
! 581: .I flex
! 582: input is:
! 583: .nf
! 584:
! 585: %%
! 586:
! 587: .fi
! 588: which generates a scanner that simply copies its input (one character
! 589: at a time) to its output.
! 590: .PP
! 591: Note that
! 592: .B yytext
! 593: can be defined in two different ways: either as a character
! 594: .I pointer
! 595: or as a character
! 596: .I array.
! 597: You can control which definition
! 598: .I flex
! 599: uses by including one of the special directives
! 600: .B %pointer
! 601: or
! 602: .B %array
! 603: in the first (definitions) section of your flex input. The default is
! 604: .B %pointer,
! 605: unless you use the
! 606: .B -l
! 607: lex compatibility option, in which case
! 608: .B yytext
! 609: will be an array.
! 610: The advantage of using
! 611: .B %pointer
! 612: is substantially faster scanning and no buffer overflow when matching
! 613: very large tokens (unless you run out of dynamic memory). The disadvantage
! 614: is that you are restricted in how your actions can modify
! 615: .B yytext
! 616: (see the next section), and calls to the
! 617: .B unput()
! 618: function destroys the present contents of
! 619: .B yytext,
! 620: which can be a considerable porting headache when moving between different
! 621: .I lex
! 622: versions.
! 623: .PP
! 624: The advantage of
! 625: .B %array
! 626: is that you can then modify
! 627: .B yytext
! 628: to your heart's content, and calls to
! 629: .B unput()
! 630: do not destroy
! 631: .B yytext
! 632: (see below). Furthermore, existing
! 633: .I lex
! 634: programs sometimes access
! 635: .B yytext
! 636: externally using declarations of the form:
! 637: .nf
! 638: extern char yytext[];
! 639: .fi
! 640: This definition is erroneous when used with
! 641: .B %pointer,
! 642: but correct for
! 643: .B %array.
! 644: .PP
! 645: .B %array
! 646: defines
! 647: .B yytext
! 648: to be an array of
! 649: .B YYLMAX
! 650: characters, which defaults to a fairly large value. You can change
! 651: the size by simply #define'ing
! 652: .B YYLMAX
! 653: to a different value in the first section of your
! 654: .I flex
! 655: input. As mentioned above, with
! 656: .B %pointer
! 657: yytext grows dynamically to accommodate large tokens. While this means your
! 658: .B %pointer
! 659: scanner can accommodate very large tokens (such as matching entire blocks
! 660: of comments), bear in mind that each time the scanner must resize
! 661: .B yytext
! 662: it also must rescan the entire token from the beginning, so matching such
! 663: tokens can prove slow.
! 664: .B yytext
! 665: presently does
! 666: .I not
! 667: dynamically grow if a call to
! 668: .B unput()
! 669: results in too much text being pushed back; instead, a run-time error results.
! 670: .PP
! 671: Also note that you cannot use
! 672: .B %array
! 673: with C++ scanner classes
! 674: (the
! 675: .B c++
! 676: option; see below).
! 677: .SH ACTIONS
! 678: Each pattern in a rule has a corresponding action, which can be any
! 679: arbitrary C statement. The pattern ends at the first non-escaped
! 680: whitespace character; the remainder of the line is its action. If the
! 681: action is empty, then when the pattern is matched the input token
! 682: is simply discarded. For example, here is the specification for a program
! 683: which deletes all occurrences of "zap me" from its input:
! 684: .nf
! 685:
! 686: %%
! 687: "zap me"
! 688:
! 689: .fi
! 690: (It will copy all other characters in the input to the output since
! 691: they will be matched by the default rule.)
! 692: .PP
! 693: Here is a program which compresses multiple blanks and tabs down to
! 694: a single blank, and throws away whitespace found at the end of a line:
! 695: .nf
! 696:
! 697: %%
! 698: [ \\t]+ putchar( ' ' );
! 699: [ \\t]+$ /* ignore this token */
! 700:
! 701: .fi
! 702: .PP
! 703: If the action contains a '{', then the action spans till the balancing '}'
! 704: is found, and the action may cross multiple lines.
! 705: .I flex
! 706: knows about C strings and comments and won't be fooled by braces found
! 707: within them, but also allows actions to begin with
! 708: .B %{
! 709: and will consider the action to be all the text up to the next
! 710: .B %}
! 711: (regardless of ordinary braces inside the action).
! 712: .PP
! 713: An action consisting solely of a vertical bar ('|') means "same as
! 714: the action for the next rule." See below for an illustration.
! 715: .PP
! 716: Actions can include arbitrary C code, including
! 717: .B return
! 718: statements to return a value to whatever routine called
! 719: .B yylex().
! 720: Each time
! 721: .B yylex()
! 722: is called it continues processing tokens from where it last left
! 723: off until it either reaches
! 724: the end of the file or executes a return.
! 725: .PP
! 726: Actions are free to modify
! 727: .B yytext
! 728: except for lengthening it (adding
! 729: characters to its end--these will overwrite later characters in the
! 730: input stream). This however does not apply when using
! 731: .B %array
! 732: (see above); in that case,
! 733: .B yytext
! 734: may be freely modified in any way.
! 735: .PP
! 736: Actions are free to modify
! 737: .B yyleng
! 738: except they should not do so if the action also includes use of
! 739: .B yymore()
! 740: (see below).
! 741: .PP
! 742: There are a number of special directives which can be included within
! 743: an action:
! 744: .IP -
! 745: .B ECHO
! 746: copies yytext to the scanner's output.
! 747: .IP -
! 748: .B BEGIN
! 749: followed by the name of a start condition places the scanner in the
! 750: corresponding start condition (see below).
! 751: .IP -
! 752: .B REJECT
! 753: directs the scanner to proceed on to the "second best" rule which matched the
! 754: input (or a prefix of the input). The rule is chosen as described
! 755: above in "How the Input is Matched", and
! 756: .B yytext
! 757: and
! 758: .B yyleng
! 759: set up appropriately.
! 760: It may either be one which matched as much text
! 761: as the originally chosen rule but came later in the
! 762: .I flex
! 763: input file, or one which matched less text.
! 764: For example, the following will both count the
! 765: words in the input and call the routine special() whenever "frob" is seen:
! 766: .nf
! 767:
! 768: int word_count = 0;
! 769: %%
! 770:
! 771: frob special(); REJECT;
! 772: [^ \\t\\n]+ ++word_count;
! 773:
! 774: .fi
! 775: Without the
! 776: .B REJECT,
! 777: any "frob"'s in the input would not be counted as words, since the
! 778: scanner normally executes only one action per token.
! 779: Multiple
! 780: .B REJECT's
! 781: are allowed, each one finding the next best choice to the currently
! 782: active rule. For example, when the following scanner scans the token
! 783: "abcd", it will write "abcdabcaba" to the output:
! 784: .nf
! 785:
! 786: %%
! 787: a |
! 788: ab |
! 789: abc |
! 790: abcd ECHO; REJECT;
! 791: .|\\n /* eat up any unmatched character */
! 792:
! 793: .fi
! 794: (The first three rules share the fourth's action since they use
! 795: the special '|' action.)
! 796: .B REJECT
! 797: is a particularly expensive feature in terms of scanner performance;
! 798: if it is used in
! 799: .I any
! 800: of the scanner's actions it will slow down
! 801: .I all
! 802: of the scanner's matching. Furthermore,
! 803: .B REJECT
! 804: cannot be used with the
! 805: .I -Cf
! 806: or
! 807: .I -CF
! 808: options (see below).
! 809: .IP
! 810: Note also that unlike the other special actions,
! 811: .B REJECT
! 812: is a
! 813: .I branch;
! 814: code immediately following it in the action will
! 815: .I not
! 816: be executed.
! 817: .IP -
! 818: .B yymore()
! 819: tells the scanner that the next time it matches a rule, the corresponding
! 820: token should be
! 821: .I appended
! 822: onto the current value of
! 823: .B yytext
! 824: rather than replacing it. For example, given the input "mega-kludge"
! 825: the following will write "mega-mega-kludge" to the output:
! 826: .nf
! 827:
! 828: %%
! 829: mega- ECHO; yymore();
! 830: kludge ECHO;
! 831:
! 832: .fi
! 833: First "mega-" is matched and echoed to the output. Then "kludge"
! 834: is matched, but the previous "mega-" is still hanging around at the
! 835: beginning of
! 836: .B yytext
! 837: so the
! 838: .B ECHO
! 839: for the "kludge" rule will actually write "mega-kludge".
! 840: .PP
! 841: Two notes regarding use of
! 842: .B yymore().
! 843: First,
! 844: .B yymore()
! 845: depends on the value of
! 846: .I yyleng
! 847: correctly reflecting the size of the current token, so you must not
! 848: modify
! 849: .I yyleng
! 850: if you are using
! 851: .B yymore().
! 852: Second, the presence of
! 853: .B yymore()
! 854: in the scanner's action entails a minor performance penalty in the
! 855: scanner's matching speed.
! 856: .IP -
! 857: .B yyless(n)
! 858: returns all but the first
! 859: .I n
! 860: characters of the current token back to the input stream, where they
! 861: will be rescanned when the scanner looks for the next match.
! 862: .B yytext
! 863: and
! 864: .B yyleng
! 865: are adjusted appropriately (e.g.,
! 866: .B yyleng
! 867: will now be equal to
! 868: .I n
! 869: ). For example, on the input "foobar" the following will write out
! 870: "foobarbar":
! 871: .nf
! 872:
! 873: %%
! 874: foobar ECHO; yyless(3);
! 875: [a-z]+ ECHO;
! 876:
! 877: .fi
! 878: An argument of 0 to
! 879: .B yyless
! 880: will cause the entire current input string to be scanned again. Unless you've
! 881: changed how the scanner will subsequently process its input (using
! 882: .B BEGIN,
! 883: for example), this will result in an endless loop.
! 884: .PP
! 885: Note that
! 886: .B yyless
! 887: is a macro and can only be used in the flex input file, not from
! 888: other source files.
! 889: .IP -
! 890: .B unput(c)
! 891: puts the character
! 892: .I c
! 893: back onto the input stream. It will be the next character scanned.
! 894: The following action will take the current token and cause it
! 895: to be rescanned enclosed in parentheses.
! 896: .nf
! 897:
! 898: {
! 899: int i;
! 900: /* Copy yytext because unput() trashes yytext */
! 901: char *yycopy = strdup( yytext );
! 902: unput( ')' );
! 903: for ( i = yyleng - 1; i >= 0; --i )
! 904: unput( yycopy[i] );
! 905: unput( '(' );
! 906: free( yycopy );
! 907: }
! 908:
! 909: .fi
! 910: Note that since each
! 911: .B unput()
! 912: puts the given character back at the
! 913: .I beginning
! 914: of the input stream, pushing back strings must be done back-to-front.
! 915: .PP
! 916: An important potential problem when using
! 917: .B unput()
! 918: is that if you are using
! 919: .B %pointer
! 920: (the default), a call to
! 921: .B unput()
! 922: .I destroys
! 923: the contents of
! 924: .I yytext,
! 925: starting with its rightmost character and devouring one character to
! 926: the left with each call. If you need the value of yytext preserved
! 927: after a call to
! 928: .B unput()
! 929: (as in the above example),
! 930: you must either first copy it elsewhere, or build your scanner using
! 931: .B %array
! 932: instead (see How The Input Is Matched).
! 933: .PP
! 934: Finally, note that you cannot put back
! 935: .B EOF
! 936: to attempt to mark the input stream with an end-of-file.
! 937: .IP -
! 938: .B input()
! 939: reads the next character from the input stream. For example,
! 940: the following is one way to eat up C comments:
! 941: .nf
! 942:
! 943: %%
! 944: "/*" {
! 945: register int c;
! 946:
! 947: for ( ; ; )
! 948: {
! 949: while ( (c = input()) != '*' &&
! 950: c != EOF )
! 951: ; /* eat up text of comment */
! 952:
! 953: if ( c == '*' )
! 954: {
! 955: while ( (c = input()) == '*' )
! 956: ;
! 957: if ( c == '/' )
! 958: break; /* found the end */
! 959: }
! 960:
! 961: if ( c == EOF )
! 962: {
! 963: error( "EOF in comment" );
! 964: break;
! 965: }
! 966: }
! 967: }
! 968:
! 969: .fi
! 970: (Note that if the scanner is compiled using
! 971: .B C++,
! 972: then
! 973: .B input()
! 974: is instead referred to as
! 975: .B yyinput(),
! 976: in order to avoid a name clash with the
! 977: .B C++
! 978: stream by the name of
! 979: .I input.)
! 980: .IP -
! 981: .B YY_FLUSH_BUFFER
! 982: flushes the scanner's internal buffer
! 983: so that the next time the scanner attempts to match a token, it will
! 984: first refill the buffer using
! 985: .B YY_INPUT
! 986: (see The Generated Scanner, below). This action is a special case
! 987: of the more general
! 988: .B yy_flush_buffer()
! 989: function, described below in the section Multiple Input Buffers.
! 990: .IP -
! 991: .B yyterminate()
! 992: can be used in lieu of a return statement in an action. It terminates
! 993: the scanner and returns a 0 to the scanner's caller, indicating "all done".
! 994: By default,
! 995: .B yyterminate()
! 996: is also called when an end-of-file is encountered. It is a macro and
! 997: may be redefined.
! 998: .SH THE GENERATED SCANNER
! 999: The output of
! 1000: .I flex
! 1001: is the file
! 1002: .B lex.yy.c,
! 1003: which contains the scanning routine
! 1004: .B yylex(),
! 1005: a number of tables used by it for matching tokens, and a number
! 1006: of auxiliary routines and macros. By default,
! 1007: .B yylex()
! 1008: is declared as follows:
! 1009: .nf
! 1010:
! 1011: int yylex()
! 1012: {
! 1013: ... various definitions and the actions in here ...
! 1014: }
! 1015:
! 1016: .fi
! 1017: (If your environment supports function prototypes, then it will
! 1018: be "int yylex( void )".) This definition may be changed by defining
! 1019: the "YY_DECL" macro. For example, you could use:
! 1020: .nf
! 1021:
! 1022: #define YY_DECL float lexscan( a, b ) float a, b;
! 1023:
! 1024: .fi
! 1025: to give the scanning routine the name
! 1026: .I lexscan,
! 1027: returning a float, and taking two floats as arguments. Note that
! 1028: if you give arguments to the scanning routine using a
! 1029: K&R-style/non-prototyped function declaration, you must terminate
! 1030: the definition with a semi-colon (;).
! 1031: .PP
! 1032: Whenever
! 1033: .B yylex()
! 1034: is called, it scans tokens from the global input file
! 1035: .I yyin
! 1036: (which defaults to stdin). It continues until it either reaches
! 1037: an end-of-file (at which point it returns the value 0) or
! 1038: one of its actions executes a
! 1039: .I return
! 1040: statement.
! 1041: .PP
! 1042: If the scanner reaches an end-of-file, subsequent calls are undefined
! 1043: unless either
! 1044: .I yyin
! 1045: is pointed at a new input file (in which case scanning continues from
! 1046: that file), or
! 1047: .B yyrestart()
! 1048: is called.
! 1049: .B yyrestart()
! 1050: takes one argument, a
! 1051: .B FILE *
! 1052: pointer (which can be nil, if you've set up
! 1053: .B YY_INPUT
! 1054: to scan from a source other than
! 1055: .I yyin),
! 1056: and initializes
! 1057: .I yyin
! 1058: for scanning from that file. Essentially there is no difference between
! 1059: just assigning
! 1060: .I yyin
! 1061: to a new input file or using
! 1062: .B yyrestart()
! 1063: to do so; the latter is available for compatibility with previous versions
! 1064: of
! 1065: .I flex,
! 1066: and because it can be used to switch input files in the middle of scanning.
! 1067: It can also be used to throw away the current input buffer, by calling
! 1068: it with an argument of
! 1069: .I yyin;
! 1070: but better is to use
! 1071: .B YY_FLUSH_BUFFER
! 1072: (see above).
! 1073: Note that
! 1074: .B yyrestart()
! 1075: does
! 1076: .I not
! 1077: reset the start condition to
! 1078: .B INITIAL
! 1079: (see Start Conditions, below).
! 1080: .PP
! 1081: If
! 1082: .B yylex()
! 1083: stops scanning due to executing a
! 1084: .I return
! 1085: statement in one of the actions, the scanner may then be called again and it
! 1086: will resume scanning where it left off.
! 1087: .PP
! 1088: By default (and for purposes of efficiency), the scanner uses
! 1089: block-reads rather than simple
! 1090: .I getc()
! 1091: calls to read characters from
! 1092: .I yyin.
! 1093: The nature of how it gets its input can be controlled by defining the
! 1094: .B YY_INPUT
! 1095: macro.
! 1096: YY_INPUT's calling sequence is "YY_INPUT(buf,result,max_size)". Its
! 1097: action is to place up to
! 1098: .I max_size
! 1099: characters in the character array
! 1100: .I buf
! 1101: and return in the integer variable
! 1102: .I result
! 1103: either the
! 1104: number of characters read or the constant YY_NULL (0 on Unix systems)
! 1105: to indicate EOF. The default YY_INPUT reads from the
! 1106: global file-pointer "yyin".
! 1107: .PP
! 1108: A sample definition of YY_INPUT (in the definitions
! 1109: section of the input file):
! 1110: .nf
! 1111:
! 1112: %{
! 1113: #define YY_INPUT(buf,result,max_size) \\
! 1114: { \\
! 1115: int c = getchar(); \\
! 1116: result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \\
! 1117: }
! 1118: %}
! 1119:
! 1120: .fi
! 1121: This definition will change the input processing to occur
! 1122: one character at a time.
! 1123: .PP
! 1124: When the scanner receives an end-of-file indication from YY_INPUT,
! 1125: it then checks the
! 1126: .B yywrap()
! 1127: function. If
! 1128: .B yywrap()
! 1129: returns false (zero), then it is assumed that the
! 1130: function has gone ahead and set up
! 1131: .I yyin
! 1132: to point to another input file, and scanning continues. If it returns
! 1133: true (non-zero), then the scanner terminates, returning 0 to its
! 1134: caller. Note that in either case, the start condition remains unchanged;
! 1135: it does
! 1136: .I not
! 1137: revert to
! 1138: .B INITIAL.
! 1139: .PP
! 1140: If you do not supply your own version of
! 1141: .B yywrap(),
! 1142: then you must either use
! 1143: .B %option noyywrap
! 1144: (in which case the scanner behaves as though
! 1145: .B yywrap()
! 1146: returned 1), or you must link with
! 1147: .B \-lfl
! 1148: to obtain the default version of the routine, which always returns 1.
! 1149: .PP
! 1150: Three routines are available for scanning from in-memory buffers rather
! 1151: than files:
! 1152: .B yy_scan_string(), yy_scan_bytes(),
! 1153: and
! 1154: .B yy_scan_buffer().
! 1155: See the discussion of them below in the section Multiple Input Buffers.
! 1156: .PP
! 1157: The scanner writes its
! 1158: .B ECHO
! 1159: output to the
! 1160: .I yyout
! 1161: global (default, stdout), which may be redefined by the user simply
! 1162: by assigning it to some other
! 1163: .B FILE
! 1164: pointer.
! 1165: .SH START CONDITIONS
! 1166: .I flex
! 1167: provides a mechanism for conditionally activating rules. Any rule
! 1168: whose pattern is prefixed with "<sc>" will only be active when
! 1169: the scanner is in the start condition named "sc". For example,
! 1170: .nf
! 1171:
! 1172: <STRING>[^"]* { /* eat up the string body ... */
! 1173: ...
! 1174: }
! 1175:
! 1176: .fi
! 1177: will be active only when the scanner is in the "STRING" start
! 1178: condition, and
! 1179: .nf
! 1180:
! 1181: <INITIAL,STRING,QUOTE>\\. { /* handle an escape ... */
! 1182: ...
! 1183: }
! 1184:
! 1185: .fi
! 1186: will be active only when the current start condition is
! 1187: either "INITIAL", "STRING", or "QUOTE".
! 1188: .PP
! 1189: Start conditions
! 1190: are declared in the definitions (first) section of the input
! 1191: using unindented lines beginning with either
! 1192: .B %s
! 1193: or
! 1194: .B %x
! 1195: followed by a list of names.
! 1196: The former declares
! 1197: .I inclusive
! 1198: start conditions, the latter
! 1199: .I exclusive
! 1200: start conditions. A start condition is activated using the
! 1201: .B BEGIN
! 1202: action. Until the next
! 1203: .B BEGIN
! 1204: action is executed, rules with the given start
! 1205: condition will be active and
! 1206: rules with other start conditions will be inactive.
! 1207: If the start condition is
! 1208: .I inclusive,
! 1209: then rules with no start conditions at all will also be active.
! 1210: If it is
! 1211: .I exclusive,
! 1212: then
! 1213: .I only
! 1214: rules qualified with the start condition will be active.
! 1215: A set of rules contingent on the same exclusive start condition
! 1216: describe a scanner which is independent of any of the other rules in the
! 1217: .I flex
! 1218: input. Because of this,
! 1219: exclusive start conditions make it easy to specify "mini-scanners"
! 1220: which scan portions of the input that are syntactically different
! 1221: from the rest (e.g., comments).
! 1222: .PP
! 1223: If the distinction between inclusive and exclusive start conditions
! 1224: is still a little vague, here's a simple example illustrating the
! 1225: connection between the two. The set of rules:
! 1226: .nf
! 1227:
! 1228: %s example
! 1229: %%
! 1230:
! 1231: <example>foo do_something();
! 1232:
! 1233: bar something_else();
! 1234:
! 1235: .fi
! 1236: is equivalent to
! 1237: .nf
! 1238:
! 1239: %x example
! 1240: %%
! 1241:
! 1242: <example>foo do_something();
! 1243:
! 1244: <INITIAL,example>bar something_else();
! 1245:
! 1246: .fi
! 1247: Without the
! 1248: .B <INITIAL,example>
! 1249: qualifier, the
! 1250: .I bar
! 1251: pattern in the second example wouldn't be active (i.e., couldn't match)
! 1252: when in start condition
! 1253: .B example.
! 1254: If we just used
! 1255: .B <example>
! 1256: to qualify
! 1257: .I bar,
! 1258: though, then it would only be active in
! 1259: .B example
! 1260: and not in
! 1261: .B INITIAL,
! 1262: while in the first example it's active in both, because in the first
! 1263: example the
! 1264: .B example
! 1265: startion condition is an
! 1266: .I inclusive
! 1267: .B (%s)
! 1268: start condition.
! 1269: .PP
! 1270: Also note that the special start-condition specifier
! 1271: .B <*>
! 1272: matches every start condition. Thus, the above example could also
! 1273: have been written;
! 1274: .nf
! 1275:
! 1276: %x example
! 1277: %%
! 1278:
! 1279: <example>foo do_something();
! 1280:
! 1281: <*>bar something_else();
! 1282:
! 1283: .fi
! 1284: .PP
! 1285: The default rule (to
! 1286: .B ECHO
! 1287: any unmatched character) remains active in start conditions. It
! 1288: is equivalent to:
! 1289: .nf
! 1290:
! 1291: <*>.|\\n ECHO;
! 1292:
! 1293: .fi
! 1294: .PP
! 1295: .B BEGIN(0)
! 1296: returns to the original state where only the rules with
! 1297: no start conditions are active. This state can also be
! 1298: referred to as the start-condition "INITIAL", so
! 1299: .B BEGIN(INITIAL)
! 1300: is equivalent to
! 1301: .B BEGIN(0).
! 1302: (The parentheses around the start condition name are not required but
! 1303: are considered good style.)
! 1304: .PP
! 1305: .B BEGIN
! 1306: actions can also be given as indented code at the beginning
! 1307: of the rules section. For example, the following will cause
! 1308: the scanner to enter the "SPECIAL" start condition whenever
! 1309: .B yylex()
! 1310: is called and the global variable
! 1311: .I enter_special
! 1312: is true:
! 1313: .nf
! 1314:
! 1315: int enter_special;
! 1316:
! 1317: %x SPECIAL
! 1318: %%
! 1319: if ( enter_special )
! 1320: BEGIN(SPECIAL);
! 1321:
! 1322: <SPECIAL>blahblahblah
! 1323: ...more rules follow...
! 1324:
! 1325: .fi
! 1326: .PP
! 1327: To illustrate the uses of start conditions,
! 1328: here is a scanner which provides two different interpretations
! 1329: of a string like "123.456". By default it will treat it as
! 1330: three tokens, the integer "123", a dot ('.'), and the integer "456".
! 1331: But if the string is preceded earlier in the line by the string
! 1332: "expect-floats"
! 1333: it will treat it as a single token, the floating-point number
! 1334: 123.456:
! 1335: .nf
! 1336:
! 1337: %{
! 1338: #include <math.h>
! 1339: %}
! 1340: %s expect
! 1341:
! 1342: %%
! 1343: expect-floats BEGIN(expect);
! 1344:
! 1345: <expect>[0-9]+"."[0-9]+ {
! 1346: printf( "found a float, = %f\\n",
! 1347: atof( yytext ) );
! 1348: }
! 1349: <expect>\\n {
! 1350: /* that's the end of the line, so
! 1351: * we need another "expect-number"
! 1352: * before we'll recognize any more
! 1353: * numbers
! 1354: */
! 1355: BEGIN(INITIAL);
! 1356: }
! 1357:
! 1358: [0-9]+ {
! 1359: printf( "found an integer, = %d\\n",
! 1360: atoi( yytext ) );
! 1361: }
! 1362:
! 1363: "." printf( "found a dot\\n" );
! 1364:
! 1365: .fi
! 1366: Here is a scanner which recognizes (and discards) C comments while
! 1367: maintaining a count of the current input line.
! 1368: .nf
! 1369:
! 1370: %x comment
! 1371: %%
! 1372: int line_num = 1;
! 1373:
! 1374: "/*" BEGIN(comment);
! 1375:
! 1376: <comment>[^*\\n]* /* eat anything that's not a '*' */
! 1377: <comment>"*"+[^*/\\n]* /* eat up '*'s not followed by '/'s */
! 1378: <comment>\\n ++line_num;
! 1379: <comment>"*"+"/" BEGIN(INITIAL);
! 1380:
! 1381: .fi
! 1382: This scanner goes to a bit of trouble to match as much
! 1383: text as possible with each rule. In general, when attempting to write
! 1384: a high-speed scanner try to match as much possible in each rule, as
! 1385: it's a big win.
! 1386: .PP
! 1387: Note that start-conditions names are really integer values and
! 1388: can be stored as such. Thus, the above could be extended in the
! 1389: following fashion:
! 1390: .nf
! 1391:
! 1392: %x comment foo
! 1393: %%
! 1394: int line_num = 1;
! 1395: int comment_caller;
! 1396:
! 1397: "/*" {
! 1398: comment_caller = INITIAL;
! 1399: BEGIN(comment);
! 1400: }
! 1401:
! 1402: ...
! 1403:
! 1404: <foo>"/*" {
! 1405: comment_caller = foo;
! 1406: BEGIN(comment);
! 1407: }
! 1408:
! 1409: <comment>[^*\\n]* /* eat anything that's not a '*' */
! 1410: <comment>"*"+[^*/\\n]* /* eat up '*'s not followed by '/'s */
! 1411: <comment>\\n ++line_num;
! 1412: <comment>"*"+"/" BEGIN(comment_caller);
! 1413:
! 1414: .fi
! 1415: Furthermore, you can access the current start condition using
! 1416: the integer-valued
! 1417: .B YY_START
! 1418: macro. For example, the above assignments to
! 1419: .I comment_caller
! 1420: could instead be written
! 1421: .nf
! 1422:
! 1423: comment_caller = YY_START;
! 1424:
! 1425: .fi
! 1426: Flex provides
! 1427: .B YYSTATE
! 1428: as an alias for
! 1429: .B YY_START
! 1430: (since that is what's used by AT&T
! 1431: .I lex).
! 1432: .PP
! 1433: Note that start conditions do not have their own name-space; %s's and %x's
! 1434: declare names in the same fashion as #define's.
! 1435: .PP
! 1436: Finally, here's an example of how to match C-style quoted strings using
! 1437: exclusive start conditions, including expanded escape sequences (but
! 1438: not including checking for a string that's too long):
! 1439: .nf
! 1440:
! 1441: %x str
! 1442:
! 1443: %%
! 1444: char string_buf[MAX_STR_CONST];
! 1445: char *string_buf_ptr;
! 1446:
! 1447:
! 1448: \\" string_buf_ptr = string_buf; BEGIN(str);
! 1449:
! 1450: <str>\\" { /* saw closing quote - all done */
! 1451: BEGIN(INITIAL);
! 1452: *string_buf_ptr = '\\0';
! 1453: /* return string constant token type and
! 1454: * value to parser
! 1455: */
! 1456: }
! 1457:
! 1458: <str>\\n {
! 1459: /* error - unterminated string constant */
! 1460: /* generate error message */
! 1461: }
! 1462:
! 1463: <str>\\\\[0-7]{1,3} {
! 1464: /* octal escape sequence */
! 1465: int result;
! 1466:
! 1467: (void) sscanf( yytext + 1, "%o", &result );
! 1468:
! 1469: if ( result > 0xff )
! 1470: /* error, constant is out-of-bounds */
! 1471:
! 1472: *string_buf_ptr++ = result;
! 1473: }
! 1474:
! 1475: <str>\\\\[0-9]+ {
! 1476: /* generate error - bad escape sequence; something
! 1477: * like '\\48' or '\\0777777'
! 1478: */
! 1479: }
! 1480:
! 1481: <str>\\\\n *string_buf_ptr++ = '\\n';
! 1482: <str>\\\\t *string_buf_ptr++ = '\\t';
! 1483: <str>\\\\r *string_buf_ptr++ = '\\r';
! 1484: <str>\\\\b *string_buf_ptr++ = '\\b';
! 1485: <str>\\\\f *string_buf_ptr++ = '\\f';
! 1486:
! 1487: <str>\\\\(.|\\n) *string_buf_ptr++ = yytext[1];
! 1488:
! 1489: <str>[^\\\\\\n\\"]+ {
! 1490: char *yptr = yytext;
! 1491:
! 1492: while ( *yptr )
! 1493: *string_buf_ptr++ = *yptr++;
! 1494: }
! 1495:
! 1496: .fi
! 1497: .PP
! 1498: Often, such as in some of the examples above, you wind up writing a
! 1499: whole bunch of rules all preceded by the same start condition(s). Flex
! 1500: makes this a little easier and cleaner by introducing a notion of
! 1501: start condition
! 1502: .I scope.
! 1503: A start condition scope is begun with:
! 1504: .nf
! 1505:
! 1506: <SCs>{
! 1507:
! 1508: .fi
! 1509: where
! 1510: .I SCs
! 1511: is a list of one or more start conditions. Inside the start condition
! 1512: scope, every rule automatically has the prefix
! 1513: .I <SCs>
! 1514: applied to it, until a
! 1515: .I '}'
! 1516: which matches the initial
! 1517: .I '{'.
! 1518: So, for example,
! 1519: .nf
! 1520:
! 1521: <ESC>{
! 1522: "\\\\n" return '\\n';
! 1523: "\\\\r" return '\\r';
! 1524: "\\\\f" return '\\f';
! 1525: "\\\\0" return '\\0';
! 1526: }
! 1527:
! 1528: .fi
! 1529: is equivalent to:
! 1530: .nf
! 1531:
! 1532: <ESC>"\\\\n" return '\\n';
! 1533: <ESC>"\\\\r" return '\\r';
! 1534: <ESC>"\\\\f" return '\\f';
! 1535: <ESC>"\\\\0" return '\\0';
! 1536:
! 1537: .fi
! 1538: Start condition scopes may be nested.
! 1539: .PP
! 1540: Three routines are available for manipulating stacks of start conditions:
! 1541: .TP
! 1542: .B void yy_push_state(int new_state)
! 1543: pushes the current start condition onto the top of the start condition
! 1544: stack and switches to
! 1545: .I new_state
! 1546: as though you had used
! 1547: .B BEGIN new_state
! 1548: (recall that start condition names are also integers).
! 1549: .TP
! 1550: .B void yy_pop_state()
! 1551: pops the top of the stack and switches to it via
! 1552: .B BEGIN.
! 1553: .TP
! 1554: .B int yy_top_state()
! 1555: returns the top of the stack without altering the stack's contents.
! 1556: .PP
! 1557: The start condition stack grows dynamically and so has no built-in
! 1558: size limitation. If memory is exhausted, program execution aborts.
! 1559: .PP
! 1560: To use start condition stacks, your scanner must include a
! 1561: .B %option stack
! 1562: directive (see Options below).
! 1563: .SH MULTIPLE INPUT BUFFERS
! 1564: Some scanners (such as those which support "include" files)
! 1565: require reading from several input streams. As
! 1566: .I flex
! 1567: scanners do a large amount of buffering, one cannot control
! 1568: where the next input will be read from by simply writing a
! 1569: .B YY_INPUT
! 1570: which is sensitive to the scanning context.
! 1571: .B YY_INPUT
! 1572: is only called when the scanner reaches the end of its buffer, which
! 1573: may be a long time after scanning a statement such as an "include"
! 1574: which requires switching the input source.
! 1575: .PP
! 1576: To negotiate these sorts of problems,
! 1577: .I flex
! 1578: provides a mechanism for creating and switching between multiple
! 1579: input buffers. An input buffer is created by using:
! 1580: .nf
! 1581:
! 1582: YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )
! 1583:
! 1584: .fi
! 1585: which takes a
! 1586: .I FILE
! 1587: pointer and a size and creates a buffer associated with the given
! 1588: file and large enough to hold
! 1589: .I size
! 1590: characters (when in doubt, use
! 1591: .B YY_BUF_SIZE
! 1592: for the size). It returns a
! 1593: .B YY_BUFFER_STATE
! 1594: handle, which may then be passed to other routines (see below). The
! 1595: .B YY_BUFFER_STATE
! 1596: type is a pointer to an opaque
! 1597: .B struct yy_buffer_state
! 1598: structure, so you may safely initialize YY_BUFFER_STATE variables to
! 1599: .B ((YY_BUFFER_STATE) 0)
! 1600: if you wish, and also refer to the opaque structure in order to
! 1601: correctly declare input buffers in source files other than that
! 1602: of your scanner. Note that the
! 1603: .I FILE
! 1604: pointer in the call to
! 1605: .B yy_create_buffer
! 1606: is only used as the value of
! 1607: .I yyin
! 1608: seen by
! 1609: .B YY_INPUT;
! 1610: if you redefine
! 1611: .B YY_INPUT
! 1612: so it no longer uses
! 1613: .I yyin,
! 1614: then you can safely pass a nil
! 1615: .I FILE
! 1616: pointer to
! 1617: .B yy_create_buffer.
! 1618: You select a particular buffer to scan from using:
! 1619: .nf
! 1620:
! 1621: void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )
! 1622:
! 1623: .fi
! 1624: switches the scanner's input buffer so subsequent tokens will
! 1625: come from
! 1626: .I new_buffer.
! 1627: Note that
! 1628: .B yy_switch_to_buffer()
! 1629: may be used by yywrap() to set things up for continued scanning, instead
! 1630: of opening a new file and pointing
! 1631: .I yyin
! 1632: at it. Note also that switching input sources via either
! 1633: .B yy_switch_to_buffer()
! 1634: or
! 1635: .B yywrap()
! 1636: does
! 1637: .I not
! 1638: change the start condition.
! 1639: .nf
! 1640:
! 1641: void yy_delete_buffer( YY_BUFFER_STATE buffer )
! 1642:
! 1643: .fi
! 1644: is used to reclaim the storage associated with a buffer. (
! 1645: .B buffer
! 1646: can be nil, in which case the routine does nothing.)
! 1647: You can also clear the current contents of a buffer using:
! 1648: .nf
! 1649:
! 1650: void yy_flush_buffer( YY_BUFFER_STATE buffer )
! 1651:
! 1652: .fi
! 1653: This function discards the buffer's contents,
! 1654: so the next time the scanner attempts to match a token from the
! 1655: buffer, it will first fill the buffer anew using
! 1656: .B YY_INPUT.
! 1657: .PP
! 1658: .B yy_new_buffer()
! 1659: is an alias for
! 1660: .B yy_create_buffer(),
! 1661: provided for compatibility with the C++ use of
! 1662: .I new
! 1663: and
! 1664: .I delete
! 1665: for creating and destroying dynamic objects.
! 1666: .PP
! 1667: Finally, the
! 1668: .B YY_CURRENT_BUFFER
! 1669: macro returns a
! 1670: .B YY_BUFFER_STATE
! 1671: handle to the current buffer.
! 1672: .PP
! 1673: Here is an example of using these features for writing a scanner
! 1674: which expands include files (the
! 1675: .B <<EOF>>
! 1676: feature is discussed below):
! 1677: .nf
! 1678:
! 1679: /* the "incl" state is used for picking up the name
! 1680: * of an include file
! 1681: */
! 1682: %x incl
! 1683:
! 1684: %{
! 1685: #define MAX_INCLUDE_DEPTH 10
! 1686: YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
! 1687: int include_stack_ptr = 0;
! 1688: %}
! 1689:
! 1690: %%
! 1691: include BEGIN(incl);
! 1692:
! 1693: [a-z]+ ECHO;
! 1694: [^a-z\\n]*\\n? ECHO;
! 1695:
! 1696: <incl>[ \\t]* /* eat the whitespace */
! 1697: <incl>[^ \\t\\n]+ { /* got the include file name */
! 1698: if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
! 1699: {
! 1700: fprintf( stderr, "Includes nested too deeply" );
! 1701: exit( 1 );
! 1702: }
! 1703:
! 1704: include_stack[include_stack_ptr++] =
! 1705: YY_CURRENT_BUFFER;
! 1706:
! 1707: yyin = fopen( yytext, "r" );
! 1708:
! 1709: if ( ! yyin )
! 1710: error( ... );
! 1711:
! 1712: yy_switch_to_buffer(
! 1713: yy_create_buffer( yyin, YY_BUF_SIZE ) );
! 1714:
! 1715: BEGIN(INITIAL);
! 1716: }
! 1717:
! 1718: <<EOF>> {
! 1719: if ( --include_stack_ptr < 0 )
! 1720: {
! 1721: yyterminate();
! 1722: }
! 1723:
! 1724: else
! 1725: {
! 1726: yy_delete_buffer( YY_CURRENT_BUFFER );
! 1727: yy_switch_to_buffer(
! 1728: include_stack[include_stack_ptr] );
! 1729: }
! 1730: }
! 1731:
! 1732: .fi
! 1733: Three routines are available for setting up input buffers for
! 1734: scanning in-memory strings instead of files. All of them create
! 1735: a new input buffer for scanning the string, and return a corresponding
! 1736: .B YY_BUFFER_STATE
! 1737: handle (which you should delete with
! 1738: .B yy_delete_buffer()
! 1739: when done with it). They also switch to the new buffer using
! 1740: .B yy_switch_to_buffer(),
! 1741: so the next call to
! 1742: .B yylex()
! 1743: will start scanning the string.
! 1744: .TP
! 1745: .B yy_scan_string(const char *str)
! 1746: scans a NUL-terminated string.
! 1747: .TP
! 1748: .B yy_scan_bytes(const char *bytes, int len)
! 1749: scans
! 1750: .I len
! 1751: bytes (including possibly NUL's)
! 1752: starting at location
! 1753: .I bytes.
! 1754: .PP
! 1755: Note that both of these functions create and scan a
! 1756: .I copy
! 1757: of the string or bytes. (This may be desirable, since
! 1758: .B yylex()
! 1759: modifies the contents of the buffer it is scanning.) You can avoid the
! 1760: copy by using:
! 1761: .TP
! 1762: .B yy_scan_buffer(char *base, yy_size_t size)
! 1763: which scans in place the buffer starting at
! 1764: .I base,
! 1765: consisting of
! 1766: .I size
! 1767: bytes, the last two bytes of which
! 1768: .I must
! 1769: be
! 1770: .B YY_END_OF_BUFFER_CHAR
! 1771: (ASCII NUL).
! 1772: These last two bytes are not scanned; thus, scanning
! 1773: consists of
! 1774: .B base[0]
! 1775: through
! 1776: .B base[size-2],
! 1777: inclusive.
! 1778: .IP
! 1779: If you fail to set up
! 1780: .I base
! 1781: in this manner (i.e., forget the final two
! 1782: .B YY_END_OF_BUFFER_CHAR
! 1783: bytes), then
! 1784: .B yy_scan_buffer()
! 1785: returns a nil pointer instead of creating a new input buffer.
! 1786: .IP
! 1787: The type
! 1788: .B yy_size_t
! 1789: is an integral type to which you can cast an integer expression
! 1790: reflecting the size of the buffer.
! 1791: .SH END-OF-FILE RULES
! 1792: The special rule "<<EOF>>" indicates
! 1793: actions which are to be taken when an end-of-file is
! 1794: encountered and yywrap() returns non-zero (i.e., indicates
! 1795: no further files to process). The action must finish
! 1796: by doing one of four things:
! 1797: .IP -
! 1798: assigning
! 1799: .I yyin
! 1800: to a new input file (in previous versions of flex, after doing the
! 1801: assignment you had to call the special action
! 1802: .B YY_NEW_FILE;
! 1803: this is no longer necessary);
! 1804: .IP -
! 1805: executing a
! 1806: .I return
! 1807: statement;
! 1808: .IP -
! 1809: executing the special
! 1810: .B yyterminate()
! 1811: action;
! 1812: .IP -
! 1813: or, switching to a new buffer using
! 1814: .B yy_switch_to_buffer()
! 1815: as shown in the example above.
! 1816: .PP
! 1817: <<EOF>> rules may not be used with other
! 1818: patterns; they may only be qualified with a list of start
! 1819: conditions. If an unqualified <<EOF>> rule is given, it
! 1820: applies to
! 1821: .I all
! 1822: start conditions which do not already have <<EOF>> actions. To
! 1823: specify an <<EOF>> rule for only the initial start condition, use
! 1824: .nf
! 1825:
! 1826: <INITIAL><<EOF>>
! 1827:
! 1828: .fi
! 1829: .PP
! 1830: These rules are useful for catching things like unclosed comments.
! 1831: An example:
! 1832: .nf
! 1833:
! 1834: %x quote
! 1835: %%
! 1836:
! 1837: ...other rules for dealing with quotes...
! 1838:
! 1839: <quote><<EOF>> {
! 1840: error( "unterminated quote" );
! 1841: yyterminate();
! 1842: }
! 1843: <<EOF>> {
! 1844: if ( *++filelist )
! 1845: yyin = fopen( *filelist, "r" );
! 1846: else
! 1847: yyterminate();
! 1848: }
! 1849:
! 1850: .fi
! 1851: .SH MISCELLANEOUS MACROS
! 1852: The macro
! 1853: .B YY_USER_ACTION
! 1854: can be defined to provide an action
! 1855: which is always executed prior to the matched rule's action. For example,
! 1856: it could be #define'd to call a routine to convert yytext to lower-case.
! 1857: When
! 1858: .B YY_USER_ACTION
! 1859: is invoked, the variable
! 1860: .I yy_act
! 1861: gives the number of the matched rule (rules are numbered starting with 1).
! 1862: Suppose you want to profile how often each of your rules is matched. The
! 1863: following would do the trick:
! 1864: .nf
! 1865:
! 1866: #define YY_USER_ACTION ++ctr[yy_act]
! 1867:
! 1868: .fi
! 1869: where
! 1870: .I ctr
! 1871: is an array to hold the counts for the different rules. Note that
! 1872: the macro
! 1873: .B YY_NUM_RULES
! 1874: gives the total number of rules (including the default rule, even if
! 1875: you use
! 1876: .B \-s),
! 1877: so a correct declaration for
! 1878: .I ctr
! 1879: is:
! 1880: .nf
! 1881:
! 1882: int ctr[YY_NUM_RULES];
! 1883:
! 1884: .fi
! 1885: .PP
! 1886: The macro
! 1887: .B YY_USER_INIT
! 1888: may be defined to provide an action which is always executed before
! 1889: the first scan (and before the scanner's internal initializations are done).
! 1890: For example, it could be used to call a routine to read
! 1891: in a data table or open a logging file.
! 1892: .PP
! 1893: The macro
! 1894: .B yy_set_interactive(is_interactive)
! 1895: can be used to control whether the current buffer is considered
! 1896: .I interactive.
! 1897: An interactive buffer is processed more slowly,
! 1898: but must be used when the scanner's input source is indeed
! 1899: interactive to avoid problems due to waiting to fill buffers
! 1900: (see the discussion of the
! 1901: .B \-I
! 1902: flag below). A non-zero value
! 1903: in the macro invocation marks the buffer as interactive, a zero
! 1904: value as non-interactive. Note that use of this macro overrides
! 1905: .B %option always-interactive
! 1906: or
! 1907: .B %option never-interactive
! 1908: (see Options below).
! 1909: .B yy_set_interactive()
! 1910: must be invoked prior to beginning to scan the buffer that is
! 1911: (or is not) to be considered interactive.
! 1912: .PP
! 1913: The macro
! 1914: .B yy_set_bol(at_bol)
! 1915: can be used to control whether the current buffer's scanning
! 1916: context for the next token match is done as though at the
! 1917: beginning of a line. A non-zero macro argument makes rules anchored with
! 1918: '^' active, while a zero argument makes '^' rules inactive.
! 1919: .PP
! 1920: The macro
! 1921: .B YY_AT_BOL()
! 1922: returns true if the next token scanned from the current buffer
! 1923: will have '^' rules active, false otherwise.
! 1924: .PP
! 1925: In the generated scanner, the actions are all gathered in one large
! 1926: switch statement and separated using
! 1927: .B YY_BREAK,
! 1928: which may be redefined. By default, it is simply a "break", to separate
! 1929: each rule's action from the following rule's.
! 1930: Redefining
! 1931: .B YY_BREAK
! 1932: allows, for example, C++ users to
! 1933: #define YY_BREAK to do nothing (while being very careful that every
! 1934: rule ends with a "break" or a "return"!) to avoid suffering from
! 1935: unreachable statement warnings where because a rule's action ends with
! 1936: "return", the
! 1937: .B YY_BREAK
! 1938: is inaccessible.
! 1939: .SH VALUES AVAILABLE TO THE USER
! 1940: This section summarizes the various values available to the user
! 1941: in the rule actions.
! 1942: .IP -
! 1943: .B char *yytext
! 1944: holds the text of the current token. It may be modified but not lengthened
! 1945: (you cannot append characters to the end).
! 1946: .IP
! 1947: If the special directive
! 1948: .B %array
! 1949: appears in the first section of the scanner description, then
! 1950: .B yytext
! 1951: is instead declared
! 1952: .B char yytext[YYLMAX],
! 1953: where
! 1954: .B YYLMAX
! 1955: is a macro definition that you can redefine in the first section
! 1956: if you don't like the default value (generally 8KB). Using
! 1957: .B %array
! 1958: results in somewhat slower scanners, but the value of
! 1959: .B yytext
! 1960: becomes immune to calls to
! 1961: .I input()
! 1962: and
! 1963: .I unput(),
! 1964: which potentially destroy its value when
! 1965: .B yytext
! 1966: is a character pointer. The opposite of
! 1967: .B %array
! 1968: is
! 1969: .B %pointer,
! 1970: which is the default.
! 1971: .IP
! 1972: You cannot use
! 1973: .B %array
! 1974: when generating C++ scanner classes
! 1975: (the
! 1976: .B \-+
! 1977: flag).
! 1978: .IP -
! 1979: .B int yyleng
! 1980: holds the length of the current token.
! 1981: .IP -
! 1982: .B FILE *yyin
! 1983: is the file which by default
! 1984: .I flex
! 1985: reads from. It may be redefined but doing so only makes sense before
! 1986: scanning begins or after an EOF has been encountered. Changing it in
! 1987: the midst of scanning will have unexpected results since
! 1988: .I flex
! 1989: buffers its input; use
! 1990: .B yyrestart()
! 1991: instead.
! 1992: Once scanning terminates because an end-of-file
! 1993: has been seen, you can assign
! 1994: .I yyin
! 1995: at the new input file and then call the scanner again to continue scanning.
! 1996: .IP -
! 1997: .B void yyrestart( FILE *new_file )
! 1998: may be called to point
! 1999: .I yyin
! 2000: at the new input file. The switch-over to the new file is immediate
! 2001: (any previously buffered-up input is lost). Note that calling
! 2002: .B yyrestart()
! 2003: with
! 2004: .I yyin
! 2005: as an argument thus throws away the current input buffer and continues
! 2006: scanning the same input file.
! 2007: .IP -
! 2008: .B FILE *yyout
! 2009: is the file to which
! 2010: .B ECHO
! 2011: actions are done. It can be reassigned by the user.
! 2012: .IP -
! 2013: .B YY_CURRENT_BUFFER
! 2014: returns a
! 2015: .B YY_BUFFER_STATE
! 2016: handle to the current buffer.
! 2017: .IP -
! 2018: .B YY_START
! 2019: returns an integer value corresponding to the current start
! 2020: condition. You can subsequently use this value with
! 2021: .B BEGIN
! 2022: to return to that start condition.
! 2023: .SH INTERFACING WITH YACC
! 2024: One of the main uses of
! 2025: .I flex
! 2026: is as a companion to the
! 2027: .I yacc
! 2028: parser-generator.
! 2029: .I yacc
! 2030: parsers expect to call a routine named
! 2031: .B yylex()
! 2032: to find the next input token. The routine is supposed to
! 2033: return the type of the next token as well as putting any associated
! 2034: value in the global
! 2035: .B yylval.
! 2036: To use
! 2037: .I flex
! 2038: with
! 2039: .I yacc,
! 2040: one specifies the
! 2041: .B \-d
! 2042: option to
! 2043: .I yacc
! 2044: to instruct it to generate the file
! 2045: .B y.tab.h
! 2046: containing definitions of all the
! 2047: .B %tokens
! 2048: appearing in the
! 2049: .I yacc
! 2050: input. This file is then included in the
! 2051: .I flex
! 2052: scanner. For example, if one of the tokens is "TOK_NUMBER",
! 2053: part of the scanner might look like:
! 2054: .nf
! 2055:
! 2056: %{
! 2057: #include "y.tab.h"
! 2058: %}
! 2059:
! 2060: %%
! 2061:
! 2062: [0-9]+ yylval = atoi( yytext ); return TOK_NUMBER;
! 2063:
! 2064: .fi
! 2065: .SH OPTIONS
! 2066: .I flex
! 2067: has the following options:
! 2068: .TP
! 2069: .B \-b
! 2070: Generate backing-up information to
! 2071: .I lex.backup.
! 2072: This is a list of scanner states which require backing up
! 2073: and the input characters on which they do so. By adding rules one
! 2074: can remove backing-up states. If
! 2075: .I all
! 2076: backing-up states are eliminated and
! 2077: .B \-Cf
! 2078: or
! 2079: .B \-CF
! 2080: is used, the generated scanner will run faster (see the
! 2081: .B \-p
! 2082: flag). Only users who wish to squeeze every last cycle out of their
! 2083: scanners need worry about this option. (See the section on Performance
! 2084: Considerations below.)
! 2085: .TP
! 2086: .B \-c
! 2087: is a do-nothing, deprecated option included for POSIX compliance.
! 2088: .TP
! 2089: .B \-d
! 2090: makes the generated scanner run in
! 2091: .I debug
! 2092: mode. Whenever a pattern is recognized and the global
! 2093: .B yy_flex_debug
! 2094: is non-zero (which is the default),
! 2095: the scanner will write to
! 2096: .I stderr
! 2097: a line of the form:
! 2098: .nf
! 2099:
! 2100: --accepting rule at line 53 ("the matched text")
! 2101:
! 2102: .fi
! 2103: The line number refers to the location of the rule in the file
! 2104: defining the scanner (i.e., the file that was fed to flex). Messages
! 2105: are also generated when the scanner backs up, accepts the
! 2106: default rule, reaches the end of its input buffer (or encounters
! 2107: a NUL; at this point, the two look the same as far as the scanner's concerned),
! 2108: or reaches an end-of-file.
! 2109: .TP
! 2110: .B \-f
! 2111: specifies
! 2112: .I fast scanner.
! 2113: No table compression is done and stdio is bypassed.
! 2114: The result is large but fast. This option is equivalent to
! 2115: .B \-Cfr
! 2116: (see below).
! 2117: .TP
! 2118: .B \-h
! 2119: generates a "help" summary of
! 2120: .I flex's
! 2121: options to
! 2122: .I stdout
! 2123: and then exits.
! 2124: .B \-?
! 2125: and
! 2126: .B \-\-help
! 2127: are synonyms for
! 2128: .B \-h.
! 2129: .TP
! 2130: .B \-i
! 2131: instructs
! 2132: .I flex
! 2133: to generate a
! 2134: .I case-insensitive
! 2135: scanner. The case of letters given in the
! 2136: .I flex
! 2137: input patterns will
! 2138: be ignored, and tokens in the input will be matched regardless of case. The
! 2139: matched text given in
! 2140: .I yytext
! 2141: will have the preserved case (i.e., it will not be folded).
! 2142: .TP
! 2143: .B \-l
! 2144: turns on maximum compatibility with the original AT&T
! 2145: .I lex
! 2146: implementation. Note that this does not mean
! 2147: .I full
! 2148: compatibility. Use of this option costs a considerable amount of
! 2149: performance, and it cannot be used with the
! 2150: .B \-+, -f, -F, -Cf,
! 2151: or
! 2152: .B -CF
! 2153: options. For details on the compatibilities it provides, see the section
! 2154: "Incompatibilities With Lex And POSIX" below. This option also results
! 2155: in the name
! 2156: .B YY_FLEX_LEX_COMPAT
! 2157: being #define'd in the generated scanner.
! 2158: .TP
! 2159: .B \-n
! 2160: is another do-nothing, deprecated option included only for
! 2161: POSIX compliance.
! 2162: .TP
! 2163: .B \-p
! 2164: generates a performance report to stderr. The report
! 2165: consists of comments regarding features of the
! 2166: .I flex
! 2167: input file which will cause a serious loss of performance in the resulting
! 2168: scanner. If you give the flag twice, you will also get comments regarding
! 2169: features that lead to minor performance losses.
! 2170: .IP
! 2171: Note that the use of
! 2172: .B REJECT,
! 2173: .B %option yylineno,
! 2174: and variable trailing context (see the Deficiencies / Bugs section below)
! 2175: entails a substantial performance penalty; use of
! 2176: .I yymore(),
! 2177: the
! 2178: .B ^
! 2179: operator,
! 2180: and the
! 2181: .B \-I
! 2182: flag entail minor performance penalties.
! 2183: .TP
! 2184: .B \-s
! 2185: causes the
! 2186: .I default rule
! 2187: (that unmatched scanner input is echoed to
! 2188: .I stdout)
! 2189: to be suppressed. If the scanner encounters input that does not
! 2190: match any of its rules, it aborts with an error. This option is
! 2191: useful for finding holes in a scanner's rule set.
! 2192: .TP
! 2193: .B \-t
! 2194: instructs
! 2195: .I flex
! 2196: to write the scanner it generates to standard output instead
! 2197: of
! 2198: .B lex.yy.c.
! 2199: .TP
! 2200: .B \-v
! 2201: specifies that
! 2202: .I flex
! 2203: should write to
! 2204: .I stderr
! 2205: a summary of statistics regarding the scanner it generates.
! 2206: Most of the statistics are meaningless to the casual
! 2207: .I flex
! 2208: user, but the first line identifies the version of
! 2209: .I flex
! 2210: (same as reported by
! 2211: .B \-V),
! 2212: and the next line the flags used when generating the scanner, including
! 2213: those that are on by default.
! 2214: .TP
! 2215: .B \-w
! 2216: suppresses warning messages.
! 2217: .TP
! 2218: .B \-B
! 2219: instructs
! 2220: .I flex
! 2221: to generate a
! 2222: .I batch
! 2223: scanner, the opposite of
! 2224: .I interactive
! 2225: scanners generated by
! 2226: .B \-I
! 2227: (see below). In general, you use
! 2228: .B \-B
! 2229: when you are
! 2230: .I certain
! 2231: that your scanner will never be used interactively, and you want to
! 2232: squeeze a
! 2233: .I little
! 2234: more performance out of it. If your goal is instead to squeeze out a
! 2235: .I lot
! 2236: more performance, you should be using the
! 2237: .B \-Cf
! 2238: or
! 2239: .B \-CF
! 2240: options (discussed below), which turn on
! 2241: .B \-B
! 2242: automatically anyway.
! 2243: .TP
! 2244: .B \-F
! 2245: specifies that the
! 2246: .ul
! 2247: fast
! 2248: scanner table representation should be used (and stdio
! 2249: bypassed). This representation is
! 2250: about as fast as the full table representation
! 2251: .B (-f),
! 2252: and for some sets of patterns will be considerably smaller (and for
! 2253: others, larger). In general, if the pattern set contains both "keywords"
! 2254: and a catch-all, "identifier" rule, such as in the set:
! 2255: .nf
! 2256:
! 2257: "case" return TOK_CASE;
! 2258: "switch" return TOK_SWITCH;
! 2259: ...
! 2260: "default" return TOK_DEFAULT;
! 2261: [a-z]+ return TOK_ID;
! 2262:
! 2263: .fi
! 2264: then you're better off using the full table representation. If only
! 2265: the "identifier" rule is present and you then use a hash table or some such
! 2266: to detect the keywords, you're better off using
! 2267: .B -F.
! 2268: .IP
! 2269: This option is equivalent to
! 2270: .B \-CFr
! 2271: (see below). It cannot be used with
! 2272: .B \-+.
! 2273: .TP
! 2274: .B \-I
! 2275: instructs
! 2276: .I flex
! 2277: to generate an
! 2278: .I interactive
! 2279: scanner. An interactive scanner is one that only looks ahead to decide
! 2280: what token has been matched if it absolutely must. It turns out that
! 2281: always looking one extra character ahead, even if the scanner has already
! 2282: seen enough text to disambiguate the current token, is a bit faster than
! 2283: only looking ahead when necessary. But scanners that always look ahead
! 2284: give dreadful interactive performance; for example, when a user types
! 2285: a newline, it is not recognized as a newline token until they enter
! 2286: .I another
! 2287: token, which often means typing in another whole line.
! 2288: .IP
! 2289: .I Flex
! 2290: scanners default to
! 2291: .I interactive
! 2292: unless you use the
! 2293: .B \-Cf
! 2294: or
! 2295: .B \-CF
! 2296: table-compression options (see below). That's because if you're looking
! 2297: for high-performance you should be using one of these options, so if you
! 2298: didn't,
! 2299: .I flex
! 2300: assumes you'd rather trade off a bit of run-time performance for intuitive
! 2301: interactive behavior. Note also that you
! 2302: .I cannot
! 2303: use
! 2304: .B \-I
! 2305: in conjunction with
! 2306: .B \-Cf
! 2307: or
! 2308: .B \-CF.
! 2309: Thus, this option is not really needed; it is on by default for all those
! 2310: cases in which it is allowed.
! 2311: .IP
! 2312: You can force a scanner to
! 2313: .I not
! 2314: be interactive by using
! 2315: .B \-B
! 2316: (see above).
! 2317: .TP
! 2318: .B \-L
! 2319: instructs
! 2320: .I flex
! 2321: not to generate
! 2322: .B #line
! 2323: directives. Without this option,
! 2324: .I flex
! 2325: peppers the generated scanner
! 2326: with #line directives so error messages in the actions will be correctly
! 2327: located with respect to either the original
! 2328: .I flex
! 2329: input file (if the errors are due to code in the input file), or
! 2330: .B lex.yy.c
! 2331: (if the errors are
! 2332: .I flex's
! 2333: fault -- you should report these sorts of errors to the email address
! 2334: given below).
! 2335: .TP
! 2336: .B \-T
! 2337: makes
! 2338: .I flex
! 2339: run in
! 2340: .I trace
! 2341: mode. It will generate a lot of messages to
! 2342: .I stderr
! 2343: concerning
! 2344: the form of the input and the resultant non-deterministic and deterministic
! 2345: finite automata. This option is mostly for use in maintaining
! 2346: .I flex.
! 2347: .TP
! 2348: .B \-V
! 2349: prints the version number to
! 2350: .I stdout
! 2351: and exits.
! 2352: .B \-\-version
! 2353: is a synonym for
! 2354: .B \-V.
! 2355: .TP
! 2356: .B \-7
! 2357: instructs
! 2358: .I flex
! 2359: to generate a 7-bit scanner, i.e., one which can only recognized 7-bit
! 2360: characters in its input. The advantage of using
! 2361: .B \-7
! 2362: is that the scanner's tables can be up to half the size of those generated
! 2363: using the
! 2364: .B \-8
! 2365: option (see below). The disadvantage is that such scanners often hang
! 2366: or crash if their input contains an 8-bit character.
! 2367: .IP
! 2368: Note, however, that unless you generate your scanner using the
! 2369: .B \-Cf
! 2370: or
! 2371: .B \-CF
! 2372: table compression options, use of
! 2373: .B \-7
! 2374: will save only a small amount of table space, and make your scanner
! 2375: considerably less portable.
! 2376: .I Flex's
! 2377: default behavior is to generate an 8-bit scanner unless you use the
! 2378: .B \-Cf
! 2379: or
! 2380: .B \-CF,
! 2381: in which case
! 2382: .I flex
! 2383: defaults to generating 7-bit scanners unless your site was always
! 2384: configured to generate 8-bit scanners (as will often be the case
! 2385: with non-USA sites). You can tell whether flex generated a 7-bit
! 2386: or an 8-bit scanner by inspecting the flag summary in the
! 2387: .B \-v
! 2388: output as described above.
! 2389: .IP
! 2390: Note that if you use
! 2391: .B \-Cfe
! 2392: or
! 2393: .B \-CFe
! 2394: (those table compression options, but also using equivalence classes as
! 2395: discussed see below), flex still defaults to generating an 8-bit
! 2396: scanner, since usually with these compression options full 8-bit tables
! 2397: are not much more expensive than 7-bit tables.
! 2398: .TP
! 2399: .B \-8
! 2400: instructs
! 2401: .I flex
! 2402: to generate an 8-bit scanner, i.e., one which can recognize 8-bit
! 2403: characters. This flag is only needed for scanners generated using
! 2404: .B \-Cf
! 2405: or
! 2406: .B \-CF,
! 2407: as otherwise flex defaults to generating an 8-bit scanner anyway.
! 2408: .IP
! 2409: See the discussion of
! 2410: .B \-7
! 2411: above for flex's default behavior and the tradeoffs between 7-bit
! 2412: and 8-bit scanners.
! 2413: .TP
! 2414: .B \-+
! 2415: specifies that you want flex to generate a C++
! 2416: scanner class. See the section on Generating C++ Scanners below for
! 2417: details.
! 2418: .TP
! 2419: .B \-C[aefFmr]
! 2420: controls the degree of table compression and, more generally, trade-offs
! 2421: between small scanners and fast scanners.
! 2422: .IP
! 2423: .B \-Ca
! 2424: ("align") instructs flex to trade off larger tables in the
! 2425: generated scanner for faster performance because the elements of
! 2426: the tables are better aligned for memory access and computation. On some
! 2427: RISC architectures, fetching and manipulating longwords is more efficient
! 2428: than with smaller-sized units such as shortwords. This option can
! 2429: double the size of the tables used by your scanner.
! 2430: .IP
! 2431: .B \-Ce
! 2432: directs
! 2433: .I flex
! 2434: to construct
! 2435: .I equivalence classes,
! 2436: i.e., sets of characters
! 2437: which have identical lexical properties (for example, if the only
! 2438: appearance of digits in the
! 2439: .I flex
! 2440: input is in the character class
! 2441: "[0-9]" then the digits '0', '1', ..., '9' will all be put
! 2442: in the same equivalence class). Equivalence classes usually give
! 2443: dramatic reductions in the final table/object file sizes (typically
! 2444: a factor of 2-5) and are pretty cheap performance-wise (one array
! 2445: look-up per character scanned).
! 2446: .IP
! 2447: .B \-Cf
! 2448: specifies that the
! 2449: .I full
! 2450: scanner tables should be generated -
! 2451: .I flex
! 2452: should not compress the
! 2453: tables by taking advantages of similar transition functions for
! 2454: different states.
! 2455: .IP
! 2456: .B \-CF
! 2457: specifies that the alternate fast scanner representation (described
! 2458: above under the
! 2459: .B \-F
! 2460: flag)
! 2461: should be used. This option cannot be used with
! 2462: .B \-+.
! 2463: .IP
! 2464: .B \-Cm
! 2465: directs
! 2466: .I flex
! 2467: to construct
! 2468: .I meta-equivalence classes,
! 2469: which are sets of equivalence classes (or characters, if equivalence
! 2470: classes are not being used) that are commonly used together. Meta-equivalence
! 2471: classes are often a big win when using compressed tables, but they
! 2472: have a moderate performance impact (one or two "if" tests and one
! 2473: array look-up per character scanned).
! 2474: .IP
! 2475: .B \-Cr
! 2476: causes the generated scanner to
! 2477: .I bypass
! 2478: use of the standard I/O library (stdio) for input. Instead of calling
! 2479: .B fread()
! 2480: or
! 2481: .B getc(),
! 2482: the scanner will use the
! 2483: .B read()
! 2484: system call, resulting in a performance gain which varies from system
! 2485: to system, but in general is probably negligible unless you are also using
! 2486: .B \-Cf
! 2487: or
! 2488: .B \-CF.
! 2489: Using
! 2490: .B \-Cr
! 2491: can cause strange behavior if, for example, you read from
! 2492: .I yyin
! 2493: using stdio prior to calling the scanner (because the scanner will miss
! 2494: whatever text your previous reads left in the stdio input buffer).
! 2495: .IP
! 2496: .B \-Cr
! 2497: has no effect if you define
! 2498: .B YY_INPUT
! 2499: (see The Generated Scanner above).
! 2500: .IP
! 2501: A lone
! 2502: .B \-C
! 2503: specifies that the scanner tables should be compressed but neither
! 2504: equivalence classes nor meta-equivalence classes should be used.
! 2505: .IP
! 2506: The options
! 2507: .B \-Cf
! 2508: or
! 2509: .B \-CF
! 2510: and
! 2511: .B \-Cm
! 2512: do not make sense together - there is no opportunity for meta-equivalence
! 2513: classes if the table is not being compressed. Otherwise the options
! 2514: may be freely mixed, and are cumulative.
! 2515: .IP
! 2516: The default setting is
! 2517: .B \-Cem,
! 2518: which specifies that
! 2519: .I flex
! 2520: should generate equivalence classes
! 2521: and meta-equivalence classes. This setting provides the highest
! 2522: degree of table compression. You can trade off
! 2523: faster-executing scanners at the cost of larger tables with
! 2524: the following generally being true:
! 2525: .nf
! 2526:
! 2527: slowest & smallest
! 2528: -Cem
! 2529: -Cm
! 2530: -Ce
! 2531: -C
! 2532: -C{f,F}e
! 2533: -C{f,F}
! 2534: -C{f,F}a
! 2535: fastest & largest
! 2536:
! 2537: .fi
! 2538: Note that scanners with the smallest tables are usually generated and
! 2539: compiled the quickest, so
! 2540: during development you will usually want to use the default, maximal
! 2541: compression.
! 2542: .IP
! 2543: .B \-Cfe
! 2544: is often a good compromise between speed and size for production
! 2545: scanners.
! 2546: .TP
! 2547: .B \-ooutput
! 2548: directs flex to write the scanner to the file
! 2549: .B output
! 2550: instead of
! 2551: .B lex.yy.c.
! 2552: If you combine
! 2553: .B \-o
! 2554: with the
! 2555: .B \-t
! 2556: option, then the scanner is written to
! 2557: .I stdout
! 2558: but its
! 2559: .B #line
! 2560: directives (see the
! 2561: .B \\-L
! 2562: option above) refer to the file
! 2563: .B output.
! 2564: .TP
! 2565: .B \-Pprefix
! 2566: changes the default
! 2567: .I "yy"
! 2568: prefix used by
! 2569: .I flex
! 2570: for all globally-visible variable and function names to instead be
! 2571: .I prefix.
! 2572: For example,
! 2573: .B \-Pfoo
! 2574: changes the name of
! 2575: .B yytext
! 2576: to
! 2577: .B footext.
! 2578: It also changes the name of the default output file from
! 2579: .B lex.yy.c
! 2580: to
! 2581: .B lex.foo.c.
! 2582: Here are all of the names affected:
! 2583: .nf
! 2584:
! 2585: yy_create_buffer
! 2586: yy_delete_buffer
! 2587: yy_flex_debug
! 2588: yy_init_buffer
! 2589: yy_flush_buffer
! 2590: yy_load_buffer_state
! 2591: yy_switch_to_buffer
! 2592: yyin
! 2593: yyleng
! 2594: yylex
! 2595: yylineno
! 2596: yyout
! 2597: yyrestart
! 2598: yytext
! 2599: yywrap
! 2600:
! 2601: .fi
! 2602: (If you are using a C++ scanner, then only
! 2603: .B yywrap
! 2604: and
! 2605: .B yyFlexLexer
! 2606: are affected.)
! 2607: Within your scanner itself, you can still refer to the global variables
! 2608: and functions using either version of their name; but externally, they
! 2609: have the modified name.
! 2610: .IP
! 2611: This option lets you easily link together multiple
! 2612: .I flex
! 2613: programs into the same executable. Note, though, that using this
! 2614: option also renames
! 2615: .B yywrap(),
! 2616: so you now
! 2617: .I must
! 2618: either
! 2619: provide your own (appropriately-named) version of the routine for your
! 2620: scanner, or use
! 2621: .B %option noyywrap,
! 2622: as linking with
! 2623: .B \-lfl
! 2624: no longer provides one for you by default.
! 2625: .TP
! 2626: .B \-Sskeleton_file
! 2627: overrides the default skeleton file from which
! 2628: .I flex
! 2629: constructs its scanners. You'll never need this option unless you are doing
! 2630: .I flex
! 2631: maintenance or development.
! 2632: .PP
! 2633: .I flex
! 2634: also provides a mechanism for controlling options within the
! 2635: scanner specification itself, rather than from the flex command-line.
! 2636: This is done by including
! 2637: .B %option
! 2638: directives in the first section of the scanner specification.
! 2639: You can specify multiple options with a single
! 2640: .B %option
! 2641: directive, and multiple directives in the first section of your flex input
! 2642: file.
! 2643: .PP
! 2644: Most options are given simply as names, optionally preceded by the
! 2645: word "no" (with no intervening whitespace) to negate their meaning.
! 2646: A number are equivalent to flex flags or their negation:
! 2647: .nf
! 2648:
! 2649: 7bit -7 option
! 2650: 8bit -8 option
! 2651: align -Ca option
! 2652: backup -b option
! 2653: batch -B option
! 2654: c++ -+ option
! 2655:
! 2656: caseful or
! 2657: case-sensitive opposite of -i (default)
! 2658:
! 2659: case-insensitive or
! 2660: caseless -i option
! 2661:
! 2662: debug -d option
! 2663: default opposite of -s option
! 2664: ecs -Ce option
! 2665: fast -F option
! 2666: full -f option
! 2667: interactive -I option
! 2668: lex-compat -l option
! 2669: meta-ecs -Cm option
! 2670: perf-report -p option
! 2671: read -Cr option
! 2672: stdout -t option
! 2673: verbose -v option
! 2674: warn opposite of -w option
! 2675: (use "%option nowarn" for -w)
! 2676:
! 2677: array equivalent to "%array"
! 2678: pointer equivalent to "%pointer" (default)
! 2679:
! 2680: .fi
! 2681: Some
! 2682: .B %option's
! 2683: provide features otherwise not available:
! 2684: .TP
! 2685: .B always-interactive
! 2686: instructs flex to generate a scanner which always considers its input
! 2687: "interactive". Normally, on each new input file the scanner calls
! 2688: .B isatty()
! 2689: in an attempt to determine whether
! 2690: the scanner's input source is interactive and thus should be read a
! 2691: character at a time. When this option is used, however, then no
! 2692: such call is made.
! 2693: .TP
! 2694: .B main
! 2695: directs flex to provide a default
! 2696: .B main()
! 2697: program for the scanner, which simply calls
! 2698: .B yylex().
! 2699: This option implies
! 2700: .B noyywrap
! 2701: (see below).
! 2702: .TP
! 2703: .B never-interactive
! 2704: instructs flex to generate a scanner which never considers its input
! 2705: "interactive" (again, no call made to
! 2706: .B isatty()).
! 2707: This is the opposite of
! 2708: .B always-interactive.
! 2709: .TP
! 2710: .B stack
! 2711: enables the use of start condition stacks (see Start Conditions above).
! 2712: .TP
! 2713: .B stdinit
! 2714: if set (i.e.,
! 2715: .B %option stdinit)
! 2716: initializes
! 2717: .I yyin
! 2718: and
! 2719: .I yyout
! 2720: to
! 2721: .I stdin
! 2722: and
! 2723: .I stdout,
! 2724: instead of the default of
! 2725: .I nil.
! 2726: Some existing
! 2727: .I lex
! 2728: programs depend on this behavior, even though it is not compliant with
! 2729: ANSI C, which does not require
! 2730: .I stdin
! 2731: and
! 2732: .I stdout
! 2733: to be compile-time constant.
! 2734: .TP
! 2735: .B yylineno
! 2736: directs
! 2737: .I flex
! 2738: to generate a scanner that maintains the number of the current line
! 2739: read from its input in the global variable
! 2740: .B yylineno.
! 2741: This option is implied by
! 2742: .B %option lex-compat.
! 2743: .TP
! 2744: .B yywrap
! 2745: if unset (i.e.,
! 2746: .B %option noyywrap),
! 2747: makes the scanner not call
! 2748: .B yywrap()
! 2749: upon an end-of-file, but simply assume that there are no more
! 2750: files to scan (until the user points
! 2751: .I yyin
! 2752: at a new file and calls
! 2753: .B yylex()
! 2754: again).
! 2755: .PP
! 2756: .I flex
! 2757: scans your rule actions to determine whether you use the
! 2758: .B REJECT
! 2759: or
! 2760: .B yymore()
! 2761: features. The
! 2762: .B reject
! 2763: and
! 2764: .B yymore
! 2765: options are available to override its decision as to whether you use the
! 2766: options, either by setting them (e.g.,
! 2767: .B %option reject)
! 2768: to indicate the feature is indeed used, or
! 2769: unsetting them to indicate it actually is not used
! 2770: (e.g.,
! 2771: .B %option noyymore).
! 2772: .PP
! 2773: Three options take string-delimited values, offset with '=':
! 2774: .nf
! 2775:
! 2776: %option outfile="ABC"
! 2777:
! 2778: .fi
! 2779: is equivalent to
! 2780: .B -oABC,
! 2781: and
! 2782: .nf
! 2783:
! 2784: %option prefix="XYZ"
! 2785:
! 2786: .fi
! 2787: is equivalent to
! 2788: .B -PXYZ.
! 2789: Finally,
! 2790: .nf
! 2791:
! 2792: %option yyclass="foo"
! 2793:
! 2794: .fi
! 2795: only applies when generating a C++ scanner (
! 2796: .B \-+
! 2797: option). It informs
! 2798: .I flex
! 2799: that you have derived
! 2800: .B foo
! 2801: as a subclass of
! 2802: .B yyFlexLexer,
! 2803: so
! 2804: .I flex
! 2805: will place your actions in the member function
! 2806: .B foo::yylex()
! 2807: instead of
! 2808: .B yyFlexLexer::yylex().
! 2809: It also generates a
! 2810: .B yyFlexLexer::yylex()
! 2811: member function that emits a run-time error (by invoking
! 2812: .B yyFlexLexer::LexerError())
! 2813: if called.
! 2814: See Generating C++ Scanners, below, for additional information.
! 2815: .PP
! 2816: A number of options are available for lint purists who want to suppress
! 2817: the appearance of unneeded routines in the generated scanner. Each of the
! 2818: following, if unset
! 2819: (e.g.,
! 2820: .B %option nounput
! 2821: ), results in the corresponding routine not appearing in
! 2822: the generated scanner:
! 2823: .nf
! 2824:
! 2825: input, unput
! 2826: yy_push_state, yy_pop_state, yy_top_state
! 2827: yy_scan_buffer, yy_scan_bytes, yy_scan_string
! 2828:
! 2829: .fi
! 2830: (though
! 2831: .B yy_push_state()
! 2832: and friends won't appear anyway unless you use
! 2833: .B %option stack).
! 2834: .SH PERFORMANCE CONSIDERATIONS
! 2835: The main design goal of
! 2836: .I flex
! 2837: is that it generate high-performance scanners. It has been optimized
! 2838: for dealing well with large sets of rules. Aside from the effects on
! 2839: scanner speed of the table compression
! 2840: .B \-C
! 2841: options outlined above,
! 2842: there are a number of options/actions which degrade performance. These
! 2843: are, from most expensive to least:
! 2844: .nf
! 2845:
! 2846: REJECT
! 2847: %option yylineno
! 2848: arbitrary trailing context
! 2849:
! 2850: pattern sets that require backing up
! 2851: %array
! 2852: %option interactive
! 2853: %option always-interactive
! 2854:
! 2855: '^' beginning-of-line operator
! 2856: yymore()
! 2857:
! 2858: .fi
! 2859: with the first three all being quite expensive and the last two
! 2860: being quite cheap. Note also that
! 2861: .B unput()
! 2862: is implemented as a routine call that potentially does quite a bit of
! 2863: work, while
! 2864: .B yyless()
! 2865: is a quite-cheap macro; so if just putting back some excess text you
! 2866: scanned, use
! 2867: .B yyless().
! 2868: .PP
! 2869: .B REJECT
! 2870: should be avoided at all costs when performance is important.
! 2871: It is a particularly expensive option.
! 2872: .PP
! 2873: Getting rid of backing up is messy and often may be an enormous
! 2874: amount of work for a complicated scanner. In principal, one begins
! 2875: by using the
! 2876: .B \-b
! 2877: flag to generate a
! 2878: .I lex.backup
! 2879: file. For example, on the input
! 2880: .nf
! 2881:
! 2882: %%
! 2883: foo return TOK_KEYWORD;
! 2884: foobar return TOK_KEYWORD;
! 2885:
! 2886: .fi
! 2887: the file looks like:
! 2888: .nf
! 2889:
! 2890: State #6 is non-accepting -
! 2891: associated rule line numbers:
! 2892: 2 3
! 2893: out-transitions: [ o ]
! 2894: jam-transitions: EOF [ \\001-n p-\\177 ]
! 2895:
! 2896: State #8 is non-accepting -
! 2897: associated rule line numbers:
! 2898: 3
! 2899: out-transitions: [ a ]
! 2900: jam-transitions: EOF [ \\001-` b-\\177 ]
! 2901:
! 2902: State #9 is non-accepting -
! 2903: associated rule line numbers:
! 2904: 3
! 2905: out-transitions: [ r ]
! 2906: jam-transitions: EOF [ \\001-q s-\\177 ]
! 2907:
! 2908: Compressed tables always back up.
! 2909:
! 2910: .fi
! 2911: The first few lines tell us that there's a scanner state in
! 2912: which it can make a transition on an 'o' but not on any other
! 2913: character, and that in that state the currently scanned text does not match
! 2914: any rule. The state occurs when trying to match the rules found
! 2915: at lines 2 and 3 in the input file.
! 2916: If the scanner is in that state and then reads
! 2917: something other than an 'o', it will have to back up to find
! 2918: a rule which is matched. With
! 2919: a bit of headscratching one can see that this must be the
! 2920: state it's in when it has seen "fo". When this has happened,
! 2921: if anything other than another 'o' is seen, the scanner will
! 2922: have to back up to simply match the 'f' (by the default rule).
! 2923: .PP
! 2924: The comment regarding State #8 indicates there's a problem
! 2925: when "foob" has been scanned. Indeed, on any character other
! 2926: than an 'a', the scanner will have to back up to accept "foo".
! 2927: Similarly, the comment for State #9 concerns when "fooba" has
! 2928: been scanned and an 'r' does not follow.
! 2929: .PP
! 2930: The final comment reminds us that there's no point going to
! 2931: all the trouble of removing backing up from the rules unless
! 2932: we're using
! 2933: .B \-Cf
! 2934: or
! 2935: .B \-CF,
! 2936: since there's no performance gain doing so with compressed scanners.
! 2937: .PP
! 2938: The way to remove the backing up is to add "error" rules:
! 2939: .nf
! 2940:
! 2941: %%
! 2942: foo return TOK_KEYWORD;
! 2943: foobar return TOK_KEYWORD;
! 2944:
! 2945: fooba |
! 2946: foob |
! 2947: fo {
! 2948: /* false alarm, not really a keyword */
! 2949: return TOK_ID;
! 2950: }
! 2951:
! 2952: .fi
! 2953: .PP
! 2954: Eliminating backing up among a list of keywords can also be
! 2955: done using a "catch-all" rule:
! 2956: .nf
! 2957:
! 2958: %%
! 2959: foo return TOK_KEYWORD;
! 2960: foobar return TOK_KEYWORD;
! 2961:
! 2962: [a-z]+ return TOK_ID;
! 2963:
! 2964: .fi
! 2965: This is usually the best solution when appropriate.
! 2966: .PP
! 2967: Backing up messages tend to cascade.
! 2968: With a complicated set of rules it's not uncommon to get hundreds
! 2969: of messages. If one can decipher them, though, it often
! 2970: only takes a dozen or so rules to eliminate the backing up (though
! 2971: it's easy to make a mistake and have an error rule accidentally match
! 2972: a valid token. A possible future
! 2973: .I flex
! 2974: feature will be to automatically add rules to eliminate backing up).
! 2975: .PP
! 2976: It's important to keep in mind that you gain the benefits of eliminating
! 2977: backing up only if you eliminate
! 2978: .I every
! 2979: instance of backing up. Leaving just one means you gain nothing.
! 2980: .PP
! 2981: .I Variable
! 2982: trailing context (where both the leading and trailing parts do not have
! 2983: a fixed length) entails almost the same performance loss as
! 2984: .B REJECT
! 2985: (i.e., substantial). So when possible a rule like:
! 2986: .nf
! 2987:
! 2988: %%
! 2989: mouse|rat/(cat|dog) run();
! 2990:
! 2991: .fi
! 2992: is better written:
! 2993: .nf
! 2994:
! 2995: %%
! 2996: mouse/cat|dog run();
! 2997: rat/cat|dog run();
! 2998:
! 2999: .fi
! 3000: or as
! 3001: .nf
! 3002:
! 3003: %%
! 3004: mouse|rat/cat run();
! 3005: mouse|rat/dog run();
! 3006:
! 3007: .fi
! 3008: Note that here the special '|' action does
! 3009: .I not
! 3010: provide any savings, and can even make things worse (see
! 3011: Deficiencies / Bugs below).
! 3012: .LP
! 3013: Another area where the user can increase a scanner's performance
! 3014: (and one that's easier to implement) arises from the fact that
! 3015: the longer the tokens matched, the faster the scanner will run.
! 3016: This is because with long tokens the processing of most input
! 3017: characters takes place in the (short) inner scanning loop, and
! 3018: does not often have to go through the additional work of setting up
! 3019: the scanning environment (e.g.,
! 3020: .B yytext)
! 3021: for the action. Recall the scanner for C comments:
! 3022: .nf
! 3023:
! 3024: %x comment
! 3025: %%
! 3026: int line_num = 1;
! 3027:
! 3028: "/*" BEGIN(comment);
! 3029:
! 3030: <comment>[^*\\n]*
! 3031: <comment>"*"+[^*/\\n]*
! 3032: <comment>\\n ++line_num;
! 3033: <comment>"*"+"/" BEGIN(INITIAL);
! 3034:
! 3035: .fi
! 3036: This could be sped up by writing it as:
! 3037: .nf
! 3038:
! 3039: %x comment
! 3040: %%
! 3041: int line_num = 1;
! 3042:
! 3043: "/*" BEGIN(comment);
! 3044:
! 3045: <comment>[^*\\n]*
! 3046: <comment>[^*\\n]*\\n ++line_num;
! 3047: <comment>"*"+[^*/\\n]*
! 3048: <comment>"*"+[^*/\\n]*\\n ++line_num;
! 3049: <comment>"*"+"/" BEGIN(INITIAL);
! 3050:
! 3051: .fi
! 3052: Now instead of each newline requiring the processing of another
! 3053: action, recognizing the newlines is "distributed" over the other rules
! 3054: to keep the matched text as long as possible. Note that
! 3055: .I adding
! 3056: rules does
! 3057: .I not
! 3058: slow down the scanner! The speed of the scanner is independent
! 3059: of the number of rules or (modulo the considerations given at the
! 3060: beginning of this section) how complicated the rules are with
! 3061: regard to operators such as '*' and '|'.
! 3062: .PP
! 3063: A final example in speeding up a scanner: suppose you want to scan
! 3064: through a file containing identifiers and keywords, one per line
! 3065: and with no other extraneous characters, and recognize all the
! 3066: keywords. A natural first approach is:
! 3067: .nf
! 3068:
! 3069: %%
! 3070: asm |
! 3071: auto |
! 3072: break |
! 3073: ... etc ...
! 3074: volatile |
! 3075: while /* it's a keyword */
! 3076:
! 3077: .|\\n /* it's not a keyword */
! 3078:
! 3079: .fi
! 3080: To eliminate the back-tracking, introduce a catch-all rule:
! 3081: .nf
! 3082:
! 3083: %%
! 3084: asm |
! 3085: auto |
! 3086: break |
! 3087: ... etc ...
! 3088: volatile |
! 3089: while /* it's a keyword */
! 3090:
! 3091: [a-z]+ |
! 3092: .|\\n /* it's not a keyword */
! 3093:
! 3094: .fi
! 3095: Now, if it's guaranteed that there's exactly one word per line,
! 3096: then we can reduce the total number of matches by a half by
! 3097: merging in the recognition of newlines with that of the other
! 3098: tokens:
! 3099: .nf
! 3100:
! 3101: %%
! 3102: asm\\n |
! 3103: auto\\n |
! 3104: break\\n |
! 3105: ... etc ...
! 3106: volatile\\n |
! 3107: while\\n /* it's a keyword */
! 3108:
! 3109: [a-z]+\\n |
! 3110: .|\\n /* it's not a keyword */
! 3111:
! 3112: .fi
! 3113: One has to be careful here, as we have now reintroduced backing up
! 3114: into the scanner. In particular, while
! 3115: .I we
! 3116: know that there will never be any characters in the input stream
! 3117: other than letters or newlines,
! 3118: .I flex
! 3119: can't figure this out, and it will plan for possibly needing to back up
! 3120: when it has scanned a token like "auto" and then the next character
! 3121: is something other than a newline or a letter. Previously it would
! 3122: then just match the "auto" rule and be done, but now it has no "auto"
! 3123: rule, only a "auto\\n" rule. To eliminate the possibility of backing up,
! 3124: we could either duplicate all rules but without final newlines, or,
! 3125: since we never expect to encounter such an input and therefore don't
! 3126: how it's classified, we can introduce one more catch-all rule, this
! 3127: one which doesn't include a newline:
! 3128: .nf
! 3129:
! 3130: %%
! 3131: asm\\n |
! 3132: auto\\n |
! 3133: break\\n |
! 3134: ... etc ...
! 3135: volatile\\n |
! 3136: while\\n /* it's a keyword */
! 3137:
! 3138: [a-z]+\\n |
! 3139: [a-z]+ |
! 3140: .|\\n /* it's not a keyword */
! 3141:
! 3142: .fi
! 3143: Compiled with
! 3144: .B \-Cf,
! 3145: this is about as fast as one can get a
! 3146: .I flex
! 3147: scanner to go for this particular problem.
! 3148: .PP
! 3149: A final note:
! 3150: .I flex
! 3151: is slow when matching NUL's, particularly when a token contains
! 3152: multiple NUL's.
! 3153: It's best to write rules which match
! 3154: .I short
! 3155: amounts of text if it's anticipated that the text will often include NUL's.
! 3156: .PP
! 3157: Another final note regarding performance: as mentioned above in the section
! 3158: How the Input is Matched, dynamically resizing
! 3159: .B yytext
! 3160: to accommodate huge tokens is a slow process because it presently requires that
! 3161: the (huge) token be rescanned from the beginning. Thus if performance is
! 3162: vital, you should attempt to match "large" quantities of text but not
! 3163: "huge" quantities, where the cutoff between the two is at about 8K
! 3164: characters/token.
! 3165: .SH GENERATING C++ SCANNERS
! 3166: .I flex
! 3167: provides two different ways to generate scanners for use with C++. The
! 3168: first way is to simply compile a scanner generated by
! 3169: .I flex
! 3170: using a C++ compiler instead of a C compiler. You should not encounter
! 3171: any compilations errors (please report any you find to the email address
! 3172: given in the Author section below). You can then use C++ code in your
! 3173: rule actions instead of C code. Note that the default input source for
! 3174: your scanner remains
! 3175: .I yyin,
! 3176: and default echoing is still done to
! 3177: .I yyout.
! 3178: Both of these remain
! 3179: .I FILE *
! 3180: variables and not C++
! 3181: .I streams.
! 3182: .PP
! 3183: You can also use
! 3184: .I flex
! 3185: to generate a C++ scanner class, using the
! 3186: .B \-+
! 3187: option (or, equivalently,
! 3188: .B %option c++),
! 3189: which is automatically specified if the name of the flex
! 3190: executable ends in a '+', such as
! 3191: .I flex++.
! 3192: When using this option, flex defaults to generating the scanner to the file
! 3193: .B lex.yy.cc
! 3194: instead of
! 3195: .B lex.yy.c.
! 3196: The generated scanner includes the header file
! 3197: .I FlexLexer.h,
! 3198: which defines the interface to two C++ classes.
! 3199: .PP
! 3200: The first class,
! 3201: .B FlexLexer,
! 3202: provides an abstract base class defining the general scanner class
! 3203: interface. It provides the following member functions:
! 3204: .TP
! 3205: .B const char* YYText()
! 3206: returns the text of the most recently matched token, the equivalent of
! 3207: .B yytext.
! 3208: .TP
! 3209: .B int YYLeng()
! 3210: returns the length of the most recently matched token, the equivalent of
! 3211: .B yyleng.
! 3212: .TP
! 3213: .B int lineno() const
! 3214: returns the current input line number
! 3215: (see
! 3216: .B %option yylineno),
! 3217: or
! 3218: .B 1
! 3219: if
! 3220: .B %option yylineno
! 3221: was not used.
! 3222: .TP
! 3223: .B void set_debug( int flag )
! 3224: sets the debugging flag for the scanner, equivalent to assigning to
! 3225: .B yy_flex_debug
! 3226: (see the Options section above). Note that you must build the scanner
! 3227: using
! 3228: .B %option debug
! 3229: to include debugging information in it.
! 3230: .TP
! 3231: .B int debug() const
! 3232: returns the current setting of the debugging flag.
! 3233: .PP
! 3234: Also provided are member functions equivalent to
! 3235: .B yy_switch_to_buffer(),
! 3236: .B yy_create_buffer()
! 3237: (though the first argument is an
! 3238: .B istream*
! 3239: object pointer and not a
! 3240: .B FILE*),
! 3241: .B yy_flush_buffer(),
! 3242: .B yy_delete_buffer(),
! 3243: and
! 3244: .B yyrestart()
! 3245: (again, the first argument is a
! 3246: .B istream*
! 3247: object pointer).
! 3248: .PP
! 3249: The second class defined in
! 3250: .I FlexLexer.h
! 3251: is
! 3252: .B yyFlexLexer,
! 3253: which is derived from
! 3254: .B FlexLexer.
! 3255: It defines the following additional member functions:
! 3256: .TP
! 3257: .B
! 3258: yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 )
! 3259: constructs a
! 3260: .B yyFlexLexer
! 3261: object using the given streams for input and output. If not specified,
! 3262: the streams default to
! 3263: .B cin
! 3264: and
! 3265: .B cout,
! 3266: respectively.
! 3267: .TP
! 3268: .B virtual int yylex()
! 3269: performs the same role is
! 3270: .B yylex()
! 3271: does for ordinary flex scanners: it scans the input stream, consuming
! 3272: tokens, until a rule's action returns a value. If you derive a subclass
! 3273: .B S
! 3274: from
! 3275: .B yyFlexLexer
! 3276: and want to access the member functions and variables of
! 3277: .B S
! 3278: inside
! 3279: .B yylex(),
! 3280: then you need to use
! 3281: .B %option yyclass="S"
! 3282: to inform
! 3283: .I flex
! 3284: that you will be using that subclass instead of
! 3285: .B yyFlexLexer.
! 3286: In this case, rather than generating
! 3287: .B yyFlexLexer::yylex(),
! 3288: .I flex
! 3289: generates
! 3290: .B S::yylex()
! 3291: (and also generates a dummy
! 3292: .B yyFlexLexer::yylex()
! 3293: that calls
! 3294: .B yyFlexLexer::LexerError()
! 3295: if called).
! 3296: .TP
! 3297: .B
! 3298: virtual void switch_streams(istream* new_in = 0,
! 3299: .B
! 3300: ostream* new_out = 0)
! 3301: reassigns
! 3302: .B yyin
! 3303: to
! 3304: .B new_in
! 3305: (if non-nil)
! 3306: and
! 3307: .B yyout
! 3308: to
! 3309: .B new_out
! 3310: (ditto), deleting the previous input buffer if
! 3311: .B yyin
! 3312: is reassigned.
! 3313: .TP
! 3314: .B
! 3315: int yylex( istream* new_in, ostream* new_out = 0 )
! 3316: first switches the input streams via
! 3317: .B switch_streams( new_in, new_out )
! 3318: and then returns the value of
! 3319: .B yylex().
! 3320: .PP
! 3321: In addition,
! 3322: .B yyFlexLexer
! 3323: defines the following protected virtual functions which you can redefine
! 3324: in derived classes to tailor the scanner:
! 3325: .TP
! 3326: .B
! 3327: virtual int LexerInput( char* buf, int max_size )
! 3328: reads up to
! 3329: .B max_size
! 3330: characters into
! 3331: .B buf
! 3332: and returns the number of characters read. To indicate end-of-input,
! 3333: return 0 characters. Note that "interactive" scanners (see the
! 3334: .B \-B
! 3335: and
! 3336: .B \-I
! 3337: flags) define the macro
! 3338: .B YY_INTERACTIVE.
! 3339: If you redefine
! 3340: .B LexerInput()
! 3341: and need to take different actions depending on whether or not
! 3342: the scanner might be scanning an interactive input source, you can
! 3343: test for the presence of this name via
! 3344: .B #ifdef.
! 3345: .TP
! 3346: .B
! 3347: virtual void LexerOutput( const char* buf, int size )
! 3348: writes out
! 3349: .B size
! 3350: characters from the buffer
! 3351: .B buf,
! 3352: which, while NUL-terminated, may also contain "internal" NUL's if
! 3353: the scanner's rules can match text with NUL's in them.
! 3354: .TP
! 3355: .B
! 3356: virtual void LexerError( const char* msg )
! 3357: reports a fatal error message. The default version of this function
! 3358: writes the message to the stream
! 3359: .B cerr
! 3360: and exits.
! 3361: .PP
! 3362: Note that a
! 3363: .B yyFlexLexer
! 3364: object contains its
! 3365: .I entire
! 3366: scanning state. Thus you can use such objects to create reentrant
! 3367: scanners. You can instantiate multiple instances of the same
! 3368: .B yyFlexLexer
! 3369: class, and you can also combine multiple C++ scanner classes together
! 3370: in the same program using the
! 3371: .B \-P
! 3372: option discussed above.
! 3373: .PP
! 3374: Finally, note that the
! 3375: .B %array
! 3376: feature is not available to C++ scanner classes; you must use
! 3377: .B %pointer
! 3378: (the default).
! 3379: .PP
! 3380: Here is an example of a simple C++ scanner:
! 3381: .nf
! 3382:
! 3383: // An example of using the flex C++ scanner class.
! 3384:
! 3385: %{
! 3386: int mylineno = 0;
! 3387: %}
! 3388:
! 3389: string \\"[^\\n"]+\\"
! 3390:
! 3391: ws [ \\t]+
! 3392:
! 3393: alpha [A-Za-z]
! 3394: dig [0-9]
! 3395: name ({alpha}|{dig}|\\$)({alpha}|{dig}|[_.\\-/$])*
! 3396: num1 [-+]?{dig}+\\.?([eE][-+]?{dig}+)?
! 3397: num2 [-+]?{dig}*\\.{dig}+([eE][-+]?{dig}+)?
! 3398: number {num1}|{num2}
! 3399:
! 3400: %%
! 3401:
! 3402: {ws} /* skip blanks and tabs */
! 3403:
! 3404: "/*" {
! 3405: int c;
! 3406:
! 3407: while((c = yyinput()) != 0)
! 3408: {
! 3409: if(c == '\\n')
! 3410: ++mylineno;
! 3411:
! 3412: else if(c == '*')
! 3413: {
! 3414: if((c = yyinput()) == '/')
! 3415: break;
! 3416: else
! 3417: unput(c);
! 3418: }
! 3419: }
! 3420: }
! 3421:
! 3422: {number} cout << "number " << YYText() << '\\n';
! 3423:
! 3424: \\n mylineno++;
! 3425:
! 3426: {name} cout << "name " << YYText() << '\\n';
! 3427:
! 3428: {string} cout << "string " << YYText() << '\\n';
! 3429:
! 3430: %%
! 3431:
! 3432: int main( int /* argc */, char** /* argv */ )
! 3433: {
! 3434: FlexLexer* lexer = new yyFlexLexer;
! 3435: while(lexer->yylex() != 0)
! 3436: ;
! 3437: return 0;
! 3438: }
! 3439: .fi
! 3440: If you want to create multiple (different) lexer classes, you use the
! 3441: .B \-P
! 3442: flag (or the
! 3443: .B prefix=
! 3444: option) to rename each
! 3445: .B yyFlexLexer
! 3446: to some other
! 3447: .B xxFlexLexer.
! 3448: You then can include
! 3449: .B <FlexLexer.h>
! 3450: in your other sources once per lexer class, first renaming
! 3451: .B yyFlexLexer
! 3452: as follows:
! 3453: .nf
! 3454:
! 3455: #undef yyFlexLexer
! 3456: #define yyFlexLexer xxFlexLexer
! 3457: #include <FlexLexer.h>
! 3458:
! 3459: #undef yyFlexLexer
! 3460: #define yyFlexLexer zzFlexLexer
! 3461: #include <FlexLexer.h>
! 3462:
! 3463: .fi
! 3464: if, for example, you used
! 3465: .B %option prefix="xx"
! 3466: for one of your scanners and
! 3467: .B %option prefix="zz"
! 3468: for the other.
! 3469: .PP
! 3470: IMPORTANT: the present form of the scanning class is
! 3471: .I experimental
! 3472: and may change considerably between major releases.
! 3473: .SH INCOMPATIBILITIES WITH LEX AND POSIX
! 3474: .I flex
! 3475: is a rewrite of the AT&T Unix
! 3476: .I lex
! 3477: tool (the two implementations do not share any code, though),
! 3478: with some extensions and incompatibilities, both of which
! 3479: are of concern to those who wish to write scanners acceptable
! 3480: to either implementation. Flex is fully compliant with the POSIX
! 3481: .I lex
! 3482: specification, except that when using
! 3483: .B %pointer
! 3484: (the default), a call to
! 3485: .B unput()
! 3486: destroys the contents of
! 3487: .B yytext,
! 3488: which is counter to the POSIX specification.
! 3489: .PP
! 3490: In this section we discuss all of the known areas of incompatibility
! 3491: between flex, AT&T lex, and the POSIX specification.
! 3492: .PP
! 3493: .I flex's
! 3494: .B \-l
! 3495: option turns on maximum compatibility with the original AT&T
! 3496: .I lex
! 3497: implementation, at the cost of a major loss in the generated scanner's
! 3498: performance. We note below which incompatibilities can be overcome
! 3499: using the
! 3500: .B \-l
! 3501: option.
! 3502: .PP
! 3503: .I flex
! 3504: is fully compatible with
! 3505: .I lex
! 3506: with the following exceptions:
! 3507: .IP -
! 3508: The undocumented
! 3509: .I lex
! 3510: scanner internal variable
! 3511: .B yylineno
! 3512: is not supported unless
! 3513: .B \-l
! 3514: or
! 3515: .B %option yylineno
! 3516: is used.
! 3517: .IP
! 3518: .B yylineno
! 3519: should be maintained on a per-buffer basis, rather than a per-scanner
! 3520: (single global variable) basis.
! 3521: .IP
! 3522: .B yylineno
! 3523: is not part of the POSIX specification.
! 3524: .IP -
! 3525: The
! 3526: .B input()
! 3527: routine is not redefinable, though it may be called to read characters
! 3528: following whatever has been matched by a rule. If
! 3529: .B input()
! 3530: encounters an end-of-file the normal
! 3531: .B yywrap()
! 3532: processing is done. A ``real'' end-of-file is returned by
! 3533: .B input()
! 3534: as
! 3535: .I EOF.
! 3536: .IP
! 3537: Input is instead controlled by defining the
! 3538: .B YY_INPUT
! 3539: macro.
! 3540: .IP
! 3541: The
! 3542: .I flex
! 3543: restriction that
! 3544: .B input()
! 3545: cannot be redefined is in accordance with the POSIX specification,
! 3546: which simply does not specify any way of controlling the
! 3547: scanner's input other than by making an initial assignment to
! 3548: .I yyin.
! 3549: .IP -
! 3550: The
! 3551: .B unput()
! 3552: routine is not redefinable. This restriction is in accordance with POSIX.
! 3553: .IP -
! 3554: .I flex
! 3555: scanners are not as reentrant as
! 3556: .I lex
! 3557: scanners. In particular, if you have an interactive scanner and
! 3558: an interrupt handler which long-jumps out of the scanner, and
! 3559: the scanner is subsequently called again, you may get the following
! 3560: message:
! 3561: .nf
! 3562:
! 3563: fatal flex scanner internal error--end of buffer missed
! 3564:
! 3565: .fi
! 3566: To reenter the scanner, first use
! 3567: .nf
! 3568:
! 3569: yyrestart( yyin );
! 3570:
! 3571: .fi
! 3572: Note that this call will throw away any buffered input; usually this
! 3573: isn't a problem with an interactive scanner.
! 3574: .IP
! 3575: Also note that flex C++ scanner classes
! 3576: .I are
! 3577: reentrant, so if using C++ is an option for you, you should use
! 3578: them instead. See "Generating C++ Scanners" above for details.
! 3579: .IP -
! 3580: .B output()
! 3581: is not supported.
! 3582: Output from the
! 3583: .B ECHO
! 3584: macro is done to the file-pointer
! 3585: .I yyout
! 3586: (default
! 3587: .I stdout).
! 3588: .IP
! 3589: .B output()
! 3590: is not part of the POSIX specification.
! 3591: .IP -
! 3592: .I lex
! 3593: does not support exclusive start conditions (%x), though they
! 3594: are in the POSIX specification.
! 3595: .IP -
! 3596: When definitions are expanded,
! 3597: .I flex
! 3598: encloses them in parentheses.
! 3599: With lex, the following:
! 3600: .nf
! 3601:
! 3602: NAME [A-Z][A-Z0-9]*
! 3603: %%
! 3604: foo{NAME}? printf( "Found it\\n" );
! 3605: %%
! 3606:
! 3607: .fi
! 3608: will not match the string "foo" because when the macro
! 3609: is expanded the rule is equivalent to "foo[A-Z][A-Z0-9]*?"
! 3610: and the precedence is such that the '?' is associated with
! 3611: "[A-Z0-9]*". With
! 3612: .I flex,
! 3613: the rule will be expanded to
! 3614: "foo([A-Z][A-Z0-9]*)?" and so the string "foo" will match.
! 3615: .IP
! 3616: Note that if the definition begins with
! 3617: .B ^
! 3618: or ends with
! 3619: .B $
! 3620: then it is
! 3621: .I not
! 3622: expanded with parentheses, to allow these operators to appear in
! 3623: definitions without losing their special meanings. But the
! 3624: .B <s>, /,
! 3625: and
! 3626: .B <<EOF>>
! 3627: operators cannot be used in a
! 3628: .I flex
! 3629: definition.
! 3630: .IP
! 3631: Using
! 3632: .B \-l
! 3633: results in the
! 3634: .I lex
! 3635: behavior of no parentheses around the definition.
! 3636: .IP
! 3637: The POSIX specification is that the definition be enclosed in parentheses.
! 3638: .IP -
! 3639: Some implementations of
! 3640: .I lex
! 3641: allow a rule's action to begin on a separate line, if the rule's pattern
! 3642: has trailing whitespace:
! 3643: .nf
! 3644:
! 3645: %%
! 3646: foo|bar<space here>
! 3647: { foobar_action(); }
! 3648:
! 3649: .fi
! 3650: .I flex
! 3651: does not support this feature.
! 3652: .IP -
! 3653: The
! 3654: .I lex
! 3655: .B %r
! 3656: (generate a Ratfor scanner) option is not supported. It is not part
! 3657: of the POSIX specification.
! 3658: .IP -
! 3659: After a call to
! 3660: .B unput(),
! 3661: .I yytext
! 3662: is undefined until the next token is matched, unless the scanner
! 3663: was built using
! 3664: .B %array.
! 3665: This is not the case with
! 3666: .I lex
! 3667: or the POSIX specification. The
! 3668: .B \-l
! 3669: option does away with this incompatibility.
! 3670: .IP -
! 3671: The precedence of the
! 3672: .B {}
! 3673: (numeric range) operator is different.
! 3674: .I lex
! 3675: interprets "abc{1,3}" as "match one, two, or
! 3676: three occurrences of 'abc'", whereas
! 3677: .I flex
! 3678: interprets it as "match 'ab'
! 3679: followed by one, two, or three occurrences of 'c'". The latter is
! 3680: in agreement with the POSIX specification.
! 3681: .IP -
! 3682: The precedence of the
! 3683: .B ^
! 3684: operator is different.
! 3685: .I lex
! 3686: interprets "^foo|bar" as "match either 'foo' at the beginning of a line,
! 3687: or 'bar' anywhere", whereas
! 3688: .I flex
! 3689: interprets it as "match either 'foo' or 'bar' if they come at the beginning
! 3690: of a line". The latter is in agreement with the POSIX specification.
! 3691: .IP -
! 3692: The special table-size declarations such as
! 3693: .B %a
! 3694: supported by
! 3695: .I lex
! 3696: are not required by
! 3697: .I flex
! 3698: scanners;
! 3699: .I flex
! 3700: ignores them.
! 3701: .IP -
! 3702: The name
! 3703: .bd
! 3704: FLEX_SCANNER
! 3705: is #define'd so scanners may be written for use with either
! 3706: .I flex
! 3707: or
! 3708: .I lex.
! 3709: Scanners also include
! 3710: .B YY_FLEX_MAJOR_VERSION
! 3711: and
! 3712: .B YY_FLEX_MINOR_VERSION
! 3713: indicating which version of
! 3714: .I flex
! 3715: generated the scanner
! 3716: (for example, for the 2.5 release, these defines would be 2 and 5
! 3717: respectively).
! 3718: .PP
! 3719: The following
! 3720: .I flex
! 3721: features are not included in
! 3722: .I lex
! 3723: or the POSIX specification:
! 3724: .nf
! 3725:
! 3726: C++ scanners
! 3727: %option
! 3728: start condition scopes
! 3729: start condition stacks
! 3730: interactive/non-interactive scanners
! 3731: yy_scan_string() and friends
! 3732: yyterminate()
! 3733: yy_set_interactive()
! 3734: yy_set_bol()
! 3735: YY_AT_BOL()
! 3736: <<EOF>>
! 3737: <*>
! 3738: YY_DECL
! 3739: YY_START
! 3740: YY_USER_ACTION
! 3741: YY_USER_INIT
! 3742: #line directives
! 3743: %{}'s around actions
! 3744: multiple actions on a line
! 3745:
! 3746: .fi
! 3747: plus almost all of the flex flags.
! 3748: The last feature in the list refers to the fact that with
! 3749: .I flex
! 3750: you can put multiple actions on the same line, separated with
! 3751: semi-colons, while with
! 3752: .I lex,
! 3753: the following
! 3754: .nf
! 3755:
! 3756: foo handle_foo(); ++num_foos_seen;
! 3757:
! 3758: .fi
! 3759: is (rather surprisingly) truncated to
! 3760: .nf
! 3761:
! 3762: foo handle_foo();
! 3763:
! 3764: .fi
! 3765: .I flex
! 3766: does not truncate the action. Actions that are not enclosed in
! 3767: braces are simply terminated at the end of the line.
! 3768: .SH DIAGNOSTICS
! 3769: .PP
! 3770: .I warning, rule cannot be matched
! 3771: indicates that the given rule
! 3772: cannot be matched because it follows other rules that will
! 3773: always match the same text as it. For
! 3774: example, in the following "foo" cannot be matched because it comes after
! 3775: an identifier "catch-all" rule:
! 3776: .nf
! 3777:
! 3778: [a-z]+ got_identifier();
! 3779: foo got_foo();
! 3780:
! 3781: .fi
! 3782: Using
! 3783: .B REJECT
! 3784: in a scanner suppresses this warning.
! 3785: .PP
! 3786: .I warning,
! 3787: .B \-s
! 3788: .I
! 3789: option given but default rule can be matched
! 3790: means that it is possible (perhaps only in a particular start condition)
! 3791: that the default rule (match any single character) is the only one
! 3792: that will match a particular input. Since
! 3793: .B \-s
! 3794: was given, presumably this is not intended.
! 3795: .PP
! 3796: .I reject_used_but_not_detected undefined
! 3797: or
! 3798: .I yymore_used_but_not_detected undefined -
! 3799: These errors can occur at compile time. They indicate that the
! 3800: scanner uses
! 3801: .B REJECT
! 3802: or
! 3803: .B yymore()
! 3804: but that
! 3805: .I flex
! 3806: failed to notice the fact, meaning that
! 3807: .I flex
! 3808: scanned the first two sections looking for occurrences of these actions
! 3809: and failed to find any, but somehow you snuck some in (via a #include
! 3810: file, for example). Use
! 3811: .B %option reject
! 3812: or
! 3813: .B %option yymore
! 3814: to indicate to flex that you really do use these features.
! 3815: .PP
! 3816: .I flex scanner jammed -
! 3817: a scanner compiled with
! 3818: .B \-s
! 3819: has encountered an input string which wasn't matched by
! 3820: any of its rules. This error can also occur due to internal problems.
! 3821: .PP
! 3822: .I token too large, exceeds YYLMAX -
! 3823: your scanner uses
! 3824: .B %array
! 3825: and one of its rules matched a string longer than the
! 3826: .B YYLMAX
! 3827: constant (8K bytes by default). You can increase the value by
! 3828: #define'ing
! 3829: .B YYLMAX
! 3830: in the definitions section of your
! 3831: .I flex
! 3832: input.
! 3833: .PP
! 3834: .I scanner requires \-8 flag to
! 3835: .I use the character 'x' -
! 3836: Your scanner specification includes recognizing the 8-bit character
! 3837: .I 'x'
! 3838: and you did not specify the \-8 flag, and your scanner defaulted to 7-bit
! 3839: because you used the
! 3840: .B \-Cf
! 3841: or
! 3842: .B \-CF
! 3843: table compression options. See the discussion of the
! 3844: .B \-7
! 3845: flag for details.
! 3846: .PP
! 3847: .I flex scanner push-back overflow -
! 3848: you used
! 3849: .B unput()
! 3850: to push back so much text that the scanner's buffer could not hold
! 3851: both the pushed-back text and the current token in
! 3852: .B yytext.
! 3853: Ideally the scanner should dynamically resize the buffer in this case, but at
! 3854: present it does not.
! 3855: .PP
! 3856: .I
! 3857: input buffer overflow, can't enlarge buffer because scanner uses REJECT -
! 3858: the scanner was working on matching an extremely large token and needed
! 3859: to expand the input buffer. This doesn't work with scanners that use
! 3860: .B
! 3861: REJECT.
! 3862: .PP
! 3863: .I
! 3864: fatal flex scanner internal error--end of buffer missed -
! 3865: This can occur in an scanner which is reentered after a long-jump
! 3866: has jumped out (or over) the scanner's activation frame. Before
! 3867: reentering the scanner, use:
! 3868: .nf
! 3869:
! 3870: yyrestart( yyin );
! 3871:
! 3872: .fi
! 3873: or, as noted above, switch to using the C++ scanner class.
! 3874: .PP
! 3875: .I too many start conditions in <> construct! -
! 3876: you listed more start conditions in a <> construct than exist (so
! 3877: you must have listed at least one of them twice).
! 3878: .SH FILES
! 3879: .TP
! 3880: .B \-lfl
! 3881: library with which scanners must be linked.
! 3882: .TP
! 3883: .I lex.yy.c
! 3884: generated scanner (called
! 3885: .I lexyy.c
! 3886: on some systems).
! 3887: .TP
! 3888: .I lex.yy.cc
! 3889: generated C++ scanner class, when using
! 3890: .B -+.
! 3891: .TP
! 3892: .I <FlexLexer.h>
! 3893: header file defining the C++ scanner base class,
! 3894: .B FlexLexer,
! 3895: and its derived class,
! 3896: .B yyFlexLexer.
! 3897: .TP
! 3898: .I flex.skl
! 3899: skeleton scanner. This file is only used when building flex, not when
! 3900: flex executes.
! 3901: .TP
! 3902: .I lex.backup
! 3903: backing-up information for
! 3904: .B \-b
! 3905: flag (called
! 3906: .I lex.bck
! 3907: on some systems).
! 3908: .SH DEFICIENCIES / BUGS
! 3909: .PP
! 3910: Some trailing context
! 3911: patterns cannot be properly matched and generate
! 3912: warning messages ("dangerous trailing context"). These are
! 3913: patterns where the ending of the
! 3914: first part of the rule matches the beginning of the second
! 3915: part, such as "zx*/xy*", where the 'x*' matches the 'x' at
! 3916: the beginning of the trailing context. (Note that the POSIX draft
! 3917: states that the text matched by such patterns is undefined.)
! 3918: .PP
! 3919: For some trailing context rules, parts which are actually fixed-length are
! 3920: not recognized as such, leading to the abovementioned performance loss.
! 3921: In particular, parts using '|' or {n} (such as "foo{3}") are always
! 3922: considered variable-length.
! 3923: .PP
! 3924: Combining trailing context with the special '|' action can result in
! 3925: .I fixed
! 3926: trailing context being turned into the more expensive
! 3927: .I variable
! 3928: trailing context. For example, in the following:
! 3929: .nf
! 3930:
! 3931: %%
! 3932: abc |
! 3933: xyz/def
! 3934:
! 3935: .fi
! 3936: .PP
! 3937: Use of
! 3938: .B unput()
! 3939: invalidates yytext and yyleng, unless the
! 3940: .B %array
! 3941: directive
! 3942: or the
! 3943: .B \-l
! 3944: option has been used.
! 3945: .PP
! 3946: Pattern-matching of NUL's is substantially slower than matching other
! 3947: characters.
! 3948: .PP
! 3949: Dynamic resizing of the input buffer is slow, as it entails rescanning
! 3950: all the text matched so far by the current (generally huge) token.
! 3951: .PP
! 3952: Due to both buffering of input and read-ahead, you cannot intermix
! 3953: calls to <stdio.h> routines, such as, for example,
! 3954: .B getchar(),
! 3955: with
! 3956: .I flex
! 3957: rules and expect it to work. Call
! 3958: .B input()
! 3959: instead.
! 3960: .PP
! 3961: The total table entries listed by the
! 3962: .B \-v
! 3963: flag excludes the number of table entries needed to determine
! 3964: what rule has been matched. The number of entries is equal
! 3965: to the number of DFA states if the scanner does not use
! 3966: .B REJECT,
! 3967: and somewhat greater than the number of states if it does.
! 3968: .PP
! 3969: .B REJECT
! 3970: cannot be used with the
! 3971: .B \-f
! 3972: or
! 3973: .B \-F
! 3974: options.
! 3975: .PP
! 3976: The
! 3977: .I flex
! 3978: internal algorithms need documentation.
! 3979: .SH SEE ALSO
! 3980: .PP
! 3981: lex(1), yacc(1), sed(1), awk(1).
! 3982: .PP
! 3983: John Levine, Tony Mason, and Doug Brown,
! 3984: .I Lex & Yacc,
! 3985: O'Reilly and Associates. Be sure to get the 2nd edition.
! 3986: .PP
! 3987: M. E. Lesk and E. Schmidt,
! 3988: .I LEX \- Lexical Analyzer Generator
! 3989: .PP
! 3990: Alfred Aho, Ravi Sethi and Jeffrey Ullman,
! 3991: .I Compilers: Principles, Techniques and Tools,
! 3992: Addison-Wesley (1986). Describes the pattern-matching techniques used by
! 3993: .I flex
! 3994: (deterministic finite automata).
! 3995: .SH AUTHOR
! 3996: Vern Paxson, with the help of many ideas and much inspiration from
! 3997: Van Jacobson. Original version by Jef Poskanzer. The fast table
! 3998: representation is a partial implementation of a design done by Van
! 3999: Jacobson. The implementation was done by Kevin Gong and Vern Paxson.
! 4000: .PP
! 4001: Thanks to the many
! 4002: .I flex
! 4003: beta-testers, feedbackers, and contributors, especially Francois Pinard,
! 4004: Casey Leedom,
! 4005: Robert Abramovitz,
! 4006: Stan Adermann, Terry Allen, David Barker-Plummer, John Basrai,
! 4007: Neal Becker, Nelson H.F. Beebe, benson@odi.com,
! 4008: Karl Berry, Peter A. Bigot, Simon Blanchard,
! 4009: Keith Bostic, Frederic Brehm, Ian Brockbank, Kin Cho, Nick Christopher,
! 4010: Brian Clapper, J.T. Conklin,
! 4011: Jason Coughlin, Bill Cox, Nick Cropper, Dave Curtis, Scott David
! 4012: Daniels, Chris G. Demetriou, Theo Deraadt,
! 4013: Mike Donahue, Chuck Doucette, Tom Epperly, Leo Eskin,
! 4014: Chris Faylor, Chris Flatters, Jon Forrest, Jeffrey Friedl,
! 4015: Joe Gayda, Kaveh R. Ghazi, Wolfgang Glunz,
! 4016: Eric Goldman, Christopher M. Gould, Ulrich Grepel, Peer Griebel,
! 4017: Jan Hajic, Charles Hemphill, NORO Hideo,
! 4018: Jarkko Hietaniemi, Scott Hofmann,
! 4019: Jeff Honig, Dana Hudes, Eric Hughes, John Interrante,
! 4020: Ceriel Jacobs, Michal Jaegermann, Sakari Jalovaara, Jeffrey R. Jones,
! 4021: Henry Juengst, Klaus Kaempf, Jonathan I. Kamens, Terrence O Kane,
! 4022: Amir Katz, ken@ken.hilco.com, Kevin B. Kenny,
! 4023: Steve Kirsch, Winfried Koenig, Marq Kole, Ronald Lamprecht,
! 4024: Greg Lee, Rohan Lenard, Craig Leres, John Levine, Steve Liddle,
! 4025: David Loffredo, Mike Long,
! 4026: Mohamed el Lozy, Brian Madsen, Malte, Joe Marshall,
! 4027: Bengt Martensson, Chris Metcalf,
! 4028: Luke Mewburn, Jim Meyering, R. Alexander Milowski, Erik Naggum,
! 4029: G.T. Nicol, Landon Noll, James Nordby, Marc Nozell,
! 4030: Richard Ohnemus, Karsten Pahnke,
! 4031: Sven Panne, Roland Pesch, Walter Pelissero, Gaumond
! 4032: Pierre, Esmond Pitt, Jef Poskanzer, Joe Rahmeh, Jarmo Raiha,
! 4033: Frederic Raimbault, Pat Rankin, Rick Richardson,
! 4034: Kevin Rodgers, Kai Uwe Rommel, Jim Roskind, Alberto Santini,
! 4035: Andreas Scherer, Darrell Schiebel, Raf Schietekat,
! 4036: Doug Schmidt, Philippe Schnoebelen, Andreas Schwab,
! 4037: Larry Schwimmer, Alex Siegel, Eckehard Stolz, Jan-Erik Strvmquist,
! 4038: Mike Stump, Paul Stuart, Dave Tallman, Ian Lance Taylor,
! 4039: Chris Thewalt, Richard M. Timoney, Jodi Tsai,
! 4040: Paul Tuinenga, Gary Weik, Frank Whaley, Gerhard Wilhelms, Kent Williams, Ken
! 4041: Yap, Ron Zellar, Nathan Zelle, David Zuhn,
! 4042: and those whose names have slipped my marginal
! 4043: mail-archiving skills but whose contributions are appreciated all the
! 4044: same.
! 4045: .PP
! 4046: Thanks to Keith Bostic, Jon Forrest, Noah Friedman,
! 4047: John Gilmore, Craig Leres, John Levine, Bob Mulcahy, G.T.
! 4048: Nicol, Francois Pinard, Rich Salz, and Richard Stallman for help with various
! 4049: distribution headaches.
! 4050: .PP
! 4051: Thanks to Esmond Pitt and Earle Horton for 8-bit character support; to
! 4052: Benson Margulies and Fred Burke for C++ support; to Kent Williams and Tom
! 4053: Epperly for C++ class support; to Ove Ewerlid for support of NUL's; and to
! 4054: Eric Hughes for support of multiple buffers.
! 4055: .PP
! 4056: This work was primarily done when I was with the Real Time Systems Group
! 4057: at the Lawrence Berkeley Laboratory in Berkeley, CA. Many thanks to all there
! 4058: for the support I received.
! 4059: .PP
! 4060: Send comments to vern@ee.lbl.gov.
![[BACK]](/icons/back.gif){kind=icon}
Return to flex.1 CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / src / usr.bin / lex