StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>[From a guy who make a custom lexer tool, and still trying to learn ANTLR]</p> <p><strong>Boring extensive answer:</strong></p> <p>You are right. Many books & courses mix both tools. And sometimes "generating/detecting tokens" and "interpreting tokens" may mix.</p> <p>Sometimes, a developer is trying to do a scanner, and still, mixes scanning & parsing in its mind ;-)</p> <p>Usually, when detecting tokens, you also have to do an action ("interpretation"), as simple, as printing a message or the found token to string. Example: "{ cout << "Hey, I found a integer constant" << "\n" }"</p> <p>There are also several cases that may make scanning difficult for a begginner in the topic.</p> <p><strong>One case is that several text may be used for different tokens.</strong></p> <p>Example:</p> <p>"-" as the substraction binary operator, and "-" as the negative prefix operator. Or, treating 5 both as an integer and a float. In scanners, "-" can be seen as the same token, while in parsers, you may treat it as different tokens.</p> <p>In order to fix this, my favorite approach its to use "generic tokens", in the scanning/lexer process, and later, convert them as "custom tokens" in the parsing/syntax process.</p> <p><strong>Quick answer:</strong></p> <p>As mentioned in previous answers, start with making a grammar, in fact, I suggest try it in a whiteboard or notebook, and later in your favorite (ANTLRL, other) scanning tool.</p> <p>Consider those special cases, where there could be some token overlappings.</p> <p>Good Luck.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload