Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p><code>.+?</code> at the end of a lexer rule will always match a single character. But <code>.+</code> will consume as much as possible, which was illegal at the end of a rule in ANTLR v3 (v4 probably as well).</p> <p>What you can do is just match a single char, and "glue" these together in the parser:</p> <pre><code>unknowns : Unknown+ ; ... Unknown : . ; </code></pre> <h2>EDIT</h2> <blockquote> <p>... but I only have a lexer, no parsers ...</p> </blockquote> <p>Ah, I see. Then you could override the <code>nextToken()</code> method:</p> <pre class="lang-java prettyprint-override"><code>lexer grammar Lex; @members { public static void main(String[] args) { Lex lex = new Lex(new ANTLRInputStream("foo, bar...\n")); for(Token t : lex.getAllTokens()) { System.out.printf("%-15s '%s'\n", tokenNames[t.getType()], t.getText()); } } private java.util.Queue&lt;Token&gt; queue = new java.util.LinkedList&lt;Token&gt;(); @Override public Token nextToken() { if(!queue.isEmpty()) { return queue.poll(); } Token next = super.nextToken(); if(next.getType() != Unknown) { return next; } StringBuilder builder = new StringBuilder(); while(next.getType() == Unknown) { builder.append(next.getText()); next = super.nextToken(); } // The `next` will _not_ be an Unknown-token, store it in // the queue to return the next time! queue.offer(next); return new CommonToken(Unknown, builder.toString()); } } Whitespace : [ \t\n\r]+ -&gt; skip ; Punctuation : [.,:;?!] ; Unknown : . ; </code></pre> <p>Running it:</p> <pre>java -cp antlr-4.0-complete.jar org.antlr.v4.Tool Lex.g4 javac -cp antlr-4.0-complete.jar *.java java -cp .:antlr-4.0-complete.jar Lex</pre> <p>will print:</p> <pre>Unknown 'foo' Punctuation ',' Unknown 'bar' Punctuation '.' Punctuation '.' Punctuation '.'</pre>
    singulars
    1. This table or related slice is empty.
    1. POIn antlr4 lexer, How to have a rule that catches all remaining "words" as Unknown token?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USBart Kiers
    1. USBart Kiers
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POIn antlr4 lexer, How to have a rule that catches all remaining "words" as Unknown token?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThanks, but I only have a lexer, no parsers, I am using this to tokenize texts for a NLP tool. What we want is to have an unknown or unrecognized token for whatever lexer can not identify as token. Maybe I can intercept these unknown chars tokens and concatenate myself.
      singulars
      1. PO
      1. USmdakin
    2. CO@mdakin, checkout my **EDIT**.
      singulars
      1. PO
      1. USBart Kiers
    3. COThis works for the limited example I gave, but now I understand my problem is deeper and not actually easy to solve. If I have another rule like: " word: [a-zA-Z]+; " , Lexer would tokenize this: "~A~a" as "~":Unknown "A":Word "~":Unknown "a":word . The reason is, I skip the whitespaces, so I have no context on where a word starts or ends. But anyway your answer still helped me to learn about the working of the lexer.
      singulars
      1. PO
      1. USmdakin
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload