Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <ul> <li>Use PChar incrementing for speed of processing</li> <li>If some tokens are not needed, only copy token data on demand</li> <li>Copy PChar to local variable when actually scanning through characters</li> <li>Keep source data in a single buffer unless you must handle line by line, and even then, consider handling line processing as a separate token in the lexer recognizer</li> <li>Consider processing a byte array buffer that has come straight from the file, if you definitely know the encoding; if using Delphi 2009, use PAnsiChar instead of PChar, unless of course you know the encoding is UTF16-LE.</li> <li>If you know that the only whitespace is going to be #32 (ASCII space), or a similarly limited set of characters, there may be some clever bit manipulation hacks that can let you process 4 bytes at a time using Integer scanning. I wouldn't expect big wins here though, and the code will be as clear as mud.</li> </ul> <p>Here's a sample lexer that should be pretty efficient, but it assumes that all source data is in a single string. Reworking it to handle buffers is moderately tricky due to very long tokens.</p> <pre><code>type TLexer = class private FData: string; FTokenStart: PChar; FCurrPos: PChar; function GetCurrentToken: string; public constructor Create(const AData: string); function GetNextToken: Boolean; property CurrentToken: string read GetCurrentToken; end; { TLexer } constructor TLexer.Create(const AData: string); begin FData := AData; FCurrPos := PChar(FData); end; function TLexer.GetCurrentToken: string; begin SetString(Result, FTokenStart, FCurrPos - FTokenStart); end; function TLexer.GetNextToken: Boolean; var cp: PChar; begin cp := FCurrPos; // copy to local to permit register allocation // skip whitespace; this test could be converted to an unsigned int // subtraction and compare for only a single branch while (cp^ &gt; #0) and (cp^ &lt;= #32) do Inc(cp); // using null terminater for end of file Result := cp^ &lt;&gt; #0; if Result then begin FTokenStart := cp; Inc(cp); while cp^ &gt; #32 do Inc(cp); end; FCurrPos := cp; end; </code></pre>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
    3. VO
      singulars
      1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload