Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>There's an excellent article on <a href="http://effbot.org/zone/xml-scanner.htm" rel="noreferrer">Using Regular Expressions for Lexical Analysis</a> at <a href="http://effbot.org/" rel="noreferrer">effbot.org</a>.</p> <p>Adapting the tokenizer to your problem:</p> <pre><code>import re token_pattern = r""" (?P&lt;identifier&gt;[a-zA-Z_][a-zA-Z0-9_]*) |(?P&lt;integer&gt;[0-9]+) |(?P&lt;dot&gt;\.) |(?P&lt;open_variable&gt;[$][{]) |(?P&lt;open_curly&gt;[{]) |(?P&lt;close_curly&gt;[}]) |(?P&lt;newline&gt;\n) |(?P&lt;whitespace&gt;\s+) |(?P&lt;equals&gt;[=]) |(?P&lt;slash&gt;[/]) """ token_re = re.compile(token_pattern, re.VERBOSE) class TokenizerException(Exception): pass def tokenize(text): pos = 0 while True: m = token_re.match(text, pos) if not m: break pos = m.end() tokname = m.lastgroup tokvalue = m.group(tokname) yield tokname, tokvalue if pos != len(text): raise TokenizerException('tokenizer stopped at pos %r of %r' % ( pos, len(text))) </code></pre> <p>To test it, we do:</p> <pre><code>stuff = r'property.${general.name}.ip = ${general.ip}' stuff2 = r''' general { name = myname ip = 127.0.0.1 } ''' print ' stuff '.center(60, '=') for tok in tokenize(stuff): print tok print ' stuff2 '.center(60, '=') for tok in tokenize(stuff2): print tok </code></pre> <p>for:</p> <pre><code>========================== stuff =========================== ('identifier', 'property') ('dot', '.') ('open_variable', '${') ('identifier', 'general') ('dot', '.') ('identifier', 'name') ('close_curly', '}') ('dot', '.') ('identifier', 'ip') ('whitespace', ' ') ('equals', '=') ('whitespace', ' ') ('open_variable', '${') ('identifier', 'general') ('dot', '.') ('identifier', 'ip') ('close_curly', '}') ========================== stuff2 ========================== ('newline', '\n') ('identifier', 'general') ('whitespace', ' ') ('open_curly', '{') ('newline', '\n') ('whitespace', ' ') ('identifier', 'name') ('whitespace', ' ') ('equals', '=') ('whitespace', ' ') ('identifier', 'myname') ('newline', '\n') ('whitespace', ' ') ('identifier', 'ip') ('whitespace', ' ') ('equals', '=') ('whitespace', ' ') ('integer', '127') ('dot', '.') ('integer', '0') ('dot', '.') ('integer', '0') ('dot', '.') ('integer', '1') ('newline', '\n') ('close_curly', '}') ('newline', '\n') </code></pre>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
    3. VO
      singulars
      1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload