Databases
StackOverflow2013
Postsdbo
POUnescaping HTML entities (including named ones)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POUnescaping HTML entities (including named ones)
    primarykey
    data
    text
    <p>This question is similar to the <a href="https://stackoverflow.com/questions/4218205/haskell-remove-html-character-entities-in-a-string">Remove html character entities in a string</a> question asked earlier on Stack Overflow. <strong>The accepted answer, however, does not address the issue of named HTML entities, e.g. <code>&amp;auml;</code> for the character <code>ä</code>;</strong> It therefore cannot unescape all HTML.</p> <p>I have some legacy HTML which uses named HTML entities for non-ASCII characters. That is, <code>&amp;ouml;</code> instead of <code>ö</code>, <code>&amp;auml;</code> instead of <code>ä</code> and so on. <a href="http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references" rel="nofollow noreferrer">A full list of all named HTML entities</a> is available on Wikipedia.</p> <p>I'd like to unescape these HTML entities into their character equivalents, in a quick and efficient manner.</p> <hr> <p>I have the code to do this in Python 3, using regular expressions:</p> <pre><code>import re import html.entities s = re.sub(r'&amp;(\w+?);', lambda m: chr(html.entities.name2codepoint[m.group(1)]), s) </code></pre> <p>Regular expressions, however, don't seem very popular, fast or easy to use in Haskell.</p> <hr> <p><code>Text.HTML.TagSoup.Entity</code> (tagsoup) has a useful table and functions for mapping named entities tpo codepoints. Using this, and the regex-tdfa package, I've fashioned an <em>extremely slow</em> equivalent in Haskell:</p> <pre><code>{-# LANGUAGE OverloadedStrings #-} import Data.ByteString.Lazy.Char8 as L import Data.ByteString.Lazy.UTF8 as UTF8 import Text.HTML.TagSoup.Entity (lookupEntity) import Text.Regex.TDFA ((=~~)) unescapeEntites :: L.ByteString -&gt; L.ByteString unescapeEntites = regexReplaceBy "&amp;#?[[:alnum:]]+;" $ lookupMatch where lookupMatch m = case lookupEntity (L.unpack . L.tail . L.init $ m) of Nothing -&gt; m Just x -&gt; UTF8.fromString [x] -- regex replace taken from http://mutelight.org/articles/generating-a-permalink-slug-in-haskell regexReplaceBy :: L.ByteString -&gt; (L.ByteString -&gt; L.ByteString) -&gt; L.ByteString -&gt; L.ByteString regexReplaceBy regex f text = go text [] where go str res = if L.null str then L.concat . reverse $ res else case (str =~~ regex) :: Maybe (L.ByteString, L.ByteString, L.ByteString) of Nothing -&gt; L.concat . reverse $ (str : res) Just (bef, match , aft) -&gt; go aft (f match : bef : res) </code></pre> <p>The <code>unescapeEntities</code> function runs several orders of magnitude slower than the Python version above. The Python code can convert about 130 MB in 7 seconds, whereas my Haskell version runs for several minutes.</p> <p>I'm looking for a better solution, primarily in terms of speed. But I'd also like to avoid regular expressions, if possible (speed and avoiding regular expressions seem to go hand in hand in Haskell anyway).</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USvicvicvic
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POUnescaping HTML entities (including named ones)
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIt's not clear what your actual question is here. Are you looking for a better solution? Want help improving the current one?
      singulars
      1. POUnescaping HTML entities (including named ones)
      1. USC. A. McCann
    2. COSorry if the question was unclear. Yes, I want a better solution because mine is 1. too slow 2. uses regular expressions which doesn't seem very Haskell-idiomatic (given how little information there is about them). My solution was primarily given as a "this is what I've got so far" starting point. I'm open to radical ideas.
      singulars
      1. POUnescaping HTML entities (including named ones)
      1. USvicvicvic
    3. COHow are you reading the file? If I make `main = Data.ByteString.interact unescapeEntites` and do `time cat big.txt | ./regex >>/dev/null` I get 30 seconds for 143M big.txt (which is all the entities listed in TagSoup with lots of 'a' s interspersed). Still clunky from all this indirection, but not minutes.
      singulars
      1. POUnescaping HTML entities (including named ones)
      1. USapplicative
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload