StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>If you load the target page's HTML into a variable htmlsource, this pyparsing webscraper will format your data in the given CSV format:</p> <pre><code>from pyparsing import * th,thEnd = makeHTMLTags("th") thCategory = th.setParseAction(withAttribute(**{'class':'clickable', 'colspan':'3'})) category = thCategory.suppress() + SkipTo(thEnd)('category') + thEnd # set up tag recognizers, with specialized patterns based on class attribute td, tdEnd = makeHTMLTags("td") tdWithClass = lambda cls : td.copy().setParseAction(withAttribute(**{'class':cls})) itemTd = tdWithClass('item-name') price1Td = tdWithClass('price city-1') price2Td = tdWithClass('price city-2') # define some currencies currency = oneOf("NZ$ AU$ US$ SG$").setName("currency") # define a currency amount as a real number amount = Regex(r'\d+,\d{3}|\d+(\.\d+)?').setParseAction(lambda t:float(t[0].replace(',',''))) # define the format of a city value cityval = Group((price1Td | price2Td) + currency("currency") + amount("amt") + SkipTo(tdEnd) + tdEnd) # define a comparison item, including item name and item cost in city1 and city2 comparison = Group(itemTd + SkipTo(tdEnd)("item") + tdEnd + (cityval*2)("valuedata")) # attach a parse action to clean up automated token naming def assignPriceTags(t): for v in t[0].valuedata: if v['class'] == 'price city-1': t[0]['price1'] = v else: t[0]['price2'] = v # remove extraneous results names created by makeHTMLTags for tg in 'class tag startTd endTd empty'.split(): del t[0][tg] for v in t[0].valuedata: del v[tg] del t[0]['valuedata'] comparison.setParseAction(assignPriceTags) currentcategory = '' for compdata in (category|comparison).searchString(htmlsource): if 'category' in compdata: currentcategory = compdata.category continue compdata = compdata[0] #~ print compdata.dump() print "%s, %s, %s%s, %s%s" % (currentcategory, compdata.item, compdata.price1.currency, compdata.price1.amt, compdata.price2.currency, compdata.price2.amt) </code></pre> <p>prints:</p> <pre><code>Food, Daily menu in the business district, AU$15.82, NZ$15.62 Food, Combo meal in fast food restaurant (Big Mac Meal or similar), AU$7.4, NZ$7.91 Food, 1/2 Kg (1 lb.) of chicken breast, AU$6.07, NZ$10.25 Food, 1 liter (1 qt.) of whole fat milk, AU$1.8, NZ$2.65 Food, 500 gr (16 oz.) of local cheese, AU$5.99, NZ$7.2 Food, 1 kg (2 lb.) of apples, AU$4.29, NZ$3.46 Food, 2 kg (4,5 lb.) of potatoes, AU$4.31, NZ$5.29 Food, 0.5 l (16 oz) beer in the supermarket, AU$4.12, NZ$4.36 Food, 2 liters of Coca-Cola, AU$3.07, NZ$2.64 Food, bread for 2 people for 1 day, AU$2.32, NZ$1.93 Housing, monthly rent for a 85 m2 (900 Sqft) furnished apartment in expensive area of the city, AU$1766.0, NZ$2034.0 Housing, Internet 8MB (1 month), AU$49.0, NZ$61.0 Housing, 40” flat screen TV, AU$865.0, NZ$1041.0 Housing, utilities 1 month (heating, electricity, gas ...), AU$211.0, NZ$170.0 Clothes, 1 pair of Levis 501, AU$119.0, NZ$123.0 Clothes, 1 summer dress in a chain store (Zara, H&M, ...), AU$63.0, NZ$50.0 Clothes, 1 pair of Adidas trainers, AU$142.0, NZ$166.0 Clothes, 1 pair of average business shoes, AU$130.0, NZ$133.0 Transportation, Volkswagen Golf 2.0 TDI 140 CV 6 vel. (or equivalent), with no extras, new, AU$28321.0, NZ$45574.0 Transportation, 1 liter (1/4 gallon) of gas, AU$1.43, NZ$2.13 Transportation, monthly ticket public transport, AU$110.0, NZ$138.0 Personal Care, medicine against cold for 6 days (Frenadol, Coldrex, ...), AU$14.27, NZ$17.85 Personal Care, 1 box of 32 tampons (Tampax, OB, ...), AU$5.51, NZ$7.71 Personal Care, 4 rolls of toilet paper, AU$3.57, NZ$3.07 Personal Care, Tube of toothpaste, AU$3.37, NZ$3.39 Personal Care, Standard men's haircut in expat area of the city, AU$27.0, NZ$27.0 Entertainment, 2 tickets to the movies, AU$33.0, NZ$30.0 Entertainment, 2 tickets to the theater (best available seats), AU$163.0, NZ$139.0 Entertainment, dinner out for two in Italian restaurant with wine and dessert, AU$100.0, NZ$100.0 Entertainment, basic dinner out for two in neighborhood pub, AU$46.0, NZ$46.0 Entertainment, 1 cocktail drink in downtown club, AU$14.31, NZ$14.38 Entertainment, 1 beer in neighbourhood pub, AU$4.69, NZ$6.72 Entertainment, iPod nano 8GB (6th generation), AU$176.0, NZ$252.0 Entertainment, 1 min. of prepaid mobile tariff (no discounts or plans), AU$1.14, NZ$0.84 Entertainment, 1 month of gym in business district, AU$90.0, NZ$91.0 Entertainment, 1 package of Marlboro cigarretes, AU$15.97, NZ$14.47 </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload