StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>These are just a few helpful tips for python development (and scraping):</p> <h3>Python Requests library.</h3> <p>The <a href="http://docs.python-requests.org/en/latest/" rel="nofollow">python requests</a> library is excellent at simplifying the requests process.</p> <h3>No need to use a <code>while</code> loop</h3> <p><code>for</code> loops are really useful in this situation. </p> <pre><code>symbolslist = ["aapl","spy", "goog","nflx"] for symbol in symbolslist: # Do logic here... </code></pre> <h3>Use xpath over regular expressions</h3> <pre><code>import requests import lxml url = "http://www.google.co.uk/finance?q="+symbol+"&q1=1" r = requests.get(url) xpath = '//your/xpath' root = lxml.html.fromstring(r.content) </code></pre> <h3>No need to compile your regular expressions each time.</h3> <p>Compiling regex's takes time and effort. You can abstract these out of your loop.</p> <pre><code>regex = '<span id="yfs_l84_' + symbolslist[i] + '">(.+?)</span>' pattern = re.compile(regex) for symbol in symbolslist: # do logic </code></pre> <h3>External Libraries</h3> <p>As mentioned in the comment by <code>drewk</code> both Pandas and Matplot have native functions to get Yahoo quotes or you can use the <a href="https://github.com/cgoldberg/ystockquote" rel="nofollow">ystockquote</a> library to scrape from Yahoo. This is used like so:</p> <pre><code>#!/bin/env python import ystockquote symbolslist = ["aapl","spy", "goog","nflx"] for symbol in symbolslist: print (ystockquote.get_price(symbol)) </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload