Databases
StackOverflow2013
Postsdbo
POLearning Python; How can I make this more Pythonic?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POLearning Python; How can I make this more Pythonic?
    primarykey
    data
    text
    <p>I am a PHP developer exploring the outside world. I have decided to start learning Python.</p> <p>The below script is my first attempt at porting a PHP script to Python. Its job is to take tweets from a Redis store. The tweets are coming from Twitter's Streaming API and stored as JSON objects. Then the information needed is extracted and dumped into a CSV file to be imported into MySQL using the <code>LOAD DATA LOCAL INFILE</code> that is hosted on a different server.</p> <p>So, the question is: Now that I have my first Python script running, how can I make it more Pythonic? Are there any suggestions that you guys have? Make it better? Tricks I should know about? Constructive Criticism? </p> <p><strong>Update:</strong> Having taken everyone's suggestions thus far, here is the updated version:<br> <strong>Update2:</strong> Ran the code through pylint. Now scores a 9.89/10. Any other suggestions?</p> <pre><code># -*- coding: utf-8 -*- """Redis IO Loop for Tweelay Bot""" from __future__ import with_statement import simplejson import re import datetime import time import csv import hashlib # Bot Modules import tweelay.red as red import tweelay.upload as upload import tweelay.openanything as openanything __version__ = "4" def process_tweets(): """Processes 0-20 tweets from Redis store""" data = [] last_id = 0 for i in range(20): last = red.pop_tweet() if not last: break t = TweetHandler(last) t.cleanup() t.extract() if t.get_tweet_id() == last_id: break tweet = t.proc() if tweet: data = data + [tweet] last_id = t.get_tweet_id() time.sleep(0.01) if not data: return False ch = CSVHandler(data) ch.pack_csv() ch.uploadr() source = "http://bot.tweelay.net/tweets.php" openanything.openAnything( source, etag=None, lastmodified=None, agent="Tweelay/%s (Redis)" % __version__ ) class TweetHandler: """Cleans, Builds and returns needed data from Tweet""" def __init__(self, json): self.json = json self.tweet = None self.tweet_id = 0 self.j = None def cleanup(self): """Takes JSON encoded tweet and cleans it up for processing""" self.tweet = unicode(self.json, "utf-8") self.tweet = re.sub('^s:[0-9]+:["]+', '', self.tweet) self.tweet = re.sub('\n["]+;$', '', self.tweet) def extract(self): """Takes cleaned up JSON encoded tweet and extracts the datas we need""" self.j = simplejson.loads(self.tweet) def proc(self): """Builds the datas from the JSON object""" try: return self.build() except KeyError: if 'delete' in self.j: return None else: print ";".join(["%s=%s" % (k, v) for k, v in self.j.items()]) return None def build(self): """Builds tuple from JSON tweet""" return ( self.j['user']['id'], self.j['user']['screen_name'].encode('utf-8'), self.j['text'].encode('utf-8'), self.j['id'], self.j['in_reply_to_status_id'], self.j['in_reply_to_user_id'], self.j['created_at'], __version__ ) def get_tweet_id(self): """Return Tweet ID""" if 'id' in self.j: return self.j['id'] if 'delete' in self.j: return self.j['delete']['status']['id'] class CSVHandler: """Takes list of tweets and saves them to a CSV file to be inserted into MySQL data store""" def __init__(self, data): self.data = data self.file_name = self.gen_file_name() def gen_file_name(self): """Generate unique file name""" now = datetime.datetime.now() hashr = hashlib.sha1() hashr.update(str(now)) hashr.update(str(len(self.data))) hash_str = hashr.hexdigest() return hash_str+'.csv' def pack_csv(self): """Save tweet data to CSV file""" with open('tmp/'+self.file_name, mode='ab') as ofile: writer = csv.writer( ofile, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL) writer.writerows(self.data) def uploadr(self): """Upload file to remote host""" url = "http://example.com/up.php?filename="+self.file_name uploadr = upload.upload_file(url, 'tmp/'+self.file_name) if uploadr[0] == 200: print "Upload: 200 - ("+str(len(self.data))+")", self.file_name print "-------" #os.remove('tmp/'+self.file_name) else: print "Upload Error:", uploadr[0] if __name__ == "__main__": while True: process_tweets() time.sleep(1) </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJayrox
    1. USJayrox
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POLearning Python; How can I make this more Pythonic?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POLearning Python; How can I make this more Pythonic?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POLearning Python; How can I make this more Pythonic?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COSee http://www.python.org/dev/peps/pep-0008/
      singulars
      1. POLearning Python; How can I make this more Pythonic?
      1. USNicholas Knight
    2. COFunctions and documentation strings are your friends. As far as I can tell, this script gets a tweet, does nothing with it, and then waits a little, rinse, repeat.
      singulars
      1. POLearning Python; How can I make this more Pythonic?
      1. USmsw
    3. CO+1 for "I have decided to start learning Python."
      singulars
      1. POLearning Python; How can I make this more Pythonic?
      1. USSeth
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload