Databases
StackOverflow2013
Postsdbo
POoptimize python processing json retrieved from the fb-graph-api
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POoptimize python processing json retrieved from the fb-graph-api
    primarykey
    data
    text
    <p>i'm getting json data from the facebook-graph-api about:</p> <ol> <li>my relationship with my friends</li> <li>my friends relationships with each other. </li> </ol> <p>right now my program looks like this (in python pseudo code, please note some variables have been changed for privacy):</p> <pre><code>import json import requests # protected _accessCode = "someAccessToken" _accessStr = "?access_token=" + _accessCode _myID = "myIDNumber" r = requests.get("https://graph.facebook.com/" + _myID + "/friends/" + _accessStr) raw = json.loads(r.text) terminate = len(raw["data"]) # list used to store the friend/friend relationships a = list() for j in range(0, terminate + 1): # calculate terminating displacement: term_displacement = terminate - (j + 1) print("Currently processing: " + str(j) + " of " + str(terminate)) for dj in range(1, term_displacement + 1): # construct urls based on the raw data: url = "https://graph.facebook.com/" + raw["data"][j]["id"] + "/friends/" + raw["data"][j + dj]["id"] + "/" + _accessStr # visit site *THIS IS THE BOTTLENECK*: reqTemp = requests.get(url) rawTemp = json.loads(reqTemp.text) if len(rawTemp["data"]) != 0: # data dumps to list which dumps to file a.append(str(raw["data"][j]["id"]) + "," + str(rawTemp["data"][0]["id"])) outputFile = "C:/Users/franklin/Documents/gen/friendsRaw.csv" output = open(outputFile, "w") # write all me/friend relationship to file for k in range(0, terminate): output.write(_myID + "," + raw["data"][k]["id"] + "\n") # write all friend/friend relationships to file for i in range(0, len(a)): output.write(a[i]) output.close() </code></pre> <p>So what its doing is: first it calls my page and gets my friend list (this is allowed through the facebook api using an access_token) calling a friend's friend list is NOT allowed but I can work around that by requesting a relationship between a friend on my list and another friend on my list. so in part two (indicated by the double for loops) i'm making another request to see if some friend, a, is also a friend of b, (both of which are on my list); if so there will be a json object of length one with friend a's name.</p> <p>but with about 357 friends there's literally thousands of page requests that need to be made. in other words the program is spending a lot of time just waiting around for the json-requests. </p> <p>my question is then can this be rewritten to be more efficient? currently, due to security restrictions, calling a friend's friend list attribute is disallowed. and it doesn't look like the api will allow this. are there any python tricks that can make this run faster? maybe parallelism? </p> <p><strong>Update</strong> modified code is pasted below in the answers section.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USfranklin
    1. USfranklin
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POoptimize python processing json retrieved from the fb-graph-api
      1. USIT Ninja
      1. VTFavorite
    1. COI would be worried about running into an API call limit of this type of query. Have you tried using FQL to make this request?
      singulars
      1. POoptimize python processing json retrieved from the fb-graph-api
      1. USDMCS
    2. COi had suspected there was a better way. for example the wolfram|alpha site makes the request in less than 1 minute. but no i have never used fql. do you have any suggestions as to the implementation? i am intent on sticking to python
      singulars
      1. POoptimize python processing json retrieved from the fb-graph-api
      1. USfranklin
    3. COPython Requests can get results asynchronously -see: http://stackoverflow.com/questions/9110593/asynchronous-requests-with-python-requests
      singulars
      1. POoptimize python processing json retrieved from the fb-graph-api
      1. USGerrat
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload