Databases
StackOverflow2013
Postsdbo
POOptimize algorithm for creating a list of items rated together, in Python
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOptimize algorithm for creating a list of items rated together, in Python
    primarykey
    data
    text
    <p>given a list of purchase events (customer_id,item)</p> <pre><code>1-hammer 1-screwdriver 1-nails 2-hammer 2-nails 3-screws 3-screwdriver 4-nails 4-screws </code></pre> <p>i'm trying to build a data structure that tells how many times an item was bought with another item. Not bought at the same time, but bought since I started saving data. the result would look like</p> <pre><code>{ hammer : {screwdriver : 1, nails : 2}, screwdriver : {hammer : 1, screws : 1, nails : 1}, screws : {screwdriver : 1, nails : 1}, nails : {hammer : 1, screws : 1, screwdriver : 1} } </code></pre> <p>indicating That a hammer was bought with nails twice (persons 1,3) and a screwdriver once (person 1), screws were bought with a screwdriver once (person 3), and so on...</p> <p>my current approach is</p> <p>users = dict where userid is the key and a list of items bought is the value<p> usersForItem = dict where itemid is the key and list of users who bought item is the value<p> userlist = temporary list of users who have rated the current item<p></p> <pre><code>pseudo: for each event(customer,item)(sorted by item): add user to users dict if not exists, and add the items add item to items dict if not exists, and add the user ---------- for item,user in rows: # add the user to the users dict if they don't already exist. users[user]=users.get(user,[]) # append the current item_id to the list of items rated by the current user users[user].append(item) if item != last_item: # we just started a new item which means we just finished processing an item # write the userlist for the last item to the usersForItem dictionary. if last_item != None: usersForItem[last_item]=userlist userlist=[user] last_item = item items.append(item) else: userlist.append(user) usersForItem[last_item]=userlist </code></pre> <p>So, at this point, I have 2 dicts - who bought what, and what was bought by whom. Here's where it gets tricky. Now that usersForItem is populated, I loop through it, loop through each user who bought the item, and look at the users' other purchases. I acknowledge that this is not the most pythonic way of doing things - I'm trying to make sure I get the correct result(which I am) before getting fancy with the Python.</p> <pre><code>relatedItems = {} for key,listOfUsers in usersForItem.iteritems(): relatedItems[key]={} related=[] for ux in listOfReaders: for itemRead in users[ux]: if itemRead != key: if itemRead not in related: related.append(itemRead) relatedItems[key][itemRead]= relatedItems[key].get(itemRead,0) + 1 calc jaccard/tanimoto similarity between relatedItems[key] and its values </code></pre> <p>Is there a more efficient way that I can be doing this? Additionally, if there is a proper academic name for this type of operation, I'd love to hear it.</p> <p>edit: clarified to include the fact that I'm not restricting purchases to items bought together at the same time. Items can be bought at any time. </p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USNeil Kodner
    1. USNeil Kodner
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POOptimize algorithm for creating a list of items rated together, in Python
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POOptimize algorithm for creating a list of items rated together, in Python
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POOptimize algorithm for creating a list of items rated together, in Python
      1. This table or related slice is empty.
      1. VTUpMod
    1. COHow big is your dataset? Amazon uses some algos on hadoop to look through every item it has ever sold and generate the suggested items... Also, please see http://stackoverflow.com/questions/1849693/simple-suggestion-recommendation-algorithm
      singulars
      1. POOptimize algorithm for creating a list of items rated together, in Python
      1. USbwawok
    2. COlets say 2 million events with about 5000 users and 10000 items. Wouldn't K-nearest neighbors involve comparing every item against every other item, even ones with no overlap?
      singulars
      1. POOptimize algorithm for creating a list of items rated together, in Python
      1. USNeil Kodner
    3. COIf you're only interested in how many times items were bought together, why are you worried about users? You could group simply by purchase events, so that you'd have a dictionary of products keyed to other products bought with them. In fact, you could key each product to a max-heap of other products with values of how many times they're bough together, such that you'd be able to efficiently get the most commonly purchased item(s) for any given item. (The heap bit might go too far). If this is a workable idea for you I can write up a more complete answer.
      singulars
      1. POOptimize algorithm for creating a list of items rated together, in Python
      1. USnearlymonolith
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload