Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I see a couple things amiss with this code. First is this line:</p> <pre><code>record = open(sys.argv[1]) </code></pre> <p>I find it odd that this <code>record</code> variable is never used anywhere else in the code. Even though the <code>mapper</code> function is defined as follows:</p> <pre><code>def mapper(record): ... </code></pre> <p>...<em>that</em> <code>record</code> is local to the <code>mapper</code> function. It's in a different scope than the first <code>record</code>. Whatever data is passed to <code>mapper</code> is assigned to its local <code>record</code> and used accordingly, and the file object assigned to the first <code>record</code> is never touched. I don't feel that this is tied to the error, though. Because that first <code>record</code> is not used anywhere else, you can pretty safely delete that line.</p> <p>Then there's the <code>reducer</code> function:</p> <pre><code>def reducer(key, value): # reducer should take 2 inputs according to the task if key in new_dict: # checking if key already added to dict new_dict[key].extend(list(value)) # if yes just append all records to the value new_dict[key] = list(value) # if not create new key and assign record to value for key in new_dict: if len(new_dict[key]) == 27: # checks to emit only records found in both tables mr.emit(new_dict[key]) </code></pre> <p>Your own comments offer the clue to the problem here. First you say you're checking to see if the key is already in the dict. If so, just append all records to the value. If not, create a new key and assign the record to the value.</p> <p>The problem is with the line associated with the "if not" comment. If it's truly what should be done if the first <code>if</code> test fails, then it should be prefaced by an <code>else</code> line:</p> <pre><code> ... if key in new_dict: # checking if key already added to dict new_dict[key].extend(list(value)) # if yes just append all records to the value else: new_dict[key] = list(value) # if not create new key and assign record to value ... </code></pre> <p>The way you wrote it, even if that <code>if</code> test succeeds and it appends the data to the existing value for the key, it's going to immediately stomp over that change. In other words, the value for that key isn't going to grow. It's always going to represent the most recently submitted value for the key.</p> <p>Here's the full code edited with all the suggested changes:</p> <pre><code>import MapReduce import sys """ Word Count Example in the Simple Python MapReduce Framework """ mr = MapReduce.MapReduce() # ============================= # Do not modify above this line def mapper(record): key = record[1] # assign order_id from each record as key value = list(record) # assign whole record as value for each key mr.emit_intermediate(key, value) # emit key-value pairs new_dict = {} # create dict to keep track of records def reducer(key, value): if not key in new_dict: new_dict[key] = value else: new_dict[key].extend(value) for key in new_dict: if len(new_dict[key]) == 27: mr.emit(new_dict[key]) # Do not modify below this line # ============================= if __name__ == '__main__': inputdata = open(sys.argv[1]) mr.execute(inputdata, mapper, reducer) </code></pre>
    singulars
    1. This table or related slice is empty.
    1. POPython - implementing join in MapReduce - problems with reducer output
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USJustin S Barrett
    1. USJustin S Barrett
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. COThanks for pointing to my error with missing else statement. I have fixed that as you instructed. Alas, I am still getting the same error. I thought it may be because of my conditions (if len(new_dict[key]) == 27), so I removed that, but program still return 0 values. For comparison, I am also posting code for similar task, which worked well for me, see my next comment below.
      singulars
      1. PO
      1. USSergey Samusev
    2. COOops, I actually could not post code as comment, so I added that at the end of my initial problem description. Thanks again Justin!
      singulars
      1. PO
      1. USSergey Samusev
    3. COI just edited my answer to add a response regarding the new code. Perhaps that solves the problem?
      singulars
      1. PO
      1. USJustin S Barrett
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload