Databases
StackOverflow2013
Postsdbo
POSetting the number of map tasks and reduce tasks
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSetting the number of map tasks and reduce tasks
    primarykey
    data
    text
    <p>I am currently running a job I fixed the number of map task to 20 but and getting a higher number. I also set the reduce task to zero but I am still getting a number other than zero. The total time for the MapReduce job to complete is also not display. Can someone tell me what I am doing wrong. I am using this command</p> <pre><code>hadoop jar Test_Parallel_for.jar Test_Parallel_for Matrix/test4.txt Result 3 \ -D mapred.map.tasks = 20 \ -D mapred.reduce.tasks =0 </code></pre> <p>Output:</p> <pre><code>11/07/30 19:48:56 INFO mapred.JobClient: Job complete: job_201107291018_0164 11/07/30 19:48:56 INFO mapred.JobClient: Counters: 18 11/07/30 19:48:56 INFO mapred.JobClient: Job Counters 11/07/30 19:48:56 INFO mapred.JobClient: Launched reduce tasks=13 11/07/30 19:48:56 INFO mapred.JobClient: Rack-local map tasks=12 11/07/30 19:48:56 INFO mapred.JobClient: Launched map tasks=24 11/07/30 19:48:56 INFO mapred.JobClient: Data-local map tasks=12 11/07/30 19:48:56 INFO mapred.JobClient: FileSystemCounters 11/07/30 19:48:56 INFO mapred.JobClient: FILE_BYTES_READ=4020792636 11/07/30 19:48:56 INFO mapred.JobClient: HDFS_BYTES_READ=1556534680 11/07/30 19:48:56 INFO mapred.JobClient: FILE_BYTES_WRITTEN=6026699058 11/07/30 19:48:56 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1928893942 11/07/30 19:48:56 INFO mapred.JobClient: Map-Reduce Framework 11/07/30 19:48:56 INFO mapred.JobClient: Reduce input groups=40000000 11/07/30 19:48:56 INFO mapred.JobClient: Combine output records=0 11/07/30 19:48:56 INFO mapred.JobClient: Map input records=40000000 11/07/30 19:48:56 INFO mapred.JobClient: Reduce shuffle bytes=1974162269 11/07/30 19:48:56 INFO mapred.JobClient: Reduce output records=40000000 11/07/30 19:48:56 INFO mapred.JobClient: Spilled Records=120000000 11/07/30 19:48:56 INFO mapred.JobClient: Map output bytes=1928893942 11/07/30 19:48:56 INFO mapred.JobClient: Combine input records=0 11/07/30 19:48:56 INFO mapred.JobClient: Map output records=40000000 11/07/30 19:48:56 INFO mapred.JobClient: Reduce input records=40000000 [hcrc1425n30]s0907855: </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USZeissS
    1. USasembereng
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTDuplicate
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSetting the number of map tasks and reduce tasks
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POSetting the number of map tasks and reduce tasks
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POSetting the number of map tasks and reduce tasks
      1. USAmit
      1. VTFavorite
    1. COAre you also setting mapred.map.tasks in an xml configuration and/or the main of the class you're running? If so, does changing those settings change the number of tasks being performed? It looks like you are doing this correctly since properties specified at the command line should have the highest precedence.
      singulars
      1. POSetting the number of map tasks and reduce tasks
      1. USMatt D
    2. COIt should work but I am getting more map tasks than specified. And why is it that I am not getting the total time taken to run job?
      singulars
      1. POSetting the number of map tasks and reduce tasks
      1. USasembereng
    3. COI'm not sure about the time not being printed, but a possible source of error for the number of tasks is the spacing in your `-D` properties. Make sure you are either spelling it `-Dproperty=value` (with no spaces) or `-Dproperty value` (with one space) or else it might be parsed wrong.
      singulars
      1. POSetting the number of map tasks and reduce tasks
      1. USMatt D
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload