Databases
StackOverflow2013
Postsdbo
PORunning a standalone Hadoop application on multiple CPU cores
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORunning a standalone Hadoop application on multiple CPU cores
    primarykey
    data
    text
    <p>My team built a Java application using the Hadoop libraries to transform a bunch of input files into useful output. Given the current load a single multicore server will do fine for the coming year or so. We do not (yet) have the need to go for a multiserver Hadoop cluster, yet we chose to start this project "being prepared".</p> <p>When I run this app on the command-line (or in eclipse or netbeans) I have not yet been able to convince it to use more that one map and/or reduce thread at a time. Given the fact that the tool is very CPU intensive this "single threadedness" is my current bottleneck.</p> <p>When running it in the netbeans profiler I do see that the app starts several threads for various purposes, but only a single map/reduce is running at the same moment.</p> <p>The input data consists of several input files so Hadoop should at least be able to run 1 thread per input file at the same time for the map phase.</p> <p>What do I do to at least have 2 or even 4 active threads running (which should be possible for most of the processing time of this application)?</p> <p>I'm expecting this to be something very silly that I've overlooked.</p> <hr> <p>I just found this: <a href="https://issues.apache.org/jira/browse/MAPREDUCE-1367" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/MAPREDUCE-1367</a> This implements the feature I was looking for in Hadoop 0.21 It introduces the flag mapreduce.local.map.tasks.maximum to control it.</p> <p>For now I've also found the solution described <a href="https://stackoverflow.com/questions/3546025/is-it-possible-to-run-hadoop-in-pseudo-distributed-operation-without-hdfs">here in this question</a>. </p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USNiels Basjes
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PORunning a standalone Hadoop application on multiple CPU cores
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PORunning a standalone Hadoop application on multiple CPU cores
      1. USjandot
      1. VTFavorite
    3. VO
      singulars
      1. PORunning a standalone Hadoop application on multiple CPU cores
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload