Databases
StackOverflow2013
Postsdbo
POElastic Mapreduce Map output lost
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POElastic Mapreduce Map output lost
    primarykey
    data
    text
    <p>I'm running a large (more than 100 nodes) series of mapreduce jobs on Amazon Elastic MapReduce.</p> <p>In the reduce phase, already-completed map tasks keep failing with</p> <pre><code>Map output lost, rescheduling: getMapOutput(attempt_201204182047_0053_m_001053_0,299) failed : java.io.IOException: Error Reading IndexFile at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:113) at org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:66) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3810) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at java.io.DataInputStream.readLong(DataInputStream.java:399) at org.apache.hadoop.mapred.SpillRecord.&lt;init&gt;(SpillRecord.java:74) at org.apache.hadoop.mapred.SpillRecord.&lt;init&gt;(SpillRecord.java:54) at org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:109) ... 23 more </code></pre> <p>The proportion of mappers for which this happens is few enough that I wouldn't mind except that when it does, the reducers all pause and wait for the 1 map-task to rerun so the entire job keeps pausing for 1-5 minutes each time.</p> <p>I think this is related to this bug -> <a href="https://issues.apache.org/jira/browse/MAPREDUCE-2980" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/MAPREDUCE-2980</a> Does anyone know how to run an EMR job without this happening?</p> <p>EDIT: Here's some more information if it's any help. The input format is <code>SequenceFileInputFormat</code>. The output format is a slightly modified version of <code>SequenceFileOutputFormat</code>. The key-value pairs are user-defined (the value is large and implements <code>Configurable</code>). There's no <code>Combiner</code>, just <code>Mapper</code> and <code>Reducer</code>. I'm using block compression for the input and output (and also there's record compression going on for intermediate kv pairs. This is default for EMR). The codec is the default which is <code>SnappyCodec</code> I believe. Finally, it's actually a series of jobs which are run in sequence, each one using the output of the previous job as the input to the next. The first couple jobs are small and run fine. It's only when the jobs start to grow really big that this happens.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCharles Menguy
    1. USdspyz
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POElastic Mapreduce Map output lost
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POElastic Mapreduce Map output lost
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POElastic Mapreduce Map output lost
      1. USdspyz
      1. VTBountyStart
    1. COI attempted using AMI 1.0 and hadoop 20.2, but I ran into a completely different bug. Something having to do with memory, starting processes and forking.
      singulars
      1. POElastic Mapreduce Map output lost
      1. USdspyz
    2. COseems like a configuration problem to me. did you define tmp folder for map-reduce job? checkout hadoop.tmp.dir property and if you have access to that folder. can you please post your code / config ?
      singulars
      1. POElastic Mapreduce Map output lost
      1. USfrail
    3. COI'm using all EMR defaults. It's a whole project. There's no specific part of my code I can post that will be particularly helpful here.
      singulars
      1. POElastic Mapreduce Map output lost
      1. USdspyz
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload