Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <blockquote> <p>But what should I do if I want to merge these outputs after the job by HDFS API for java?</p> </blockquote> <p>Guessing, because I haven't tried this myself, but I think the method you are looking for is <a href="http://hadoop.apache.org/docs/r0.22.0/api/org/apache/hadoop/fs/FileUtil.html#copyMerge%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20boolean,%20org.apache.hadoop.conf.Configuration,%20java.lang.String%29" rel="nofollow noreferrer">FileUtil.copyMerge</a>, which is the method that FsShell invokes when you run the <code>-getmerge</code> command. <code>FileUtil.copyMerge</code> takes two FileSystem objects as arguments - FsShell uses FileSystem.getLocal to retrieve the destination FileSystem, but I don't see any reason you couldn't instead use Path.getFileSystem on the destination to obtain an OutputStream</p> <p>That said, I don't think it wins you very much -- the merge is still happening in the local JVM; so you aren't really saving very much over <code>-getmerge</code> followed by <code>-put</code>.</p>
    singulars
    1. This table or related slice is empty.
    1. POHadoop: How can i merge reducer outputs to a single file?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USAndrea Bergonzo
    1. USVoiceOfUnreason
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POHadoop: How can i merge reducer outputs to a single file?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThanks for your answer. I have just tried like this: `String srcPath = "/user/hadoop/output"; String dstPath = "/user/hadoop/merged_file"; Configuration conf = new Configuration(); try { FileSystem hdfs = FileSystem.get(conf); FileUtil.copyMerge(hdfs, new Path(srcPath), hdfs, new Path(dstPath), false, conf, null); } catch (IOException e) { }`. That succesully merged output files as a single file on hdfs, and the order is just as my expection. But I have another question now. How does the function konw the files order ?
      singulars
      1. PO
      1. USthomaslee
    2. COHere's the implementation of copyMerge: http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-320/org/apache/hadoop/fs/FileUtil.java#FileUtil.copyMerge%28org.apache.hadoop.fs.FileSystem%2Corg.apache.hadoop.fs.Path%2Corg.apache.hadoop.fs.FileSystem%2Corg.apache.hadoop.fs.Path%2Cboolean%2Corg.apache.hadoop.conf.Configuration%2Cjava.lang.String%29 It looks like it's all down to the ordering of the items returned by the FileSystem's listStatus method. I'd guess that your output files are just concatenated together.
      singulars
      1. PO
      1. USBen McCracken
    3. CO@ Thomas, Ben : I am trying to merge files from my reducer's output using FileUtil.copyMerge. However I have a question here, the source destination contains _SUCCESS and _log files too apart from part-r-00000. part-r-00001. Does copyMerge take in only reducer output files or should I explicitly filter what files have to me merged? If yes, how can I do that? Thanks.
      singulars
      1. PO
      1. USuser1707141
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload