Databases
StackOverflow2013
Postsdbo
POOvercoming HDFS-2556: unit tests directory and permissions
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOvercoming HDFS-2556: unit tests directory and permissions
    primarykey
    data
    text
    <p>I'm trying to a JUnit tests on hadoop, I wrote a test case that I run on a local mini-hadoop environment using MiniDFSCluster,MiniMRCluster and JUnit.</p> <p>however I'm getting this error:</p> <pre><code> WARNING: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-datanode.properties, hadoop-metrics2.properties Dec 18, 2012 4:42:29 PM org.apache.hadoop.hdfs.server.datanode.DataNode makeInstance WARNING: Invalid directory in dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data1, expected: rwxr-xr-x, while actual: rwxrwxr-x Dec 18, 2012 4:42:29 PM org.apache.hadoop.hdfs.server.datanode.DataNode makeInstance WARNING: Invalid directory in dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data2, expected: rwxr-xr-x, while actual: rwxrwxr-x Dec 18, 2012 4:42:29 PM org.apache.hadoop.hdfs.server.datanode.DataNode makeInstance SEVERE: All directories in dfs.data.dir are invalid. </code></pre> <p>This looks like the HDFS bug <a href="https://issues.apache.org/jira/browse/HDFS-2556" rel="nofollow">https://issues.apache.org/jira/browse/HDFS-2556</a> that is still pending.</p> <p>Since I can't change the hadoop version i'm using, is there a way to force hadoop to accept these permissions?</p> <p>I tried: System.setProperty("dfs.datanode.data.dir.perm", "775"); But hadoop ignores them.</p> <p>Also, I can't change the permissions manually, it seems that the mini-cluster re-creates them every run.</p> <p>Is there a way to overcome this without modifing hadoop or changing version? Thanks.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USApriOri
    1. USApriOri
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POOvercoming HDFS-2556: unit tests directory and permissions
      1. This table or related slice is empty.
      1. VTUpMod
    1. COPlease also post the specific version of Hadoop you are using. The solution lies in configuring the ``MiniDFSCluster`` to use a proper ``dfs.datanode.data.dir.perm`` but this feature is not present in some older/maintenance/stable releases.
      singulars
      1. POOvercoming HDFS-2556: unit tests directory and permissions
      1. USHarsh J
    2. COI had this problem on both 1.0.4 and 1.1.1. Can you please tell which hadoop version respect this property ?
      singulars
      1. POOvercoming HDFS-2556: unit tests directory and permissions
      1. USApriOri
    3. COThe property was added in, and is present under versions 2.x+, or in other distributions of Apache Hadoop such as CDH3 (1.x + backports) and CDH4 (2.x + backports). To set it you will need to directly set it into the configuration object you pass to the MiniDFSCluster constructor, and not via a system property.
      singulars
      1. POOvercoming HDFS-2556: unit tests directory and permissions
      1. USHarsh J
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload