Databases
StackOverflow2013
Postsdbo
POCan't ssh permission denied. HBase Hadoop
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POCan't ssh permission denied. HBase Hadoop
    primarykey
    data
    text
    <p>I have an amazon ec2 instance that I'm running a tool called seqware on. It's basically a query engine for genomic data that uses an hbase backend. I'm running on an ami that comes with hbase set up in pseudo distributed mode. I would like to use it in a fully distributed mode however. So I have set up a 2 node hadoop cluster. One node is the master and the other is a slave. I can run hadoop examples in fully distributed mode and everything. To make seqware use my fully ditrbuted set up, it needs 6 things, the zookeeper quorum, zookeeper client port, hbase master, the mapred job tracker , the fs default fs and fs name. Which are specified, in a setting file. I have it set up in the file as so:</p> <pre><code>HBASE.ZOOKEEPER.QUORUM=ip-10-x.x.x HBASE.ZOOKEEPER.PROPERTY.CLIENTPORT=2181 HBASE.MASTER=ip-10-x.x.x:60010 MAPRED.JOB.TRACKER=ip-10-x.x.x:9001 FS.DEFAULT.NAME=hdfs://ip-10-x.x.x:9000 FS.DEFAULTFS=hdfs://ip-10-x.x.x:9000 </code></pre> <p>However, when I started using the query engine, I was getting a zookeeper connection loss exception. I had the the public key for the master in the authorized_keys for seqware and vice versa, however, I couldn't ssh as so</p> <pre><code>ssh ip-10.x.x.x </code></pre> <p>or even using the public dns:</p> <pre><code>ssh {public DNS of instance} </code></pre> <p>where ip-10.x.x.x is the ip address of the instance, I had to do it using the username:</p> <pre><code>ssh {username}@ip-10-x.x.x </code></pre> <p>or</p> <pre><code>ssh username@{public DNS of instance} </code></pre> <p>I can ssh without the username from the master to slave hadoop instance and vice versa, and I have the ip address in the config files without the username</p> <p>I tryed adding the username before the ip address in the settings, thinking that there a 99% chance it wouldn't work, and I wasn't disappointed, I still got the same exception</p> <p>What do I need to do so that I can ssh from the seqware instance into the hadoop and hbase master without having to specify a username, as I do between the master and slave nodes.</p> <p>This is how zookeeper is configured on the hadoop master:</p> <pre><code>&lt;configuration&gt; &lt;property&gt; &lt;name&gt;hbase.rootdir&lt;/name&gt; &lt;value&gt;hdfs://ip-10-x.x.x:9000/hbase&lt;/value&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;hbase.cluster.distributed&lt;/name&gt; &lt;value&gt;true&lt;/value&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt; &lt;value&gt;ip-10-x.x.x&lt;/value&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;dfs.replication&lt;/name&gt; &lt;value&gt;2&lt;/value&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt; &lt;value&gt;/home/{username}/hbase/zookeeper&lt;/value&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;hbase.zookeeper.property.clientPort&lt;/name&gt; &lt;value&gt;2181&lt;/value&gt; &lt;/property&gt; &lt;/configuration&gt; </code></pre> <p>I can't speak confidently about the internal implementation of seqware, but I know it uses the settings file to set the location of zookeepr and hbase master. In the default, pseudo-distributed working setup, these were the values for the variables I mentioned previously:</p> <pre><code>HBASE.ZOOKEEPER.QUORUM=localhost HBASE.ZOOKEEPER.PROPERTY.CLIENTPORT=2181 HBASE.MASTER=localhost:60000 MAPRED.JOB.TRACKER=localhost:8021 FS.DEFAULT.NAME=hdfs://localhost:8020 FS.DEFAULTFS=hdfs://localhost:8020 </code></pre> <p>This is what the zoo.cfg file looks like:</p> <pre><code># Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. maxClientCnxns=50 # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. dataDir=/var/lib/zookeeper # the port at which the clients will connect clientPort=2181 </code></pre> <p><strong>Zookeeper stacktrace:</strong></p> <p><code>[seqware@master target]$ java -classpath seqware-distribution-0.13.6.8-qe-full.jar</code> </p> <pre><code>com.github.seqware.queryengine.system.ReferenceCreator hg_19 keyValue_ref.out [SeqWare Query Engine] 0 [main] ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - ZooKeeper exists failed after 3 retries [SeqWare Query Engine] 1 [main] ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher - hconnection Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeep‌​er.java:154) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracke‌​r.java:82) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setu‌​pZookeeperTrackers(HConnectionManager.java:580) </code></pre> <hr> <p>08/13/2013 Apparently the variables that need to be set for the remote hbase setup were not the ones I was editing, based on the the seqware constants.java file, they are the qe variables: <a href="https://github.com/SeqWare/seqware/blob/develop/seqware-queryengine/src/main/java/com/github/seqware/queryengine/Constants.java" rel="nofollow noreferrer">https://github.com/SeqWare/seqware/blob/develop/seqware-queryengine/src/main/java/com/github/seqware/queryengine/Constants.java</a></p> <p>I have edited them as such:</p> <pre><code># SEQWARE QUERY ENGINE AND GENERAL HADOOP SETTINGS # HBASE.ZOOKEEPER.QUORUM=localhost HBASE.ZOOKEEPER.PROPERTY.CLIENTPORT=2181 HBASE.MASTER=localhost:60000 MAPRED.JOB.TRACKER=localhost:8021 FS.DEFAULT.NAME=hdfs://localhost:8020 FS.DEFAULTFS=hdfs://localhost:8020 FS.HDFS.IMPL=org.apache.hadoop.hdfs.DistributedFileSystem # # SEQWARE QUERY ENGINE SETTINGS # QE_NAMESPACE=SeqWareQE QE_DEVELOPMENT_DEPENDENCY=file:/home/seqware/jars/seqware-distribution-0.13.6.5-qe-full.jar QE_PERSIST=true QE_HBASE_REMOTE_TESTING=true QE_HBASE_PROPERTIES=HBOOT QE_HBOOT_HBASE_ZOOKEEPER_QUORUM=ip-10-x.x.x.ec2.internal QE_HBOOT_HBASE_ZOOKEEPER_PROPERTY_CLIENTPORT=2181 QE_HBOOT_HBASE_MASTER=ip-10-x.x.x.ec2.internal:60010 QE_HBOOT_MAPRED_JOB_TRACKER=ip-10-x.x.x.ec2.internal:9001 QE_HBOOT_FS_DEFAULT_NAME=hdfs://ip-10-x.x.x.ec2.internal:9000 QE_HBOOT_FS_DEFAULTFS=hdfs://ip-10-x.x.x.ec2.internal:9000 QE_HBOOT_FS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem </code></pre> <p>I don't get the zookeeper exceptiona anymore, but the command to create the workspace just hangs for several minutes before I stop it.</p> <p>I found this in my zookepper log, I'm not sure if it means, that zookeeper crashed or it lost connectivity to the clients it states. I don't know why it's accepting socket connections from the ports 36997,36998, 37000, and 37034, and I haven't even granted permission to them on the ec2 security group:</p> <pre><code>2013-08-13 16:44:55,560 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1407890cb630000 with negotiated timeout 180000 for client /10.x.x.x:36997 2013-08-13 16:44:57,633 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.x.x.x:36998 2013-08-13 16:44:57,662 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.x.x.x:36998 2013-08-13 16:44:57,666 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1407890cb630001 with negotiated timeout 180000 for client /10.x.x.x:36998 2013-08-13 16:44:57,917 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1407890cb630001 type:create cxid:0x8 zxid:0x219 txntype:-1 reqpath:n/a Error Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase/online-snapshot/acquired 2013-08-13 16:44:58,450 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1407890cb630000 type:create cxid:0xb zxid:0x21a txntype:-1 reqpath:n/a Error Path:/hbase/master Error:KeeperErrorCode = NodeExists for /hbase/master 2013-08-13 16:45:00,927 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.x.x.x:37000 2013-08-13 16:45:00,928 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.x.x.x:37000 2013-08-13 16:45:00,930 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1407890cb630002 with negotiated timeout 180000 for client /10.x.x.x:37000 2013-08-13 16:45:02,165 INFO org.apache.zookeeper.server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x1407890cb630000 type:create cxid:0x24 zxid:0x221 txntype:-1 reqpath:n/a Error Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase/online-snapshot/acquired 2013-08-13 16:45:14,172 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.x.x.x:37034 2013-08-13 16:45:14,173 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.x.x.x:37034 2013-08-13 16:45:14,178 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x1407890cb630003 with negotiated timeout 180000 for client /10.x.x.x:37034 2013-08-13 16:47:51,000 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1407800784a0003, timeout of 180000ms exceeded 2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1407800784a0001, timeout of 180000ms exceeded 2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1407800784a0000, timeout of 180000ms exceeded 2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1407800784a0002, timeout of 180000ms exceeded 2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1407800784a0003 2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1407800784a0001 2013-08-13 16:47:51,001 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1407800784a0000 2013-08-13 16:47:51,002 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x1407800784a0002 </code></pre> <p>I looked on the hbase web interface, its showing that the tables are actually being created, but the commands to create are never returning a reponse back, they just hang. <img src="https://i.stack.imgur.com/DAFhF.jpg" alt="Hbase master web interface tables"></p> <p><img src="https://i.stack.imgur.com/6Ne67.jpg" alt="HBase Web Interface"></p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAmre
    1. USAmre
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POCan't ssh permission denied. HBase Hadoop
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    1. COBy which user is executing the ssh ip-10.x.x.x command? Type 'which' in the terminal and reply here.
      singulars
      1. POCan't ssh permission denied. HBase Hadoop
      1. USSSaikia_JtheRocker
    2. COIt's being executed a user called seqware This is what I get when I type in which '[seqware@master ~]$ which Usage: /usr/bin/which [options] [--] programname [...] Options: --version, -[vV] Print version and exit successfully. --help, Print this help and exit successfully. . . .' it's not a sudoer, but I have the same problem for the instance running the hadoop master
      singulars
      1. POCan't ssh permission denied. HBase Hadoop
      1. USAmre
    3. COSo, where is your problem now? Only the ssh thing or Zookeeper connetion error? Because, I think ssh is not needed for connecting to a zookeeper server.
      singulars
      1. POCan't ssh permission denied. HBase Hadoop
      1. USSSaikia_JtheRocker
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload