Databases
StackOverflow2013
Postsdbo
PONot able to connect to remote EC2-hbase cluster from EMR hive cluster
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PONot able to connect to remote EC2-hbase cluster from EMR hive cluster
    primarykey
    data
    text
    <p>Before i ask my question let me first explain the scenario. I had setup the <code>hbase</code> cluster on <code>ec2</code> with 3 instances: </p> <pre><code>i-xxxxxxx-- master, zookeeper1, regionserver1 i-xxxxxxx-- slave1, zookeeper2, regionserver2 i-xxxxxxx-- slave2, zookeeper3, regionserver3. </code></pre> <p>Which was working perfectly fine. Now, I was trying to connect to master of this cluster with the remote EMR instance with hive installed. So, I followed this link of amazon aws documentation :</p> <pre><code>http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hbase-access-hive.html </code></pre> <p>Which says before you create some table. Let hive knows the public-DNS of remote hbase master. set hbase.zookeeper.quorum=public-DNS-name; I did exactly which has been mentioned. But, I was not able to connect to the master. I tried create one table using the following script :</p> <pre><code>CREATE TABLE hauto(cookie string, timespent string, pageviews string, visit string, logdate string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = "m:timespent, m:pageviews, m:visit, m:logdate") TBLPROPERTIES ("hbase.table.name" = "hauto"); </code></pre> <p>So, it gave this error :</p> <pre><code>FAILED: Error in metadata: MetaException(message:org.apache.hadoop.hbase.MasterNotRunningException: Retried 10 times at org.apache.hadoop.hbase.client.HBaseAdmin.&lt;init&gt;(HBaseAdmin.java:127) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getHBaseAdmin(HBaseStorageHandler.java:74) at org.apache.hadoop.hive.hbase.HBaseStorageHandler.preCreateTable(HBaseStorageHandler.java:148) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:467) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:460) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74) at com.sun.proxy.$Proxy14.createTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:600) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3791) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:258) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1355) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1139) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:945) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:310) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:231) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:466) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:819) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:674) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:187) ) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask </code></pre> <p>FYI, <code>EMR</code> hive installed instance and EC2 <code>Hbase-cluster</code> are in same security group. I am able to telnet to 2181, 60000, 60010, 60020 ports from EMR to EC2 as well as EC2 to EMR. And in that group i have allowed the tcp, udp and ICMP connections on all ports within the group. </p> <p>My /etc/hosts file of master look like this:</p> <pre><code>XXXXXXXXXXX master localhost XXXXXXXXXXX zkserver1 rgserver1 XXXXXXXXXXX zkserver2 rgserver2 XXXXXXXXXXX zkserver3 rgserver3 </code></pre> <p>I am using <code>hadoop 1.0.1</code>, <code>hbase 0.94.11</code>, <code>hive 0.11.0</code> and <code>zookeeper-3.4.3</code>. And zookeeper is manager externally.</p> <p>Also, after that i tried other way around. This time i tried to connect from hive CLI installed on EC2 Hbase cluster to EMR Hbase. Now, i was able to create the same table on EMR hbase. </p> <p><strong>Updated Question:</strong></p> <p>It seems like this is an issue with amazon ec2. I am providing you the snapshot of those issues also the link :</p> <pre><code>http://hbase.apache.org/book.html#trouble.log.gc </code></pre> <p><img src="https://i.stack.imgur.com/PtpK2.png" alt="enter image description here"></p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USNaresh
    1. USNaresh
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PONot able to connect to remote EC2-hbase cluster from EMR hive cluster
      1. USNaresh
      1. VTBountyStart
    2. VO
      singulars
      1. PONot able to connect to remote EC2-hbase cluster from EMR hive cluster
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PONot able to connect to remote EC2-hbase cluster from EMR hive cluster
      1. This table or related slice is empty.
      1. VTUpMod
    1. CODid you enable all the necessary ports in the security group (firewall) ?
      singulars
      1. PONot able to connect to remote EC2-hbase cluster from EMR hive cluster
      1. USOussama
    2. COYes, I have kept the my Ec2 hbase cluster on the same security group in which EMR by default spin the Hadoop-hive instances. And in that group i have allowed the tcp connections on all ports within the group.
      singulars
      1. PONot able to connect to remote EC2-hbase cluster from EMR hive cluster
      1. USNaresh
    3. COYou should allow the udp too
      singulars
      1. PONot able to connect to remote EC2-hbase cluster from EMR hive cluster
      1. USOussama
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload