- 浏览: 230523 次
- 性别:
- 来自: 上海
文章分类
最新评论
-
lwb314:
你的这个是创建的临时的hive表,数据也是通过文件录入进去的, ...
Spark SQL操作Hive数据库 -
yixiaoqi2010:
你好 我的提交上去 总是报错,找不到hive表,可能是哪里 ...
Spark SQL操作Hive数据库 -
bo_hai:
target jvm版本也要选择正确。不能选择太高。2.10对 ...
eclipse开发spark程序配置本地运行
在使用derby数据库时,遇到了很多错误,报错信息和解决方法如下,记录下来:
一,配置问题
$HIVE_HOME/conf/hive-site.xml文件中缺少配置,修改下面的配置即可:
二,jar版本问题
hadoop的版本是2.6.0,hive的版本是1.2.1,$HIVE_HOME/lib目录下的jline-2.12.jar比$HADOOP_HOME/share/hadoop/yarn/lib下的jline-0.9.94.jar版本高,版本不一致导致。
解决方法:拷贝hive中的jline-2.12.jar到$HADOOP_HOME/share/hadoop/yarn/lib下,并重启hadoop即可。
root@master1:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib# mv jline-0.9.94.jar jline-0.9.94.jar20160305
root@master1:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib# cp $HIVE_HOME/lib/jline-2.12.jar ./
参考文章:http://www.linuxidc.com/Linux/2015-10/123822.htm
三,hadoop安全模式问题
解决办法:关闭安全模式
hadoop dfsadmin -safemode leave
参考文章:http://blog.csdn.net/wenlong237/article/details/11612847
一,配置问题
Logging initialized using configuration in jar:file:/usr/local/hive/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at org.apache.hadoop.fs.Path.initialize(Path.java:206) at org.apache.hadoop.fs.Path.<init>(Path.java:172) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:563) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508) ... 8 more Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D at java.net.URI.checkPath(URI.java:1823) at java.net.URI.<init>(URI.java:745) at org.apache.hadoop.fs.Path.initialize(Path.java:203) ... 11 more
$HIVE_HOME/conf/hive-site.xml文件中缺少配置,修改下面的配置即可:
<property> <name>hive.querylog.location</name> <value>/usr/local/hive/iotmp/</value> <description>Location of Hive run time structured log file</description> </property> <property> <name>hive.server2.logging.operation.log.location</name> <value>/usr/local/hive/iotmp/operation_logs</value> <description>Top level directory where operation logs are stored if logging functionality is enabled</description> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/usr/local/hive/iotmp/</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/usr/local/hive/iotmp/${hive.session.id}_resources</value> <description>Temporary local directory for added resources in the remote file system.</description> </property
二,jar版本问题
Logging initialized using configuration in file:/usr/local/hive/apache-hive-1.2.1-bin/conf/hive-log4j.properties [ERROR] Terminal initialization failed; falling back to unsupported java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected at jline.TerminalFactory.create(TerminalFactory.java:101) at jline.TerminalFactory.get(TerminalFactory.java:158) at jline.console.ConsoleReader.<init>(ConsoleReader.java:229) at jline.console.ConsoleReader.<init>(ConsoleReader.java:221) at jline.console.ConsoleReader.<init>(ConsoleReader.java:209) at org.apache.hadoop.hive.cli.CliDriver.setupConsoleReader(CliDriver.java:787) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:721) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Exception in thread "main" java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected at jline.console.ConsoleReader.<init>(ConsoleReader.java:230) at jline.console.ConsoleReader.<init>(ConsoleReader.java:221) at jline.console.ConsoleReader.<init>(ConsoleReader.java:209) at org.apache.hadoop.hive.cli.CliDriver.setupConsoleReader(CliDriver.java:787) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:721) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
hadoop的版本是2.6.0,hive的版本是1.2.1,$HIVE_HOME/lib目录下的jline-2.12.jar比$HADOOP_HOME/share/hadoop/yarn/lib下的jline-0.9.94.jar版本高,版本不一致导致。
解决方法:拷贝hive中的jline-2.12.jar到$HADOOP_HOME/share/hadoop/yarn/lib下,并重启hadoop即可。
root@master1:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib# mv jline-0.9.94.jar jline-0.9.94.jar20160305
root@master1:/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib# cp $HIVE_HOME/lib/jline-2.12.jar ./
参考文章:http://www.linuxidc.com/Linux/2015-10/123822.htm
三,hadoop安全模式问题
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/root/1069947e-cafd-4538-99d3-e4e8b5187380. Name node is in safe mode. The reported blocks 234 has reached the threshold 0.9990 of total blocks 234. The number of live datanodes 3 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 4 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4216) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4191) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:600) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/root/1069947e-cafd-4538-99d3-e4e8b5187380. Name node is in safe mode. The reported blocks 234 has reached the threshold 0.9990 of total blocks 234. The number of live datanodes 3 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 4 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4216) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4191) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:600) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy13.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:539) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy14.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2753) at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2724) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:870) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:866) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:866) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:859) at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:639) at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:574) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508) ... 8 more
解决办法:关闭安全模式
hadoop dfsadmin -safemode leave
参考文章:http://blog.csdn.net/wenlong237/article/details/11612847
发表评论
-
Spark SQL内置函数应用
2016-04-22 07:00 8567简单说明 使用Spark SQL中的内置函数对数据进行 ... -
Spark SQL操作Hive数据库
2016-04-13 22:37 17539本次例子通过scala编程实现Spark SQL操作Hive数 ... -
Spark SQL on hive配置和实战
2016-03-26 18:40 5487spark sql 官网:http://spark ... -
Hive分析搜索引擎的数据(搜狗实验室数据)
2016-03-13 12:48 1670搜狗实验室:http://www.sogou.com/labs ... -
Hive1.2.1安装和使用(基于Hadoop2.6.0)
2016-03-05 20:22 8742安装hive,这里使用mysql作为hive的metastor ... -
hive第一个字段为null
2014-03-20 23:17 1330在hive中创建表后,将准备好的数据导入到该表中,如果第一个字 ...
相关推荐
hive的一些报错及解决方法:连接报错、执行脚本卡住、内存溢出报错、表死锁问题
Hive运维中通常会用到操作元数据,这里提供了常用的sql语句
1)Failing because I am unlikely to write too. 2)Caused by: java.lang.OutOfMemoryError: Java heap space ...5)hive on tez 最终insert的表如果使用到union all 时会导致直接查询结果表数据为空的
背景:由于Hive需要开启...使用root用户登录,进入Hive命令行界面。 执行 create database test; 发现报错: Authorization failed:No privilege 'Create' found for outputs { }. Use SHOW GRANT to get more details
apache-hive-1.2.1-bin.tar.gz mysql-connector-java-5.1.6-bin 用于安装Hive并使用MySQL作为元数据库
基于Hadoop Hive健身馆可视化分析平台项目源码+数据库文件.zip启动方式 环境启动 hadoop hive2元数据库 sql导入 导入hivesql脚本,修改application.yml 启动主程序 HadoopApplication 基于Hadoop Hive健身馆可视化...
数据库修改为拉丁语的步骤。1、在hive中无法删除非空数据库,无法删除表的解决办法如下;2、在hive中加载数据导表中报错。直接下载到本地即可查看步骤
## Hive metastore 使用达梦数据库存储元数据 - 使用说明 1. **在达梦数据库中创建metastore将使用的用户和表空间** 2. **配置hive-site.xml** 3. **元数据初始化** ### 常见问题
1 Hive 概念与连接使用: 2 2 Hive支持的数据类型: 2 2.1原子数据类型: 2 2.2复杂数据类型: 2 2.3 Hive类型转换: 3 3 Hive创建/删除数据库 3 3.1创建数据库: 3 3.2 删除数据库: 3 4 Hive 表相关语句 3 4.1 Hive ...
Idea连接Hive,Idea连接Hive,Idea连接Hive,Idea连接Hive,Idea连接Hive
datax数据从hive表导入mysql表,数据缺失解决
使用dbeaver或者其他数据库管理工具连接hive时使用到的驱动
cloudera manager中添加hive数据库使用mysql的配置的详细步骤
jdbc连接hive数据库的jar包.整理可用合集.
创建hive的元数据库:create database metahive【元数据库名metahive】 5.创建hive用户:create user hive@‘localhost’ identified by ‘hive’【mysql中为hive专用的user名:hive,密码为:hive】 6.赋予...
本资源包含Hive数据库连接所需要的驱动,为了方便我们在用第三方工具链接Hive数据库的时候,解决驱动的问题,而自己在下载的时候,资源又不是很好找,所以在这里打包分享给大家。不过大家在下载使用的时候,还需要...
由于 Hive 采用了类似SQL 的查询语言 HQL(Hive Query Language),因此很容易将 Hive 理解为数据库。其实从结构上来看,Hive 和数据库除了拥有类似的查询语言,再无类似之处。本文将从多个方面来阐述 Hive ...
可能对于很多编程员来说注册表HIVE数据库都是个头痛的难题,这篇源代码或许有所帮助
hive1.2.1用mysql作为元数据库搭建DT-大数据 大数据学习可以了解一下