DataNucleus 对您几乎没有任何影响。 如果您尚未提供默认构造函数,则 DataNucleus 增强器提供添加默认构造函数的功能。 对于您使用的任何数据存储,您都需要 javax.jdo 以及 datanucleus-api-jdo、datanucleus-core 和 datanucleus-XXX jar。 datanucleus-core.jar :它提供了基本的 DataNucleus 持久化机制,所有 DataNucleus 插件都需要它。 DataNucleus JDO 提供了自己的字节码增强器,用于检测/增强您的类(在 datanucleus-core.jar 中),这包含在 DataNucleus AccessPlatform zip 、datanucleus-api-jdo.jar 和 datanucleus-{datastore}.jar(对于您正在使用的数据存储,例如使用 RDBMS 时的 datanucleus-rdbms.jar
如果set后面什么都不添加,这样可以查到Hive的所有属性配置,如下: [java] hive> set; datanucleus.autoCreateSchema=true datanucleus.autoStartMechanismMode =checked datanucleus.cache.level2=false datanucleus.cache.level2.type=none datanucleus.connectionPoolingType =DBCP datanucleus.identifierFactory=datanucleus datanucleus.plugin.pluginRegistryBundleCheck=LOG datanucleus.storeManagerType=rdbms datanucleus.transactionIsolation=read-committed datanucleus.validateColumns =false datanucleus.validateConstraints=false datanucleus.validateTables=false ....................
-3.2.6.jar, lib/datanucleus-rdbms-3.2.9.jar, lib/datanucleus-core-3.2.10.jar, lib/mysql-connector-java -3.2.6.jar, lib/datanucleus-rdbms-3.2.9.jar, lib/datanucleus-core-3.2.10.jar, lib/mysql-connector-java -3.2.6.jar, lib/datanucleus-rdbms-3.2.9.jar, lib/datanucleus-core-3.2.10.jar, lib/mysql-connector-java -3.2.6.jar, lib/datanucleus-rdbms-3.2.9.jar, lib/datanucleus-core-3.2.10.jar, lib/mysql-connector-java -3.2.6.jar, lib/datanucleus-rdbms-3.2.9.jar, lib/datanucleus-core-3.2.10.jar, lib/mysql-connector-java
</name> <value>false</value> </property> <property> <name>datanucleus.fixedDatastore </name> <value>false</value> </property> <property> <name>datanucleus.autoCreateSchema </name> <value>true</value> </property> <property> <name>datanucleus.schema.autoCreateAll </name> <value>true</value> </property> <property> <name>datanucleus.autoCreateTables </name> <value>true</value> </property> <property> <name>datanucleus.autoCreateColumns
:1187) at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356) at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration (AbstractTable.java:760) at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatementList :3190) at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2841) ) at org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:1605) at org.datanucleus.store.AbstractStoreManager.addClass (AbstractStoreManager.java:954) at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(
(NucleusJDOHelper.java:671) ~[datanucleus-api-jdo-4.2.4.jar:?] at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java :830) ~[datanucleus-api-jdo-4.2.4.jar:?] :334) ~[datanucleus-api-jdo-4.2.4.jar:?] :213) ~[datanucleus-api-jdo-4.2.4.jar:?]
appender.DRFA.strategy.max = 30 # list of all loggers loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO logger.ClientCnxnSocketNIO.level = WARN logger.DataNucleus.name = DataNucleus logger.DataNucleus.level = ERROR logger.Datastore.name = Datastore logger.Datastore.level
(PluginManager.java:325) 73 at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory (AbstractStoreManager.java:282) 74 at org.datanucleus.store.AbstractStoreManager. <init>(AbstractStoreManager.java:240) 75 at org.datanucleus.store.rdbms.RDBMSStoreManager. :1187) 83 at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356) 84 at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration (ConnectionFactoryImpl.java:131) 89 at org.datanucleus.store.rdbms.ConnectionFactoryImpl.
--</repository>--> <repository> <id>Datanucleus</id> <name>Datanucleus</name> <url>http://www.datanucleus.org/downloads/maven2/</url> </repository> <repository> <id>
运行CarbonData 在 SPARK_HOME/lib 下还有三个datanucleus开头的包,我们也通过--jars 参数加上 . 2.10-0.1.0-SNAPSHOT-shade-hadoop2.6.0.jar,/Users/allwefantasy/Softwares/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo -3.2.6.jar,/Users/allwefantasy/Softwares/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar,/Users /allwefantasy/Softwares/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar,/Users/allwefantasy -3.2.6.jar SPARK_HOME/lib/datanucleus-core-3.2.10.jar SPARK_HOME/lib/datanucleus-rdbms-3.2.9.jar 然后就运行起来了
groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>org.datanucleus </groupId> <artifactId>datanucleus-core</artifactId> </exclusion> <exclusion> groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>org.datanucleus </groupId> <artifactId>datanucleus-core</artifactId> </exclusion> <exclusion>
jar包(等下再一一说明) 包括scala,spark-assembly-1.4.1-hadoop2.5.0-cdh5.2.1.jar(按照你的spark和hadoop版本),还有在spark的lib下datanucleus 如下图的错误,是因为声明hivecontext时没有找到hive的metastore,所以一定要引入datanucleus的jar,将hive下的 hive-site-xml的配置文件放入项目中,引入mysql
javax.jdo.option.ConnectionPassword</name> <value>mypassword</value> </property> <property> <name>datanucleus.autoCreateSchema </name> <value>false</value> </property> <property> <name>datanucleus.fixedDatastore</name> <value> true</value> </property> <property> <name>datanucleus.autoStartMechanism</name> <value>SchemaTable< async.exec.threads 8192 hive.server2.async.exec.wait.queue.size 8192 hive.server2.thrift.max.worker.threads 8192 datanucleus.connectionPool.maxPoolSize
DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.autoCreateTables" --> 108 109 <property> 110 111 <name>datanucleus.autoCreateSchema</name> 112 113 <value>true</value> 114 115 </property> 116 117 <property> 118 119 <name>datanucleus.autoCreateTables</name> 120 121 <value>true</value> 122 123 </property> 124 125 <property> 126 127 <name>datanucleus.autoCreateColumns
javax.jdo.option.ConnectionPassword</name> <value>hadoop01</value> </property> <property> <name>datanucleus.autoCreateSchema </name> <value>false</value> </property> <property> <name>datanucleus.autoCreateTables
2.3.4.jar$HIVE_HOME/lib/hive-shims-common-2.3.4.jar$HIVE_HOME/lib/antlr-runtime-3.5.2.jar$HIVE_HOME/lib/datanucleus-api-jdo -4.2.4.jar$HIVE_HOME/lib/datanucleus-core-4.1.17.jar$HIVE_HOME/lib/datanucleus-rdbms-4.1.19.jar$HIVE_HOME
Metastore 通过 DataNucleus 将对象定义保存到关系数据库 (RDBMS),DataNucleus 是一个基于 Java JDO 的对象关系映射 (ORM) 层。 datanucleus.schema.autoCreateAll datanucleus.schema.autoCreateAll false 如果不存在,则自动在启动时在 RDBMS 中创建必要的模式 javax.jdo.option.ConnectionUserName User name to connect to the RDBMS with 支持的RDBMSs 由于 Metastore 使用 DataNucleus 与 RDBMS 进行通信,因此理论上 DataNucleus 支持的任何存储选项都可以与 Metastore 一起使用。
:使用连接池来访问JDBC metastore,默认是DBCP; datanucleus.validateTables:检查是否存在表的schema,默认是false; datanucleus.validateColumns :元数据存储类型,默认rdbms; datanucleus.autoCreateSchema:在不存在时是否自动创建必要的schema,默认是true; datanucleus.aotuStartMechanismMode :如果元数据表不正确,抛出异常,默认是checked; datanucleus.transactionIsolation:默认的事务隔离级别,默认是read-committed; datanucleus.cache.level2 :使用二级缓存,默认是false; datanucleus.cache.level2.type:二级缓存的类型,有两种,SOFT:软引用,WEAK:弱引用,默认是SOFT; datanucleus.identifierFactory :id工厂生产表和列名的名字,默认是datanucleus; datanucleus.plugin.pluginRegistryBundleCheck:当plugin被发现并且重复时的行为,默认是LOG
jackson-jaxrs-1.9.13.jar mysql-connector-java-5.1.38.jar api-util-1.0.0-M20.jar datanucleus-api-jdo jackson-mapper-asl-1.9.13.jar netty-3.6.2.Final.jar asm-3.1.jar datanucleus-core jackson-mapper-asl-1.9.2.jar opencsv-2.3.jar asm-3.2.jar datanucleus-rdbms
fs.defaultFS</name> <value>hdfs://bigdata:9000</value> </property> <property> <name>datanucleus.autoCreateSchema </name> <value>true</value> </property> <property> <name>datanucleus.autoStartMechanism </name> <value>SchemaTable</value> </property> <property> <name>datanucleus.schema.autoCreateTables