我正在使用Spark 1.6.2开发Python Spark客户端(它在
yarn-client
模式)。重要的是,始终在客户端机器中,
我无法提交我的Python脚本,但我需要将脚本作为Python脚本运行
.
在代码的某些地方,我需要在HDFS中加载一个CSV文件作为Spark数据帧(即使用SQL上下文)。您可能知道,Spark 1.6.2不支持基于CSV的数据帧,并且
Databricks spark-csv
必须使用。
数据加载语句如下:
df = sql_context.read.load(format='com.databricks.spark.csv', path=url, header=True, inferSchema=False, delimiter=',')
问题是
com.databricks.spark.csv
未找到
.
因为我不知道,我已经在客户端机器上尝试过了,但没有成功:
-
export PYTHONPATH=/path/where/jars/were/downloaded/
.
-
conf = SparkConf().set('spark.jars', '/path/where/jars/were/downloaded/')
.
-
conf = SparkConf().set('spark.driver.extraClassPath', '/path/where/jars/were/downloaded/')
. [
ref
]
我也曾在Spark cluster尝试过这一点,但也没有成功:
-
设置自定义spark defaults特性
spark.jars
通过Ambari。
-
设置自定义spark defaults特性
spark.driver.extraClassPath
通过Ambari。
我会记住命令行选项
--jars
或
--packages
不适合我,因为我没有运行任何Spark脚本:)
其他解决方案,例如使用
addJar()
那么,知道我的代码如何找到Databricks spark csv jar吗?
以防万一,这是错误跟踪:
java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:77)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:102)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.databricks.spark.csv.DefaultSource
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:62)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:62)
at scala.util.Try.orElse(Try.scala:82)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:62)
... 14 more
我读过的其他帖子。。。
提前谢谢。
