代码之家 › 专栏 › 技术社区 › Suanmeiguo

为什么集群模式下的Spark-on-thread会因“线程中的异常”驱动程序“java.lang.NullPointerException”而失败?

emr nullpointerexception apache-spark

Suanmeiguo · 技术社区 · 7 年前

我正在使用emr-5.4.0和Spark 2.1.0。我明白什么 NullPointerException 这个问题是关于为什么会在这个特殊的案例中抛出这个问题。

无法真正理解为什么在驱动程序线程中出现NullPointerException。

我得到了一份奇怪的工作,但由于以下错误而失败:

18/03/29 20:07:52 INFO ApplicationMaster: Starting the user application in a separate Thread
18/03/29 20:07:52 INFO ApplicationMaster: Waiting for spark context initialization...
Exception in thread "Driver" java.lang.NullPointerException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
18/03/29 20:07:52 ERROR ApplicationMaster: Uncaught exception:
java.lang.IllegalStateException: SparkContext is null but app is still running!
    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:415)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:254)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:766)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)
    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
18/03/29 20:07:52 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.IllegalStateException: SparkContext is null but app is still running!)
18/03/29 20:07:52 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: java.lang.IllegalStateException: SparkContext is null but app is still running!)
18/03/29 20:07:52 INFO ApplicationMaster: Deleting staging directory hdfs://<ip-address>.ec2.internal:8020/user/hadoop/.sparkStaging/application_1522348295743_0010
18/03/29 20:07:52 INFO ShutdownHookManager: Shutdown hook called
End of LogType:stderr

我提交的这份工作如下:

spark-submit --deploy-mode cluster --master yarn --num-executors 40 --executor-cores 16 --executor-memory 100g --driver-cores 8 --driver-memory 100g --class <package.class_name> --jars <s3://s3_path/some_lib.jar> <s3://s3_path/class.jar>

我的班级是这样的:

class MyClass {

  def main(args: Array[String]): Unit = {
    val c = new MyClass()
    c.process()
  }

  def process(): Unit = {
    val sparkConf = new SparkConf().setAppName("my-test")
    val sparkSession: SparkSession = SparkSession.builder().config(sparkConf).getOrCreate()
    import sparkSession.implicits._
    ....
  }

  ...
}

1 回复 | 直到 7 年前

Jacek Laskowski 7 年前

改变 class MyClass 到 object MyClass 你就完了。

当我们这样做的时候,我也会改变 类别MyClass 到 object MyClass extends App 并删除 def main(args: Array[String]): Unit (由给出 extends App )。

我报告了Spark 2.3.0的改进- [SPARK-23830] Spark on YARN in cluster deploy mode fail with NullPointerException when a Spark application is a Scala class not object -让它很好地向最终用户报告。

深入了解Spark on纱线的工作原理,以下信息是 ApplicationMaster of a Spark application starts the driver (您使用 --deploy-mode cluster --master yarn 具有 spark-submit )。

ApplicationMaster:在单独的线程中启动用户应用程序

在信息消息之后,您应该会看到另一条消息:

ApplicationMaster:正在等待spark上下文初始化。。。

这是 driver initialization when the ApplicationMaster runs 。

例外情况的原因 Exception in thread "Driver" java.lang.NullPointerException 是由于 following code :

val mainMethod = userClassLoader.loadClass(args.userClass)
  .getMethod("main", classOf[Array[String]])

我的理解是 mainMethod 是 null 此时 following line (其中 Main方法 是 无效的 )“触发器” NullPointerException :

mainMethod.invoke(null, userArgs.toArray)

线程确实被称为 Driver (如中所示 线程“驱动程序”java中出现异常。lang.NullPointerException )如中所述 this line :

userThread.setContextClassLoader(userClassLoader)
userThread.setName("Driver")
userThread.start()

因为我使用Spark 2.3.0来引用这些行,而您使用emr-5.4.0和Spark 2.1.0来引用这些行,所以行号有所不同。