代码之家 › 专栏 › 技术社区 › Markus

Spark异常:java.lang.nosuchMethodError:org.apache.http.conn.ssl.sslConnectionSocketFactory

apache-httpclient-4.x apache-spark-sql sbt apache-spark scala

Markus · 技术社区 · 6 年前

当我在本地代码中运行代码时,它工作正常。但是,当我在集群中运行它时,似乎我的JAR文件中缺少了一些依赖项:

18/05/23 10:46:02 ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
        at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.<init>(SdkTLSSocketFactory.java:56)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)

这是SBT文件:

scalaVersion := "2.11.8"
val sparkVersion = "2.2.0"
val mahoutVersion = "0.13.1"
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
  "org.sedis" %% "sedis" % "1.2.2",
  "org.scalactic" %% "scalactic" % "3.0.0",
  "org.scalatest" %% "scalatest" % "3.0.0" % "test",
  "com.github.nscala-time" %% "nscala-time" % "2.14.0",
  "com.github.scopt" %% "scopt" % "3.3.0"
)
unmanagedSourceDirectories in Compile += baseDirectory.value / "lib"
resolvers += "typesafe repo" at " http://repo.typesafe.com/typesafe/releases/"
resolvers += Resolver.mavenLocal
assemblyMergeStrategy in assembly := {
  case "plugin.properties" => MergeStrategy.discard
  //case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case PathList("org", "joda", "time", "base", "BaseDateTime.class") => MergeStrategy.first
  case PathList(ps @ _*) if ps.last endsWith "package-info.class" => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "WrapDynaClass.class") => MergeStrategy.first
  case PathList("javax", "inject", xs @ _*) => MergeStrategy.first
  case PathList("org", "aopalliance", "aop", xs @ _*) => MergeStrategy.first
  case PathList("org", "aopalliance", "intercept", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "spark", "unused", "UnusedStubClass.class") => MergeStrategy.first
  case PathList("org", "apache", "commons", "collections", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "locale", "converters", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "locale", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "converters", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", xs @ _*) => MergeStrategy.first
  case PathList("assets", "org", "apache", "commons", "math3", "exception", "util", "LocalizedFormats_fr.properties") => MergeStrategy.first
  case PathList("com", "google", "common", "annotations", "Beta.class") => MergeStrategy.first
  case PathList("com", "google", "common", "base", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "cache", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "collect", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "eventbus", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "hash", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "io", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "math", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "net", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "primitives", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "reflect", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "util", "concurrent", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "differentiation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "function", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "integration", "gauss", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "interpolation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "polynomials", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "solvers", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "complex", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "dfp", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "distribution", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "exception", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "exception", "util", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "filter", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "fitting", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "fraction", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "genetics", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "oned", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "threed", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "twod", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "partitioning", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ml", "clustering", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ml", "distance", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "events", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "nonstiff", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "sampling", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", "gradient", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", "noderiv", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "vector", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "univariate", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "direct", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "fitting", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "general", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "univariate", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "primes", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "random", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "special", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "clustering", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "correlation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", "moment", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", "summary", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "inference", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "ranking", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "regression", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "transform", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "util", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "logging", xs @ _*) => MergeStrategy.first
  case PathList("overview.html") => MergeStrategy.first
  case PathList("mime.types") => MergeStrategy.first
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

我的罐子里怎么了?在intellij思想中,我选择了类检查选项并粘贴 SSLConnectionSocketFactory . 找到了这个类并引用了httpclient-4.3.6.jar

1 回复 | 直到 6 年前

Pietrotull 6 年前

Spark有很多图书馆,你可能和其中一个图书馆有冲突。

我想我对io.netty.netty-all包也有类似的问题。我们最终将服务器上的包升级到了一个稍新的次要版本,但这是因为我们正在为部署在节点上的spark构建集成。

您可以尝试使用这些参数部署Spark应用程序

--conf 'spark.driver.extraJavaOptions=-verbose:class'
--conf 'spark.executor.extraJavaOptions=-verbose:class'

这些可以让你看到类路径。我很肯定你会在那里看到有冲突的包裹。我会开始检查netty的版本,因为我很确定它取决于httpclient。