代码之家  ›  专栏  ›  技术社区  ›  Pyd

使用spark的远程mysql数据库访问错误

  •  0
  • Pyd  · 技术社区  · 7 年前

    从远程mysql数据库访问表,该数据库在eclipse本地运行良好,但在尝试从 spark-submit .

    ERROR SparkContext: Error initializing SparkContext.java.lang.IllegalArgumentException: java.net.UnknownHostException: namenode
    at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
    

    spark-submit \
    --jars /home/hduser/.ivy2/cache/com.typesafe/config/bundles/config-1.3.1.jar \
    --class "retail.stack" --master local[2] \
    --driver-memory 2g --executor-memory 2g --executor-cores 2 \
    sample-spark-180417_2.11-1.0.jar \
    /home/hduser/spark-training/workspace/demos/output/extra_in_src \
    /home/hduser/spark-training/workspace/demos/output/extradest \
    /home/hduser/spark-training/workspace/demos/output/bestMatch
    

    代码如下:

      val spark = SparkSession.builder().master("local").appName("Database")
      .getOrCreate()
    
    val jdbcUsername = "root"
    val jdbcPassword = "root"
    val url = "jdbc:mysql://192.168.175.35:3306/sample?useSSL=false"
    
    val connectionProperties = new java.util.Properties()
    connectionProperties.put("user", jdbcUsername)
    connectionProperties.put("password", jdbcPassword)
    
     val querySource = "(select * from source1) as src"
    val queryDestination = "(select * from destination1) as dest"
    
    val source = spark.read.jdbc(url, querySource, connectionProperties).rdd.map(_.mkString(","))
    val destination = spark.read.jdbc(url, queryDestination, connectionProperties).rdd.map(_.mkString(","))
    

    非常感谢。

    1 回复  |  直到 7 年前
        1
  •  0
  •   Pyd    7 年前

    我通过重新配置连接字符串解决了这个问题。它工作得很好。!