AWS EMR,使用terraform提交python pyspark脚本作为步骤

一种常见的方法是从S3复制脚本并使用 command-runner.jar 执行脚本。(我不知道这是否理想……)

  step = [
    {
      name              = "Copy script"
      action_on_failure = "CONTINUE"

      hadoop_jar_step {
        jar  = "command-runner.jar"
        args = ["aws", "s3", "cp", "s3://path/to/script.py", "/home/hadoop/"]
      }
    },
    {
      name              = "Run script"
      action_on_failure = "CONTINUE"

      hadoop_jar_step {
        jar  = "command-runner.jar"
        args = ["bash", "/home/hadoop/script.py"]
      }
    },
  ]

以下是一个使用示例 hadoop_jar_step .

  step = [
    {
      action_on_failure = "TERMINATE_CLUSTER"
      name              = "Setup Hadoop Debugging"

      hadoop_jar_step = [
        {
          jar = "command-runner.jar"
          args = [
            "state-pusher-script"
          ]
          main_class = ""
          properties = {}
        }
      ]
    }
  ]

包含以下解决方法 https://github.com/hashicorp/terraform-provider-aws/issues/20911

更改 args 到你的 spark-submit 命令。