远程调试

master,worker,executor远程调试

Spark 设置环境变量可以远程调试

• org.apache.spark.launcher.Main

  List<String> cmd = builder.buildCommand(env);

@Override
  public List<String> buildCommand(Map<String, String> env) throws IOException {
    List<String> javaOptsKeys = new ArrayList<String>();
    String memKey = null;
    String extraClassPath = null;

    // Master, Worker, and HistoryServer use SPARK_DAEMON_JAVA_OPTS (and specific opts) +
    // SPARK_DAEMON_MEMORY.
    if (className.equals("org.apache.spark.deploy.master.Master")) {
      javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
      javaOptsKeys.add("SPARK_MASTER_OPTS");
      memKey = "SPARK_DAEMON_MEMORY";
    } else if (className.equals("org.apache.spark.deploy.worker.Worker")) {
      javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
      javaOptsKeys.add("SPARK_WORKER_OPTS");
      memKey = "SPARK_DAEMON_MEMORY";
    } else if (className.equals("org.apache.spark.deploy.history.HistoryServer")) {
      javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
      javaOptsKeys.add("SPARK_HISTORY_OPTS");
      memKey = "SPARK_DAEMON_MEMORY";
    } else if (className.equals("org.apache.spark.executor.CoarseGrainedExecutorBackend")) {
      javaOptsKeys.add("SPARK_JAVA_OPTS");
      javaOptsKeys.add("SPARK_EXECUTOR_OPTS");
      memKey = "SPARK_EXECUTOR_MEMORY";
    } else if (className.equals("org.apache.spark.executor.MesosExecutorBackend")) {
      javaOptsKeys.add("SPARK_EXECUTOR_OPTS");
      memKey = "SPARK_EXECUTOR_MEMORY";
    } else if (className.equals("org.apache.spark.deploy.ExternalShuffleService") ||
        className.equals("org.apache.spark.deploy.mesos.MesosExternalShuffleService")) {
      javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
      javaOptsKeys.add("SPARK_SHUFFLE_OPTS");
      memKey = "SPARK_DAEMON_MEMORY";
    } else if (className.startsWith("org.apache.spark.tools.")) {
      String sparkHome = getSparkHome();
      File toolsDir = new File(join(File.separator, sparkHome, "tools", "target",
        "scala-" + getScalaVersion()));
      checkState(toolsDir.isDirectory(), "Cannot find tools build directory.");

      Pattern re = Pattern.compile("spark-tools_.*\\.jar");
      for (File f : toolsDir.listFiles()) {
        if (re.matcher(f.getName()).matches()) {
          extraClassPath = f.getAbsolutePath();
          break;
        }
      }

      checkState(extraClassPath != null,
        "Failed to find Spark Tools Jar in %s.\n" +
        "You need to run \"build/sbt tools/package\" before running %s.",
        toolsDir.getAbsolutePath(), className);

      javaOptsKeys.add("SPARK_JAVA_OPTS");
    } else {
      javaOptsKeys.add("SPARK_JAVA_OPTS");
      memKey = "SPARK_DRIVER_MEMORY";
    }

    List<String> cmd = buildJavaCommand(extraClassPath);
    for (String key : javaOptsKeys) {
      addOptionString(cmd, System.getenv(key));
    }

    String mem = firstNonEmpty(memKey != null ? System.getenv(memKey) : null, DEFAULT_MEM);
    cmd.add("-Xms" + mem);
    cmd.add("-Xmx" + mem);
    addPermGenSizeOpt(cmd);
    cmd.add(className);
    cmd.addAll(classArgs);
    return cmd;
  }

IDEA Remote Applicate

227KB

屏幕快照 2018-12-03 下午3.39.11.png

image

Open

master远程调试

master spark-env.sh文件配置

export SPARK_MASTER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10000"

启动master

start-master.sh

idea开发工具配置

• IDEA中建立一个remote application

• Master主机IP 10000

• 在本地的代码打断点，debug按钮开始调试

worker远程调试

worker 节点环境变量配置

• 临时用，还是配置环境变量好，如果更改配置文件调试完成后，还需要改回去

export SPARK_WORKER_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10001"

启动 worker

start-slave.sh spark://MasterIp:7077

idea开发工具配置

• IDEA中建立一个remote application

• Master主机IP 10001

• 在本地的代码打断点，debug按钮开始调试

executor远程调试

在Driver端的SparkConf配置中配置

spark.executor.extraJavaOptions=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002

• Scala程序中进行配置

 def sparkContext(): SparkContext = {
    val conf = new SparkConf().setAppName(appName).setMaster(master)
    conf.set("spark.eventLog.enabled","true")
   // conf.set("spark.ui.port","10002")
    conf.set("spark.history.fs.logDirectory","hdfs://standalone.com:9000/spark/log/historyEventLog")
    conf.set("spark.eventLog.dir","hdfs://standalone.com:9000/spark/log/eventLog")
    //executor debug,是在提交作的地方读取
    conf.set("spark.executor.extraJavaOptions","-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002")
    conf.setJars(Array("/opt/n_001_workspaces/bigdata/spark-scala-maven/target/spark-scala-maven-1.0-SNAPSHOT.jar"))

    val sc = new SparkContext(conf)

    //设置日志级别
    //sc.setLogLevel("ERROR")
    sc
  }

作业远程调试

spark-submit

spark-submit \
  --class com.opensource.bigdata.spark.standalone.RunTextFileMkString2 \
  --master spark://standalone:7077 \
  --executor-memory 1G \
  --total-executor-cores 100 \
  --driver-java-options "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=10002" \
  /root/temp/spark-scala-maven-1.0-SNAPSHOT.jar \

idea开发工具配置

• IDEA中建立一个remote application

• Master主机IP 10002

• 在本地的代码打断点，debug按钮开始调试

• 在main方法类中加断点,进行调试