Remotely debugging Spark submit Jobs in Java

This extends Remote debugging in Java with Java Debug Wire Protocol (JDWP) to debug Spark jobs written in Java. We need to debug both the “Driver” and the “Executor“.

Debugging the Spark Driver in Java

Step 1: Run the Spark submit job in the remote machine, which waits on port “7777” for the eclipse debugger to connect.

Step 2: Add break points to your code in the project “myapp”.

Step 3: Right mouse click on the “myapp” project within Eclipse and select “Debug As” -> “Debug Configurations” -> “Remote Java Application“. Right mousel click and create new:

and click “Apply” and then “Debug”. This will make the paused Spark submit job to continue running and the code will pause at the first break-point in your code.

Debugging the Spark Executor in Java

Step 1: Right mouse click on the “myapp” project within Eclipse and select “Debug As” -> “Debug Configurations” -> “Remote Java Application“. Right mousel click and create new:

Step 2: Run the Spark submit job in the remote machine as shown below.

The code will pause at the breakpoints within the executor codes.

Debugging both the Spark Driver & the Executor in Java

Step 1: Add the required break points to your “myapp” code in Eclipse.

Step 2: Run the configured executor debugger “myapp executor debug“.

Step 3: Run the Spark submit command as shown below with both Driver & Executor debugging turned on.

Step 4: Run the configured executor debugger “myapp driver debug“.

Debugging via spark2-shell in Scala


300+ Java & Big Data Interview FAQs

16+ Java Key Areas Interview Q&As

800+ Java Interview Q&As

300+ Java & Big Data Tutorials

Top