There was a breaking change with version 0.4.0 that changed the name of the class that is used to load the dotnet driver in Apache Spark.

To fix the issue you need to use the new package name which adds an extra dotnet near the end, change:

spark-submit --class org.apache.spark.deploy.DotnetRunner

into:

spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner

What if I have this error but that doesn’t fix it?

When you run a spark app using spark-submit and you get a ClassNotFoundException for the driver then it boils down to either making a typo or something on your system blocking the jar from being loaded (anti-virus?).

The process goes like this:

• spark-submit reads the name of the jar file with your class in and opens the jar file
• spark-submit tries to execute the class that it was passed via the “–class” parameter

To track down the issue on my machine I used procmon to filter to the name of the jar and I could see that the jar was being opened by java.exe. I then used 7-zip to open the driver jar and check that the class did exist which is when I saw the package name was different.

Once I knew the package name was different I was able to use the power of google and find the release note for 0.4.0 - maybe i’ll take a closer look at them next time :)