If you write code to SQL Server then you might be interested in this: I have written a tSQLt tdd training course which has helped over 500 people learn both tSQLt and how to apply TDD practices to their SQL Server T-SQL development, you can join the course at https://courses.agilesql.club.

java.lang.ClassNotFoundException: org.apache.spark.deploy.DotnetRunner with 0.4.0 of spark-dotnet

There was a breaking change with version 0.4.0 that changed the name of the class that is used to load the dotnet driver in Apache Spark.

To fix the issue you need to use the new package name which adds an extra dotnet near the end, change:

spark-submit --class org.apache.spark.deploy.DotnetRunner

into:

spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner

What if I have this error but that doesn’t fix it?

When you run a spark app using spark-submit and you get a ClassNotFoundException for the driver then it boils down to either making a typo or something on your system blocking the jar from being loaded (anti-virus?).

The process goes like this:

  • spark-submit reads the name of the jar file with your class in and opens the jar file
  • spark-submit tries to execute the class that it was passed via the “–class” parameter

To track down the issue on my machine I used procmon to filter to the name of the jar and I could see that the jar was being opened by java.exe. I then used 7-zip to open the driver jar and check that the class did exist which is when I saw the package name was different.

Once I knew the package name was different I was able to use the power of google and find the release note for 0.4.0 - maybe i’ll take a closer look at them next time :)

Subscribe

* indicates required

Please select all the ways you would like to hear from Agile Sql Club:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.