If you write code to SQL Server then you might be interested in this: I have written a tSQLt tdd training course which has helped over 500 people learn both tSQLt and how to apply TDD practices to their SQL Server T-SQL development, you can join the course at https://courses.agilesql.club.
There was a breaking change with version 0.4.0 that changed the name of the class that is used to load the dotnet driver in Apache Spark.
To fix the issue you need to use the new package name which adds an extra dotnet near the end, change:
spark-submit --class org.apache.spark.deploy.DotnetRunner
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner
What if I have this error but that doesn’t fix it?
When you run a spark app using spark-submit and you get a ClassNotFoundException for the driver then it boils down to either making a typo or something on your system blocking the jar from being loaded (anti-virus?).
The process goes like this:
- spark-submit reads the name of the jar file with your class in and opens the jar file
- spark-submit tries to execute the class that it was passed via the “–class” parameter
To track down the issue on my machine I used procmon to filter to the name of the jar and I could see that the jar was being opened by java.exe. I then used 7-zip to open the driver jar and check that the class did exist which is when I saw the package name was different.
Once I knew the package name was different I was able to use the power of google and find the release note for 0.4.0 - maybe i’ll take a closer look at them next time :)