Spark Connect

References

https://spark.apache.org/docs/latest/spark-connect-overview.html

https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html

Reference Implementations

Rust

GOLang

Source gRPC proto files

https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect

Useful Videos

If this doesn’t get you excited then nothing will it is, of course, suitable for work, it is a talk about Spark Connect.

Spark Connect From .NET

https://the.agilesql.club/2024/01/using-spark-connect-from-.net/

https://the.agilesql.club/2024/01/using-spark-connect-from-.net-to-run-spark-jobs-on-databricks/

All Spark Connect Posts