Synapse Analytics and .NET for Apache Spark Example 2 - ROW_NUMBER()
The next example is how to do a ROW_NUMBER(), my favourite window function.
T-SQL
SELECT
*, ROW_NUMBER() OVER(ORDER BY dateTime) as RowNumber
FROM
chicago.safety_data
Spark SQL
SELECT
*, ROW_NUMBER() OVER(ORDER BY dateTime) as RowNumber
FROM
chicago.safety_data
DataFrame API (C#)
var dataFrame = spark.Read().Table("chicago.safety_data");
var window = Microsoft.Spark.Sql.Expressions.Window.OrderBy("dateTime");
dataFrame = dataFrame.WithColumn("RowNumber", Functions.RowNumber().Over(window));
dataFrame.Show();
To see this in action, please feel free to deploy this repo to your Synapse Analytics repo: https://github.com/GoEddie/SynapseSparkExamples