Posts

Data, Devops, Pipelines, Testing, Fun in the Agile SQL Club

2024

Spark Connect Dotnet November 2024 Where are we?

Nov 16

Spark Connect Dotnet Variant Data Type

Oct 12

Delta Lake from .NET in a Spark Connect gRPC world

Feb 17

Implementing functions and more fun in Spark Connect using gRPC and .NET

Feb 10

Apache Spark from PHP - it is not just a .NET thing

Feb 10

Moving towards the DataFrame API using the Spark Connect gRPC API in .NET

Feb 5

Exploring the Spark Connect gRPC API more

Feb 1

Exploring the Spark Connect gRPC API

Jan 26

Using Spark Connect from .NET to run Spark jobs on Databricks

Jan 23

Using Spark Connect from .NET

Jan 17

Spark Connect Dotnet First Major Milestone

Jan 17

2021

ADF: Error trying to debug pipeline: BadRequest

Mar 12

ADF: Querying JSON documents

Feb 26

ADF, XML objects and XPath in the expression language

Feb 12

Synapse Analytics and .NET for Apache Spark Example 4 - JOINS

Jan 25

Synapse Analytics and .NET for Apache Spark Example 3 - CTE()

Jan 19

Synapse Analytics and .NET for Apache Spark Example 2 - ROW_NUMBER()

Jan 18

Synapse Analytics and .NET for Apache Spark Example 1 - Group By

Jan 18

How to get data in a DataFrame via .NET for Apache Spark

Jan 17

Git in 10 commands to do 99% of gitting and keeping out of trouble

Jan 12

2020

TF-IDF in .NET for Apache Spark Using Spark ML v2

Dec 19

Blog .NET for Apache Spark UDFs Missing Shared State

Oct 29

TF-IDF in .NET for Apache Spark Using Spark ML

Jul 7

Approaches to running Databricks ETL code from Azure ADF

Jun 3

The four tenets of ETL testing

Mar 6

Passing status messages and results back from Databricks to ADF

Feb 5

Spark Delta Lake, Updates, Deletes and Time Travel

Jan 20

2019

Validating upstream data quality in ETL processes, SQL edition

Oct 28

SQLCover 0.5 - Fixes, smaller features and an exciting surprise

Oct 16

How to test ETL Processes in production

Oct 2

How to connect spark to ms sql server without '[Error] [JvmBridge] java.sql.SQLException: No suitable driver'?

Oct 1

How do test the upstream data is good in an etl process (etl testing part 3)?

Sep 30

Visual Studio tSQLt Test Adapter

Sep 2

Help! My Apache Spark configuration on windows isn't working (I think)

Aug 29

How do you debug your spark-dotnet app in visual studio?

Aug 20

How do we prove our ETL processes are correct? How do we make sure upstream changes don't break our processes and break our beautiful data?

Aug 13

spark-dotnet how to manually break a file into rows and columns

Aug 6

java.lang.ClassNotFoundException: org.apache.spark.deploy.DotnetRunner with 0.4.0 of spark-dotnet

Aug 2

How do we test ETL pipelines? Part one unit tests

Jul 25

spark-dotnet examples - reading and writing csv files

Jul 15

spark-dotnet how does user .net code run spark code in a java vm

Jul 14

Spark and dotnet in a single docker container

Jul 11

Where do you do your database development? Hopefully not production

Jul 5

As a DBA, how do I offload some of my work?

Jun 13

What is the key to automated database deployments?

Jun 11

What steps are there to move to safe automated database deployments?

Jun 10

How do you choose which columns to include in your SQL Server clustered index?

Jun 3

Why is my pivot query not returning any data? How to debug a PIVOT query

May 21

How to delete a lot of data from a SQL Server database

May 15

How to load test a SQL database

May 8

Should I pluralize table names, is it Person, Persons, People or People?

May 2

DacFx how to get the data type from a column a discussion of properties, relationships and the TSqlModel

Apr 17

How can we merge multiple dacpacs into one

Mar 5

SSDT: Build in VSCode

Jan 29

SSDT: /p:AllowIncompatiblePlatform, what platforms are compatible?

Jan 25

2018

TDD for SQL Server using tSQLt course certificates now available

Jul 18

mentor or mentee be, be both, be neither, be happy

Jul 6

Running MySQL on a VSTS build agent

May 8

Database testing and ETL processes

May 1

Interesting Azure ARM "Make it So" behaviour

Apr 5

Why didnt my Azure ARM deploy work?

Apr 4

VSTS extensions, mixing VSS.require and import ... require

Mar 5

gdpr - panic part 4

Feb 8

tSQLt Course

Feb 6

Re-Imagining an SSIS IDE

Feb 4

gdpr - panic part 3

Feb 1

gdpr - panic part 2

Feb 1

gdpr - panic part 1

Feb 1

2017

Azure ARM template function internals

Dec 18

xSQLServer is dead long live SqlServerDsc

Dec 15

What The Tool? Multiple choice quiz to help you choose a tool for SQL Server

Dec 14

My Azure ARM template development workflow

Dec 13

ARMED - Where is the source?

Dec 9

ARMED - ARM resource dependency graph inside VSCode

Dec 8

ARMED - Azure ARM VS Code extension

Dec 6

Azure Virtual Machine + Premium Disk Throughput Calculator

Dec 2

Fixing SQL AG Routing "Cannot find host" using packet analysis (and a spell checker)

Nov 30

"the input device is not a TTY" error running docker on VSTS build agent

Nov 22

How do I use VSTS variables in YAML CI Build definitions?

Nov 21

You have your database in source control now what?

Nov 16

Running any AzureRM powershell command tells you to "Run Login-AzureRmAccount to login." even though you are logged in!

Nov 15

Watching for powershell changes and running Invoke-Pester

Nov 14

SSDT How To Fix Error SQL17502

Nov 6

A virtualized CPU forced me to eat my lunch early, every day, for weeks

Oct 21

Where do you install tSQLt?

Oct 19

tSQLt Test Adapter for Visual Studio 2017

Oct 3

Learn how to unit test SQL Server T-SQL code

Sep 5

SSIS ForEach Enumerator File Order

Sep 4

SSDT-Dev-Pack

Aug 17

SQLCover

Aug 17

K-SSIS-ed

Aug 17

TSQL Tuesday - Databases and DevOps

Jun 13

SSDT: Unable to connect to master or target server.

Jun 12

SSDT DevPack + Visual Studio 2017

May 8

SSDT Dev in Visual Studio Code

Apr 27

Updating TSqlModels (DacFx)

Apr 26

My SQL Server Development Team Maturity Levels

Apr 24

SQL Server Continuous Deployment "In a Box"

Mar 7

ScriptDom parsing and NoViableAltExceptions

Mar 2

SqlPackage Deploy Performance - IgnoreXX are not your friend!

Mar 2

SSDT Deploy / Publish Performance

Mar 1

SSDT and Friends - .net meetup video

Feb 10

Devops without management buy in?

Feb 10

2016

SQLCover Fixes and Download location

Sep 30

Refactoring in SQL Server Data Tools - SSDT

Sep 27

Database Deployments in Uncontrolled Environments

Sep 20

Controlling a dacpac deployment

Aug 21

tSQLt Visual Studio Test Adapter

Aug 17

DacFxed - Powershell Nugetized DacFx wrapper

Aug 2

SQLCover v 0.2 - Bug fixes and Azure V12 Support

May 5

Where do you put your business logic? In stored procedures?

May 5

How do you audit automatically generated sql deployment scripts?

May 3

What is code coverage for?

Apr 11

SQLCover Code Coverage for SQL Server T-SQL

Apr 8

Post Deploy Scripts In Composite Dacpacs not deploying

Mar 3

Where did january go?

Feb 16

T-SQL Code Coverage in SSDT using the SSDT Dev Pack

Jan 13

What is SSDT? Part 3 - an API for me, an API for you, an API for everyone!

Jan 7

What is SSDT? Part 2 - More Reasons to Bother!

Jan 6

What is SSDT all about aka Why should I bother?

Jan 5

2015

SSDT Migrating Data Without Losing It

Dec 30

T-SQL Clippy

Dec 10

Automatically re-write non-sargable isnulls into sargable code in SSDT

Dec 10

Find duplicate indexes in SSDT

Nov 27

Convert select statements in scripts into Inline-TVFs automatically

Nov 25

ScriptDomVisualizer - Now Displays TSqlParserTokens

Nov 25

Changing case of T-SQL Keywords

Nov 24

Quickly deploying procedure/functions from SSDT (Bypass Publish)

Nov 24

Using the ScriptDom without losing comments

Nov 24

SSDT DevPack - Highlight expensive queries

Nov 22

Create stub tSQLt tests in SSDT from stored procedures

Nov 17

Automatically name primary key constraints in SSDT

Nov 16

New Version of MergeUi

Nov 12

How to get started with the ScriptDom

Nov 7

ScriptDom Visualizer V2

Nov 6

Blocking SSDT Deployments When You Dont Like Something

Nov 6

SQL Server Edition aware SSDT deployment scripts

Nov 5

Deploy SSDT INSERTS in Batches

Nov 4

ScriptDom Visualizer

Nov 3

Shared SSDT Deployment Contributor Repository and Free Help

Nov 2

Looking at SSDT upgrade scripts

Nov 1

Dir2Dac and github2dacpac - create dacpacs from script folders or the internet

Oct 16

New SSDT Projects default to the latest version of SQL Server

Oct 14

Video of sqlrelay talk - build a ci pipeline in 55 minutes

Oct 9

How do you choose a source control system for your SQL Server database

Oct 9

SQLRelay Reading - Build a CI Pipeline in 55 Minutes Talk

Oct 9

Deployment Contributor Logger

Sep 30

Deployment Contributor KeepTableColumns Filter

Sep 23

Inside an SSDT Deployment Contributor

Sep 23

Setting up a new CI server to build SSDT projects

Sep 23

ScriptDom versioning issues

Sep 15

Why is my tSQLt test not running?

Sep 10

Slides for my pass virtualisation virtual chapter talk on how sql fits into a containered world

Sep 10

We can do professional SQL Server deployments using SSDT

Sep 2

Slides and Demos for SQLSupper August meeting

Aug 26

I dont use the Visual Studio SSDT Publish

Aug 19

Ignore a warning on a dacpac reference at your peril

Aug 17

Shared databases or private databases?

Aug 17

AdventureworksCI Step 5 Adding tSQLt to the solution

Jul 22

AdventureworksCI Step 4 Pushing to GitHub

Jul 22

Keep your environments in-sync and your upgrade scripts fresh

Jul 21

Pre-Compare & Pre-Deployment Scripts to SSDT

Jul 20

WaitForSql (to start)

Jul 17

Layer a database into a SQL Server container

Jul 8

AdventureworksCI Step 3 Handling data with SSDT

Jul 7

Run SQL Server In A Container Today July 2015

Jul 7

Download SSMS, SSDT or SSDT-BI

Jun 30

Why does sqlpackage.exe not compare after the pre deployment script has run?

Jun 24

AdventureWorksCI Step 2 From MDF to Dot Sql Files

Jun 22

AdventureWorksCI Step 1 planning

Jun 12

A reference implementation of a continuous integration pipeline for legacy SQL Server databases

Jun 10

Why is SSDT so slow?

Jun 2

MergeUi is on github.com

May 21

MergUi 0.1.9

May 20

Putting the ORM debate to bed

May 20

Approaches to deploying Sql Server database code

Apr 27

BIML is better even for simple packages

Apr 27

Sql Saturday Talk

Apr 25

What Permissions do I need to generate a deploy script with SSDT?

Apr 21

HOWTO-Get-T-SQL-Into-SSDT

Apr 8

Generating tSQLt tests from a dacpac

Apr 2

SSDT Myths 1 - SSDT only works with TFS

Apr 2

Free Professional Advice on Setting up Continuous Integration for Sql Server Databases

Mar 19

Unit Test Sql Server Code

Mar 3

Introducing MergeUI

Mar 1

Source control isnt just a backup

Feb 19

SSDT all the way

Feb 16

Steps to become more agile with Sql development

Feb 12

Choosing a Development IDE for Sql Development

Feb 12

Enable SSDT and DacFx logging

Feb 5

Why does SqlPackage keep dropping and recreating constraints?

Jan 16

HOWTO: Filter Dacpac Deployments

Jan 14

2014

Enforcing T-SQL quality with SSDT Analysis Extensions

Dec 30

DacFX Links

Dec 7

Make Visual Studio Setup Slightly Faster

Dec 7

Running tSQLt tests from C/VB.Net/Java/Whatever

Nov 20

For better Sql code use fake tables

Nov 18

Unit tests help document your code!

Nov 2

What collation variables take on in T-SQL

Oct 22

DBAs - Step away from production

Oct 21

DevDev

Oct 20

DevDev

Oct 20

Blog Redirected

Oct 20