Azure Databricks is amazing: within a very short space of time you can get a Workspace deployed and start writing notebooks in a variety of languages, attach those notebooks to clusters and access large volumes of data at scale on clusters.

Inevitable though, you will want to share your code with other team members, deploy versions of the notebooks to different workspaces and configure jobs and secrets. With a simple walk-through of the constituent parts of what makes a workspace, we cover how to get your workspace into a repo, how to develop "locally in the cloud", and some of the challenges of the out-of-the-box source control features that Azure Databricks provides.

We will also touch on the challenge of monitoring and automating tests.
Note this talk is less about development best practices in Azure Databricks from a code perspective: we will be focusing on software development lifecycle and how it applies to Azure Databricks.

Presented by Simon Sabin at SQLBits 2020