From “Who Wrote This ETL?” to Databricks, Claude Saves the Day via Microsoft Foundry

Regular 50 minute session for SQLBits 2026Saturday - 25 Apr 2026 - 09:00 - 09:50 Room 3A

TL; DR

Anthropic’s Claude is now available in Microsoft Foundry—opening new possibilities for enterprise data modernization. This session shares real-world experience using Claude to build intelligent agents that migrate legacy open-source Spark ETL pipelines to Databricks. We discuss how an approach originally developed using Claude via AWS can now be applied within Microsoft Foundry, along with practical lessons learned from production implementations.

Session Details

We now have access to Anthropic’s Claude models directly within Microsoft Foundry—an exciting milestone that brings powerful, enterprise-ready language models into real-world data platform modernization workflows. This session shares hands-on experience using Claude to build intelligent agents that accelerate the migration of legacy ETL pipelines written in open-source Apache Spark to Databricks, and how this approach can now be applied within Microsoft Foundry, having originally accessed Claude via AWS.

We demonstrate an agent-based approach in which Claude analyzes existing Spark ETL workloads, infers transformation logic, identifies technical debt and anti-patterns, and generates Databricks-optimized implementations aligned with Delta Lake best practices. These agents operate across multiple stages of the migration lifecycle, including pipeline discovery, semantic understanding of business logic, automated refactoring, and validation against existing outputs.

The presentation focuses on practical lessons learned from production implementations: effective agent and prompt design, integration patterns within Microsoft Foundry, managing enterprise requirements such as security and cost controls, and techniques for reducing hallucinations when transforming complex ETL logic. Attendees will leave with concrete insights into operationalizing Claude to modernize data platforms at scale on Azure and Databricks.

3 things you'll get out of this session

Understand how Anthropic’s Claude can be accessed and operationalized within Microsoft Foundry

Learn how intelligent agents can be designed to migrate legacy Apache Spark ETL pipelines to Databricks

Identify effective agent and prompt patterns for analyzing, refactoring, and validating complex ETL logic

Apply lessons learned from real-world production use, including managing cost, security, and reliability

Evaluate how existing Claude-based workflows (originally built on AWS) can be adapted to Microsoft Foundry

Speakers

Anna-Maria Wykes

linkedin.com/in/anna-maria-wykes-31939454/recent-activity/posts

Anna-Maria Wykes's previous sessions

How to Run Code Clubs for Neurodiverse Children
Code Clubs offer an amazing opportunity to introduce our next generation to coding, with simple brightly colored drag-and-drop tooling to get them started, we are successfully inspiring many to join the tech industry. In this session I want to talk you through my journey setting up a Code Club for neurodiverse children, what I found worked, and what doesn’t. I hope that from this session you will be inspired to follow the same path I have, using your amazing tech experience to empower some of the most vulnerable children, enabling them to become inspired not just by coding, but the tech industry itself.
 
Introduction to the wonders of Azure DevOps
Azure DevOps is the leading deployment tool for build and release solutions end to end. It helps you plan your Agile project, manages Git code, and deploys solutions using Continuous Integration (CI) and Continuous Deployment (CD) pipelines. In this session we will cover some of the core components of Azure DevOps and show you how to implement a secure deployment pipeline, using unit tests and gating with your CI builds and CD releases.
 
Automate the deployment of Databricks components using Terraform
Introduction into Terraform, Databricks provider and steps required to build an automated solution to provision Databricks workspace and resources into Azure cloud platform using Terraform.
 
So you want to be a Data Engineer?
An introduction to becoming a Data Engineer, Anna, Mikey and Ust will introduce the technology stack, tools and development skills needed for data engineering and show you how and where to go to learn them. We'll also show you how the skills you already have can kickstart your journey to becoming a Data Engineer.
 
Scala for Big Data the Big Picture
An opportunity to explore Scala, and why it is truly a “Data Engineers language”. Using Azure Functions, Data Factory, Azure Data Lake Gen2 and Databricks the basics will be explored, followed by real world examples