From “Who Wrote This ETL?” to Databricks, Claude Saves the Day via Microsoft Foundry
Regular 50 minute session for SQLBits 2026Saturday - 25 Apr 2026 - 09:00 - 09:50 Room 3ATL; DR
Anthropic’s Claude is now available in Microsoft Foundry—opening new possibilities for enterprise data modernization. This session shares real-world experience using Claude to build intelligent agents that migrate legacy open-source Spark ETL pipelines to Databricks. We discuss how an approach originally developed using Claude via AWS can now be applied within Microsoft Foundry, along with practical lessons learned from production implementations.
Session Details
We now have access to Anthropic’s Claude models directly within Microsoft Foundry—an exciting milestone that brings powerful, enterprise-ready language models into real-world data platform modernization workflows. This session shares hands-on experience using Claude to build intelligent agents that accelerate the migration of legacy ETL pipelines written in open-source Apache Spark to Databricks, and how this approach can now be applied within Microsoft Foundry, having originally accessed Claude via AWS.
We demonstrate an agent-based approach in which Claude analyzes existing Spark ETL workloads, infers transformation logic, identifies technical debt and anti-patterns, and generates Databricks-optimized implementations aligned with Delta Lake best practices. These agents operate across multiple stages of the migration lifecycle, including pipeline discovery, semantic understanding of business logic, automated refactoring, and validation against existing outputs.
The presentation focuses on practical lessons learned from production implementations: effective agent and prompt design, integration patterns within Microsoft Foundry, managing enterprise requirements such as security and cost controls, and techniques for reducing hallucinations when transforming complex ETL logic. Attendees will leave with concrete insights into operationalizing Claude to modernize data platforms at scale on Azure and Databricks.
We demonstrate an agent-based approach in which Claude analyzes existing Spark ETL workloads, infers transformation logic, identifies technical debt and anti-patterns, and generates Databricks-optimized implementations aligned with Delta Lake best practices. These agents operate across multiple stages of the migration lifecycle, including pipeline discovery, semantic understanding of business logic, automated refactoring, and validation against existing outputs.
The presentation focuses on practical lessons learned from production implementations: effective agent and prompt design, integration patterns within Microsoft Foundry, managing enterprise requirements such as security and cost controls, and techniques for reducing hallucinations when transforming complex ETL logic. Attendees will leave with concrete insights into operationalizing Claude to modernize data platforms at scale on Azure and Databricks.
3 things you'll get out of this session
Understand how Anthropic’s Claude can be accessed and operationalized within Microsoft Foundry
Learn how intelligent agents can be designed to migrate legacy Apache Spark ETL pipelines to Databricks
Identify effective agent and prompt patterns for analyzing, refactoring, and validating complex ETL logic
Apply lessons learned from real-world production use, including managing cost, security, and reliability
Evaluate how existing Claude-based workflows (originally built on AWS) can be adapted to Microsoft Foundry
Learn how intelligent agents can be designed to migrate legacy Apache Spark ETL pipelines to Databricks
Identify effective agent and prompt patterns for analyzing, refactoring, and validating complex ETL logic
Apply lessons learned from real-world production use, including managing cost, security, and reliability
Evaluate how existing Claude-based workflows (originally built on AWS) can be adapted to Microsoft Foundry
Speakers
Anna-Maria Wykes's other proposed sessions for 2026
What we Learned Migrating a Financial Giant from Hudi to Delta (and Why Iceberg was in the Mix) - 2026
Anna-Maria Wykes's previous sessions
How to Run Code Clubs for Neurodiverse Children
Code Clubs offer an amazing opportunity to introduce our next generation to coding, with simple brightly colored drag-and-drop tooling to get them started, we are successfully inspiring many to join the tech industry.
In this session I want to talk you through my journey setting up a Code Club for neurodiverse children, what I found worked, and what doesn’t. I hope that from this session you will be inspired to follow the same path I have, using your amazing tech experience to empower some of the most vulnerable children, enabling them to become inspired not just by coding, but the tech industry itself.
Introduction to the wonders of Azure DevOps
Azure DevOps is the leading deployment tool for build and release solutions end to end. It helps you plan your Agile project, manages Git code, and deploys solutions using Continuous Integration (CI) and Continuous Deployment (CD) pipelines.
In this session we will cover some of the core components of Azure DevOps and show you how to implement a secure deployment pipeline, using unit tests and gating with your CI builds and CD releases.
Automate the deployment of Databricks components using Terraform
Introduction into Terraform, Databricks provider and steps required to build an automated solution to provision Databricks workspace and resources into Azure cloud platform using Terraform.
So you want to be a Data Engineer?
An introduction to becoming a Data Engineer, Anna, Mikey and Ust will introduce the technology stack, tools and development skills needed for data engineering and show you how and where to go to learn them. We'll also show you how the skills you already have can kickstart your journey to becoming a Data Engineer.
Scala for Big Data the Big Picture
An opportunity to explore Scala, and why it is truly a “Data Engineers language”. Using Azure Functions, Data Factory, Azure Data Lake Gen2 and Databricks the basics will be explored, followed by real world examples