From “Who Wrote This ETL?” to Databricks, Claude Saves the Day via Microsoft Foundry

We now have access to Anthropic’s Claude models directly within Microsoft Foundry—an exciting milestone that brings powerful, enterprise-ready language models into real-world data platform modernization workflows. This session shares hands-on experience using Claude to build intelligent agents that accelerate the migration of legacy ETL pipelines written in open-source Apache Spark to Databricks, and how this approach can now be applied within Microsoft Foundry, having originally accessed Claude via AWS.

We demonstrate an agent-based approach in which Claude analyzes existing Spark ETL workloads, infers transformation logic, identifies technical debt and anti-patterns, and generates Databricks-optimized implementations aligned with Delta Lake best practices. These agents operate across multiple stages of the migration lifecycle, including pipeline discovery, semantic understanding of business logic, automated refactoring, and validation against existing outputs.

The presentation focuses on practical lessons learned from production implementations: effective agent and prompt design, integration patterns within Microsoft Foundry, managing enterprise requirements such as security and cost controls, and techniques for reducing hallucinations when transforming complex ETL logic. Attendees will leave with concrete insights into operationalizing Claude to modernize data platforms at scale on Azure and Databricks.

Understand how Anthropic’s Claude can be accessed and operationalized within Microsoft Foundry

Learn how intelligent agents can be designed to migrate legacy Apache Spark ETL pipelines to Databricks

Identify effective agent and prompt patterns for analyzing, refactoring, and validating complex ETL logic

Apply lessons learned from real-world production use, including managing cost, security, and reliability

Evaluate how existing Claude-based workflows (originally built on AWS) can be adapted to Microsoft Foundry