Migrating the Mammoth

This session is grounded in the real-world experience of a global technology company undertaking one of the largest data-platform migrations in industry—modernizing its fragmented data estate and moving to Delta Lake at extraordinary scale. Managing more than 60 PB of data spread across four different formats, 750+ ingestion pipelines, and over 5,000 ETL jobs, the organization needed a migration strategy engineered for both massive volume and operational complexity.

We will explore how the team executed this transformation using a structured “divide and conquer” model, separating the work into targeted migration workstreams and aligning them to OKRs. As tooling sophistication grew, migration velocity increased, supported by an agile delivery model and a dedicated “Migration Machine” designed to validate data, orchestrate dependency-heavy workloads, and provide complete visibility through custom dashboards.

Attendees will gain insight into the real challenges encountered at scale, including migrating 50+ TB tables, dealing with multiple source formats, orchestrating batch backfills, validating large datasets, and designing alternate paths for tables too slow or complex for standard migration flows.

We will conclude with the tangible outcomes: up to 1600× improvement on highly selective queries for petabyte-scale tables, significant gains in table-read performance, and major acceleration of BI workloads—results internal teams described as truly “game changing”.

This session offers a deeply practical look at what it takes to migrate a “mammoth” data estate to a modern lakehouse architecture, providing actionable lessons for organizations planning or scaling their own modernization efforts.

1. Understand how to design and execute a large-scale, multi-format data migration—including strategies for converting RC, ORC, Parquet, and Avro datasets into Delta Lake while maintaining data integrity, lineage, and operational continuity. 2. Learn the architectural patterns, automation frameworks, and tooling required to orchestrate thousands of ingestion and transformation pipelines during a multi-petabyte modernization effort, with a focus on scalability, validation, and reliability. 3. Identify common edge cases and technical challenges encountered during massive enterprise migrations—such as handling 50+ TB tables, mixed-format estates, legacy dependencies, and long-running workloads—and understand the practical solutions that enabled successful outcomes.