Session room

The Agenda

Agenda Home
SQLBits 2024 runs from Tuesday 19th – Saturday 23rd March.
Big Data & Data Engineering

Mastering Delta Lakes in Azure

Description

Once upon a time we had the Data Warehouse, life was good but it had its limitations, particularly around loading/storing complex data types. As data grew larger and more varied, the warehouse became too rigid and opinionated.

So we dove headfirst into Data Lakes to store our data. Again, things were good, but missed some of the good times that the Data Warehouse had given us. The lake had become too flexible, we needed stability in our life. In particular, we needed A.C.I.D (Atomicity, Consistency, Isolation, and Durability) Transactions.

Delta Lake, hosted by the Linux Foundation, is an open-source file layout protocol for giving us back those good times, whilst retaining all of the flexibility of the lake. Delta has gone from strength to strength, and in 2022 Databricks finally open-sourced the entire code-base, including lots of advanced features that were previously Databricks-only. This workshop takes you from the absolute basics of using Delta within a Lake, through to some of those advancing engineering features, letting you really master your Delta Lake.

In this workshop we will go from Zero to Hero with Delta, including:
• Handling Schema Drift
• Applying Constraints and Database Designs
• Time-Travel & Management
• Optimize & Performance Tuning
• Streaming

We will also show you how to work with Delta inside and outside of its original home of Databricks.

This training has been designed from our hands-on experience working with Delta and implementing Delta solutions to our clients across the globe. The course is aimed at beginners, and you will leave this course with all the skills you needed to get started on your Delta journey.

The course will be delivered by Microsoft MVP and Databricks Champion working together to bring you the best.

Learning Objectives

With all the skills needed to get started on your Delta and workshops material & labs for future reference.

Previous Experience

Both I an co-speaker have presented at SQLBits before including a precon, the co-speaker is a MVP and I am still a beginner but have presented at 3 conferences (inc SQLBits, Data Relay and Dativerse) and local meetups.

Tech Covered

Azure, Data Lake, Developing, Optimising, Data Bricks, Big Data & Data Engineering, Data Loading Patterns & Techniques