Practical lessons in optimizing Data Engineering with Spark - Part 1

Based on Luke's experience in working with some of the largest Fabric deployment, this session walks practical tips for optimizing data engineering focused Fabric deployments. This session is very demo-centric to provide real world examples of the tradeoffs that occur and how simple changes can have a large impact - especially as a deployment scales.

In this Part 1 we will focus on the Delta as the primary table format within Fabric including:
- the impact of features like change data feed, delete vectors and more
- enhancements in Delta 4.0, coming as part of the Fabric 2.0 runtime
- how metadata tables and logging tables in Delta can impact performance - and ways to solve these concerns.

1 - understanding of different delta tables features and what they do. 2 - understanding of when different properties might make sense (and not) and importantly how to test these options. 3 - detailed knowledge on when delta tables are a poor fit and shouldn't be used.