Beyond Monitoring: Spark Optimization in Microsoft Fabric

In this session, I will walk through the open-source Spark monitoring solution available in the Fabric Toolbox
(https://github.com/microsoft/fabric-toolbox/tree/main/monitoring/fabric-spark-monitoring
).

The session will cover the motivation behind building this solution and the gaps it aims to address in existing Spark observability approaches within Microsoft Fabric. I will explain the overall architecture and design choices, showing how the solution collects, processes, and visualizes Spark runtime and performance metrics.

Beyond the technical implementation, the focus will be on how this OSS solution enables practical, real-world Spark monitoring. Attendees will learn how it can be used to:

Gain deeper visibility into Spark job and cluster behavior

Identify performance bottlenecks and inefficiencies

Support workload optimization and tuning

Enable informed right-sizing decisions for Spark clusters, improving both performance and cost efficiency

By the end of the session, participants will understand not only how the solution was built, but also how they can adopt, extend, and use it effectively to improve Spark performance and operational excellence in their own environments.

Beyond the technical implementation, the focus will be on how this OSS solution enables practical, real-world Spark monitoring. Attendees will learn how it can be used to: Gain deeper visibility into Spark job and cluster behavior Identify performance bottlenecks and inefficiencies Support workload optimization and tuning Enable informed right-sizing decisions for Spark clusters, improving both performance and cost efficiency