Are You Working With Python?

Python for data work, done properly

Python is now the common language of data engineering, whether you are on Fabric, Databricks, Snowflake or SQL Server. Writing it well is a different skill from writing SQL: tests, packaging, performance and upgrades all matter. SQLBits is where T-SQL teams and Spark teams learn that craft from each other.

Practical Python and Spark skills you can put to work

PySpark performance and skew, Spark Connect, UDFs and Arrow, notebooks versus packaged code, pandas, Polars and DuckDB, testing, CI/CD and Python version upgrades. Plus how coding agents are changing the way this code gets written and reviewed.

Learn from people who ship it

Practitioners, MVPs and Microsoft and Databricks specialists who run Python and Spark in production will share what works at scale, and what they would do differently.

The challenges we’ll help you solve

If any of these sound familiar, you will find people at SQLBits who have already been there.

Spark skew, shuffles and small files still eat runtime and money.
Python UDFs are slow row by row, and Spark 4.2 changes the defaults.
Python 3.10 reached end of life on 1 October 2026, and pinned environments and old kernels must move.
The Python 3.10 kernel in Fabric Python notebooks reaches end of support in October 2026.
Moving to Fabric Runtime 2.0 can break libraries that worked on earlier runtimes.
Spark 3.5 gets security fixes only until November 2027, so staying put has a clock.
pandas 3.0 breaks chained assignment and object dtype checks, so old code needs a rewrite.
Notebooks versus packaged, tested, versioned code: moving logic into libraries is hard to get right.
Testing PySpark is slow, and Spark Connect removes Py4J and RDD access.
pandas, Polars, DuckDB or Spark? Fabric guidance advises DuckDB, Polars or PyArrow over pandas to avoid out-of-memory errors on large data.
AI-generated code looks right but isn’t: in the 2025 Stack Overflow survey, 66% cite ‘almost right, but not quite’.
Our T-SQL team needs to become Python engineers without breaking production.

Databricks ai_query went GA on 15 Jun 2026 and works from SQL or Python. Most task-specific AI functions are beta, and Fabric’s AI functions and data agents are in preview.

MLflow 3.14 (June 2026) added pytest-based GenAI regression tests, and 3.9 (January 2026) added LLM-judge monitoring.

Spark 4.1 added Arrow-native UDF and UDTF decorators, and Spark 4.2 makes Arrow-optimised Python UDFs the default.

A thin client (pip install pyspark-client) lets your application keep its own dependencies and be debugged from an IDE. Spark ML on Connect is GA for the Python client in Spark 4.1.

The Python Data Source API (Spark 4.0) lets teams read proprietary or AI-service sources from Python, with filter pushdown added in 4.1.

Declarative Pipelines expectations now apply to standalone materialised views on Databricks (June 2026), and VARIANT is GA and on by default in Spark 4.1 for semi-structured data.

Databricks documents authoring agents and deploying them on Databricks Apps. Managed agent memory on Lakebase is in beta (September 2026).

Eight places agents are already earning their keep in Python Engineering (data and Spark), and where SQLBits sessions will show you how.

Genie Code, Databricks’ coding agent (agent mode GA, March 2026), handles multi-step data engineering and data science tasks and supports MCP.

Genie Code in the Lakeflow Pipelines Editor plans, writes, runs and fixes a pipeline from one prompt, with approval gates.

The same agent can move dbt or Informatica projects into Lakeflow pipelines (beta).

Fix with Copilot in Microsoft Fabric gives error summaries and approval-based fixes (preview).

Copilot chat and in-cell Copilot work in Fabric notebooks (preview). A FabricNotebook custom agent for GitHub Copilot Chat in VS Code knows the built-in Spark session and lakehouse paths (preview).

MCP servers let agents work against live platforms. The Genie One MCP server on Databricks went GA on 25 Sep 2026, and Fabric has local and remote MCP servers.

GitHub Copilot’s cloud agent went GA on 25 Sep 2025, and its code review agent skills and MCP support went GA on 29 Jul 2026. Claude Code and other agents are also used for writing tests, fixing lint errors and updating dependencies.

CoCo (formerly Cortex Code) Desktop went GA on 21 Jul 2026, and CoCo in the Snowflake VS Code extension on 27 Aug 2026.

All of these speakers have appeared on the SQLBits stage – check our 2027 agenda when it’s announced for details of our 2027 Python sessions and speakers.

Data Platform MVP with over 20 years of experience across business, computing and data, specialising in Power BI, Python, R, SQL and advanced data analytics.

Microsoft Regional Director and Azure MVP who writes books and courses and contributes to Apache open-source projects.

Databricks MVP and Microsoft Data Platform MVP, creator of the Advancing Spark YouTube series.

A Python-in-Fabric specialist for data engineers.

AI Beyond the Hype; Practical Solutions for Real Businesses

AI can feel overhyped and inaccessible. This session cuts through the noise, showing how practical AI solutions can be built using Azure AI, Copilot Studio, and AutoML. Ideal for data and tech professionals wanting real, usable AI.

AI-Augmented Query Optimization: SQL Server 2025 Meets Fabric Self-Healing Data Platform

SQL Server 2025 introduces groundbreaking capabilities for performance optimization and observability, while Microsoft Fabric brings unified analytics and AI-driven insights. But how do we combine these innovations to build a truly resilient, self-healing data platform? In this session, we’ll explore AI-Augmented Query Optimization and Fabric Observability to tackle real-world challenges like unpredictable workloads, query regressions, and performance bottlenecks. You’ll learn: Who this is for: Data engineers, DBAs, and architects managing mission-critical SQL workloads in hybrid or cloud environments. What problem it solves: Eliminates manual troubleshooting by leveraging telemetry, ML notebooks, and AI-driven root-cause analysis. What you’ll take away: A practical blueprint for creating a Self-Healing SQL Fabric Dashboard that combines SQL telemetry with Fabric ML for automated anomaly detection and performance tuning. Expect live demos, POC architecture, and actionable strategies to future-proof your data platform with SQL Server 2025 and Microsoft Fabric.

Beyond Traditional DBA Practices - Transforming T-SQL Script Expertise into AI Agents

Transform your T-SQL diagnostic scripts into AI agents that work 24/7. Learn how to convert proven queries for blocking sessions, slow procedures, and performance issues into tools that LLMs can contextually execute through the Model Context Protocol (MCP). Discover practical patterns for building autonomous database agents with enterprise-grade guardrails, turning years of troubleshooting expertise into automated intelligence that maintains production reliability standards

Cost Effective In-Memory Analytics at Scale: Using DuckDB as Application-Layer Offload

Use DuckDB to embed analytics in your application process for in-memory, scalable, low-latency, high-concurrency, low-cost queries without data warehouse complexity. Working code, live demos, performance measurements, and how to address limitations.

Cross-Stitch Your Career: Level Up from DBA to Fabric Administrator