Lots of ADF, lots of data, lots of automation and lots of lessons learned. Backups aren't as useful as they look, dedicated pools have several "interesting" features in the small print, ADF is not always your friend and how to mitigate the sometimes deadly embrace of throughput vs concurrency trade off.
The video is not available to view online.
During this session, we will talk about Azure Databricks' key features, and typical scenarios where Spark can fit, will see a lot of demos and I will share my top list of Azure Databricks best practices.
The video is not available to view online.
AutoML which stands for Automated Machine Learning empowers data teams to quickly build and deploy machine learning models. It aims to reduce the time and expertise required to generate a machine learning model by automating the heavy lifting of preprocessing, feature engineering, model creation, tuning and evaluation. When it comes to machine learning in Microsoft Azure, there are two main options for running your AutoML: (1) Azure Machine Learning Service and (2) Azure Databricks. This session will aim to introduce how to develop ML models using AutoML on both platforms, as well as the features of each and why you would choose one over the other.
The video is not available to view online.
Have you ever found yourself at the start of an Azure data engineering project, unsure about what tool to choose? Speak no more! In this session we will discuss three often used data engineering tools on Azure: - Azure Data Factory - Azure Databricks - Azure Synapse Analytics
The video is not available to view online.
In this session, seasoned data engineer and youtube grumbler Simon Whiteley takes us on a journey through the current industry trends and buzzwords, carving through the hype to get at the underlying ideals.
The video is not available to view online.
The building a Data Lakehouse with Azure Databricks session is a dynamic presentation that benefits an array of Data professionals - data engineers, data architects, data analysts - by covering practices and techniques used to move, shape, orchestrate, and organize data that is made available for analytical solutions that build business insights.
The video is not available to view online.
Based on real world experience let’s think about just how far the breadth of our knowledge now needs to reach when starting from nothing and building a complete Microsoft Azure Data Analytics solution.
The video is not available to view online.
Modern software engineering and management for ETL, so data analysts and engineers can spend less time on tooling and focus on getting value from data.
The video is not available to view online.
In this session, we will see how to use Delta Live Tables to build fast, reliable, scalable, and declarative ETL pipelines on Azure Databricks platform.
The video is not available to view online.
In this session I'll guide you from through a secure reference architecture with Data Factory, Databricks, Data Lake, and Azure Synapse, working together as a secure, fully productionised platform. Each has their own idiosyncrasies, but this session will teach you the options available and the pitfalls to avoid.
The video is not available to view online.
We're not supposed to use production in dev right! But generating proper test data is not easy, get's even harder when you need quite a lot of it. I generate Terabytes of it, and without much trouble. Let me show you how!
The video is not available to view online.
Delta Live Tables is a new framework available in Databricks that aims to accelerate building data pipelines by providing out of the box scheduling, dependency resolution, data validation and logging. We'll cover the basics, and then get into the demo's to show how we can: - Setup a notebook to hold our code and queries - Ingest quickly and easily into bronze tables using Auto Loader - Create views and tables on top of the ingested data using SQL and/or python to build our silver and gold layers - Create a pipeline to run the notebook - See how we can run the pipeline as either a batch job, or as a continuous job for low latency updates - Use APPLY CHANGES INTO to upsert changed data into a live table - Apply data validation rules to our live table definition queries, and get detailed logging info on how many records caused problems on each execution. By the end of the session you should have a good view of whether this can help you build our your next data project faster, and make it more reliable.
The video is not available to view online.
Going from an machine learning model trained on your laptop in a notebook called “trainmodelV1Final_FINAL (1).ipynb” to a system ready to deploy is difficult. However, MLOps (a set of principles to prepare your model for prime time) is here to help! This talk is an introduction to all the elements you need to get your code production-ready - CI/CD, dev/UAT/prod, pipelines, and more! We'll walk through system diagrams, with a focus on Azure, but the takeaways will all be platform agnostic. Make sure your model deployment isn’t an ML-Flop!
The video is not available to view online.
In this demo-rich session you will learn how you can implement a framework for data quality validation and monitoring, spanning your end-to-end data platform including Power BI!
The video is not available to view online.
Introduction into Spark Execution Plans for Databricks for optimizing code and execution.
The video is not available to view online.
In this demo filled session, learn how to view lineage data from Databricks Unity Catalog in Microsoft Purview.
The video is not available to view online.
Are you planning a migrating of your data platform to Azure Synapse Analytics? Is it currently based on Azure Databricks, Azure Data Factory, and Azure Data Lake Storage? Make sure to grab these 10 tips! We'll talk about Spark compatibilities, orchestration pitfalls and solution deployment differences.
Introduction into Terraform, Databricks provider and steps required to build an automated solution to provision Databricks workspace and resources into Azure cloud platform using Terraform.
Overview to some of the more popular Azure Data Engineering services used to analyze data
A intro session showing the whys, how's and what's for building a Data Lakehouse in Azure Synapse Analytics
<<12>>