SQLBits 2020

The Modern Data Platform: Automate Everything

Automation for deploying, monitoring and alerting in a data lake data platform using Terraform, DevOps, Application Insights, Log Analytics and PowerBI.

As the adoption of data lake architectures for data platform grows the use of automation becomes increasingly important. In this session we will discuss the services and tools that go into a modern data platform on Azure, how the deployments into different environments can be managed using Terraform and Azure DevOps. We will discover how both PaaS and user-developed components can be monitored using Log Analytics and Application Insights. Then we will drill down into how the same data from all components in the platform can be used to both monitor and alter to any issues using Logic Apps, as well as to build analytical reports that we can deliver in PowerBI.