A Day Full of Azure Data Factory

2020

TL; DR

In this full-day workshop, you will learn the skills you need to build and orchestrate hybrid, complex and scalable data pipelines using Azure Data Factory (ADF).

Session Details

If we want to achieve any data processing in Azure you need an umbrella service to manage, monitor and schedule your solution. For a long time when working on premises, the SQL Agent has been our go-to tool, combined with T-SQL and SSIS packages. It’s now time to upgrade our skills and start using cloud native services to achieve the same thing on the Microsoft Cloud Platform. Within a PaaS only Modern Data Platform, the primary component for delivering that orchestration is Azure Data Factory, combined with various other compute resources.

In this full day of training we’ll start with the basics and learn how to orchestrate your Azure Data Platform end to end. You will learn how to build our Azure ETL/ELT pipelines using all Data Factory has to offer. Plus, consider hybrid architectures, dynamic design patterns, think about lifting and shifting legacy packages, and explore complex bootstrapping to orchestrate everything within your solution.

COMPLETE SESSION AGENDA

Module 1: Data Factory Fundamentals
- What is it and why use it?
- Resource Components
- Common Activities
- Execution Dependencies

Module 2: Uploading Data to Azure
- Integration Runtimes
- Hosted IR Patterns
- Private Endpoints

Module 3: Using SSIS Packages in Azure
- SSIS Integration Runtime
- Packages Running on PaaS
- Scaling Out Package Execution

Module 4: Data Flows
- Mapping Data Flows
- Wrangling Data Flows
- Configuration
- Use Cases

Module 5: Metadata Driven Pipelines
- Expressions
- Dynamic Pipelines
- Orchestration Framework – ADFprocfwk.com

Module 6: Monitoring Alerting Security
- Logging – Kusto Queries
- Alerting
- Roll Based Access
- Using Azure Key Vault

Module 7: Pricing & Limitations
- Data Integration Units
- Data Flow Compute
- Wider Platform Orchestration & Cost Control
- Resource Limitations

Module 8: CI/CD with Azure DevOps
- Source Control vs Developer UI
- ARM Template Deployments
- PowerShell Deployments

Module 9: Data Factory in Production
- Testing
- Bootstrapping
- Best Practices

Module 10: Wrap Up
- Conclusions
- Questions
- Homework

If that’s not enough content for one day, you will also get access to a set of hands-on labs that you can work through at your own pace. Whether you are new to Azure Data Factory or have some experience, you will leave this workshop with new skills and ideas for your projects.