A customer requested a solution for subscribing to data changes from their on-prem SQL Server in Azure - in near real time. Azure Data Factory is a fantastic data engineering solution, but it is not the first choice for real time data transfer due to cost. Synapse Link for SQL Server might have been part of the solution - if the SQL Server version was 2022. Unfortunately, neither of these tools can handle subscriptions to events out of the box - we needed to come up with something else.
We settled on the open source project Debezium. Debezium uses SQL Server change data capture (or similar mechanisms in other supported databases) to send the changed rows to a Kafka stream - a stream that Azure Event Hub can accept. With the data in Azure Event Hub it can be used anywhere, all while completely decoupled from the data source.
Do you want to sink the data into Parquet or Delta Lake through Azure Stream Analytics? You can. Do you want to send the data to an Azure Function? You can do that too - and so much more. Azure Event Hub is the gateway to all the fun services in Azure!
In this session I will walk you through how the solution works, what parts are involved, show you how to set everything up and point out some pitfalls.
Come learn how to build your own real time data change capture into Azure!
