Streaming data and privacy by design on GCP

2022

TL; DR

Learn about streaming millions of events per day into BigQuery, access to which is controlled using column level security

Session Details

We ingest about 10million raw events per day using GCP technologies Pub/Sub, Dataflow, & BigQuery. Once its there we apply column-level security to make sure our employees can't access data they're not supposed to be able to access. I'd love to tell you about how we do it. N.B. This is nothing to do with SQL Server. Or Azure. I just thought it might be interesting for folks that aren't used to GCP technologies to know more about them.

3 things you'll get out of this session

Jamie Thomson's previous sessions

Real world SSDT

SQL Server Data Tools (SSDT) provide a new declarative model for building SQL Server databases. Come and learn more from Craig and Jamie who have both been using SSDT to do that for many months.

A lap around the SSIS Catalog

SQL Server Integration Services (SSIS) 2012 includes a new database referred to as the SSIS Catalog. Come to this session to learn what it does and how its going to change how you do SSIS development.

Back to the SSIS Future

A look at the new SSIS features in SQL Server code-named Denali

SSIS Dataflow Performance tuning

Building performant data flows takes more than just dragging a few boxes onto a design surface. In this session I'll demonstrate that SSIS perf tuning is less about fine-grained tweaks and more about designing packages correctly in the first place.

SSIS Dataflow Performance tuning

Need to eek out a bit more oomph from your dataflows. This session might be of some use.