Find the Spark as a SQL data warehouse developer

You are a data warehouse developer that´s been using SQL Server on-premise or in Azure to build your data warehouses. You are comfortable in writing SQL scripts, stored procedures and creating views to fuel your data warehouse. You have heard about the new kid in town Spark but you have not yet take the plunge and are wondering if you have to or if you should.

If the above is something like you then this is the session for you. The session is built largely on my own experience. I had been doing data warehouse in SQL for years and I had decided Spark was nothing for me. I felt there was enough to do in SQL so there was no reason to add Spark to the tools I was using. It seemed like a lot of learning and when you are an old dog that can be difficult to do 😊.

The reality is that Spark offers some very cool features and it´s not that different to how you have been doing things. Actually the concept of how to do a data warehouse do not change that much even though it´s built in a data lake and is called a data lakehouse. Yes the technology is different and the language is a lot or slightly different depending on what you choose but transitioning was lot easier than I thought.

We will start by comparing SQL data warehouse and Spark lakehouse. What is common and what is different. Then we will look at a demo on how you could do things in a Spark lakehouse all the time comparing it to how we are used to doing it in SQL data warehouse. We will then move onto understanding how to acquire the right knowledge to get started with Spark as well as how to try it out without breaking the bank.

After attending this session you will be better able to understand if Spark lakehouses are for you and how you can figure out how to learn the required skills. You will also know how to get started for free or with a very low cost.

To help attendees with SQL background to understand what the differences are between SQL Server data warehouse and Spark lakehouse To show how the transition from SQL Server to Spark does not have to be difficult To inspire SQL Server data warehouse developers to try out Spark