Data Engineering the Hard Way: Developing Spark Applications on Kubernetes

Managed Services like Databricks are fantastic for allowing data teams to get to work quickly. But what if such an option doesn’t exist in your environment?

And what if such a DIY approach created other opportunities, such as enabling developers to work in a realistic enviroment locally on the desktop?

Using containers, on top of Kubernetes has enabled us to achieve this.

Join us for a discussion of how we started on this journey, and what we learned along the way.

Feedback link

Starts: 16:40 10th Mar 2022
Ends: 17:00 10th Mar 2022


Short Description
- Self-hosting Spark in Containers
Other technologies
- Docker, Kubernetes

Slidedeck Link

Data Engineering the Hard Way.pdf


Gavin Campbell
Edward Allen


Platform AgnosticDataOpsDeploymentDevelopingSuccessful DeliverySparkIntermediateWhat

The SQL Bits Story

SQLBits was formed in 2007 by a group of volunteers who were passionate about the SQL Server product suite and wanted to provide much-needed community-driven education to the data community.

As one of the largest data platform conferences in the world, we offer more opportunities to a wider audience.

15 Years

We’ve grown and expanded a lot since 2007.

2500 Participants

SQLBits is the best place to meet fellow data professionals.

82 Countries

We welcome data professionals from all over the globe.

1140 recorded sessions

All the live sessions are recorded and offered for free, year round.

Experience the SQLBits Conference


Want to be part of the SQLBits community?

Attend the London conference in-person or virtually on 

March 8-12, 2022 at ExCel London, UK.