SESSION Details
Data Engineering the Hard Way: Developing Spark Applications on Kubernetes
Managed Services like Databricks are fantastic for allowing data teams to get to work quickly. But what if such an option doesn’t exist in your environment?
And what if such a DIY approach created other opportunities, such as enabling developers to work in a realistic enviroment locally on the desktop?
Using containers, on top of Kubernetes has enabled us to achieve this.
Join us for a discussion of how we started on this journey, and what we learned along the way.
Feedback link https://sqlb.it/?7121
Starts: 16:40 10th Mar 2022
Ends: 17:00 10th Mar 2022
Q&As
Short Description
- Self-hosting Spark in Containers
Other technologies
- Docker, Kubernetes
Slidedeck Link
Data Engineering the Hard Way.pdf
Speakers
Gavin Campbell
Edward Allen
Tags
Platform AgnosticDataOpsDeploymentDevelopingSuccessful DeliverySparkIntermediateWhat
The SQL Bits Story
SQLBits was formed in 2007 by a group of volunteers who were passionate about the SQL Server product suite and wanted to provide much-needed community-driven education to the data community.
As one of the largest data platform conferences in the world, we offer more opportunities to a wider audience.
15 Years
We’ve grown and expanded a lot since 2007.
2500 Participants
SQLBits is the best place to meet fellow data professionals.
82 Countries
We welcome data professionals from all over the globe.
1140 recorded sessions
All the live sessions are recorded and offered for free, year round.
Experience the SQLBits Conference
Want to be part of the SQLBits community?
Attend the London conference in-person or virtually on
March 8-12, 2022 at ExCel London, UK.
