Leveraging Apache Spark for Efficient Data Encryption at Scale
Whilst data encryption is not a new concept at the file level, the ability to encrypt data at the row level in a table has massive benefits in terms of data security and control. In responding to the challenges of large-scale, row-level data encryption we have built Gecko: an efficient, auditable, and simple encryption ecosystem designed for Spark and Delta Lake.
Gecko has allowed us to simultaneously achieve the following benefits within our data platform:
– Automatically handle data deletion.
– Increase the overall security of PII data in our data lake.
– Maintain Non-PII data structure, in order to continue to provide analytical value and overall data integrity.
– Make PII data accessible when required.
This presentation will share:
– The core concepts behind the ecosystem.
– How Spark & Delta lake have been leveraged in these applications.
– Why these technologies have been essential in achieving the necessary requirements.
Feedback Link - https://sqlb.it/?6983
Starts: 11:30 11th Mar 2022
Ends: 11:50 11th Mar 2022
- This session will discuss a custom solution we have built to solve the problem of row level encryption in a highly complex data lake.
The SQL Bits Story
SQLBits was formed in 2007 by a group of volunteers who were passionate about the SQL Server product suite and wanted to provide much-needed community-driven education to the data community.
As one of the largest data platform conferences in the world, we offer more opportunities to a wider audience.
We’ve grown and expanded a lot since 2007.
SQLBits is the best place to meet fellow data professionals.
We welcome data professionals from all over the globe.
1140 recorded sessions
All the live sessions are recorded and offered for free, year round.
Experience the SQLBits Conference
Want to be part of the SQLBits community?
Attend the London conference in-person or virtually on