Leveraging Apache Spark for Efficient Data Encryption at Scale

Whilst data encryption is not a new concept at the file level, the ability to encrypt data at the row level in a table has massive benefits in terms of data security and control. In responding to the challenges of large-scale, row-level data encryption we have built Gecko: an efficient, auditable, and simple encryption ecosystem designed for Spark and Delta Lake.

Gecko has allowed us to simultaneously achieve the following benefits within our data platform:
– Automatically handle data deletion.
– Increase the overall security of PII data in our data lake.
– Maintain Non-PII data structure, in order to continue to provide analytical value and overall data integrity.
– Make PII data accessible when required.

This presentation will share:
– The core concepts behind the ecosystem.
– How Spark & Delta lake have been leveraged in these applications.
– Why these technologies have been essential in achieving the necessary requirements.

Feedback Link -

Starts: 11:30 11th Mar 2022
Ends: 11:50 11th Mar 2022


Short Description
- This session will discuss a custom solution we have built to solve the problem of row level encryption in a highly complex data lake.


Daniel Harrington


AzureManaging Big DataSecure and GovernDevelopingOptimisingSecurityData LakeData BricksPythonSparkIntermediateHow

The SQL Bits Story

SQLBits was formed in 2007 by a group of volunteers who were passionate about the SQL Server product suite and wanted to provide much-needed community-driven education to the data community.

As one of the largest data platform conferences in the world, we offer more opportunities to a wider audience.

15 Years

We’ve grown and expanded a lot since 2007.

2500 Participants

SQLBits is the best place to meet fellow data professionals.

82 Countries

We welcome data professionals from all over the globe.

1140 recorded sessions

All the live sessions are recorded and offered for free, year round.

Experience the SQLBits Conference


Want to be part of the SQLBits community?

Attend the London conference in-person or virtually on 

March 8-12, 2022 at ExCel London, UK.