Creating a Metadata Driven Orchestration Framework Using Azure Data Integration Pipelines
Azure Data Factory and Synapse Integration Pipeline are the undisputed PaaS resources within the Microsoft Cloud for orchestrating data workloads. With a 100+ Linked Service connections, a flexible array of both control flow and data flow Activities there isn't much these pipelines can’t do as a wrapper over our data platform solutions. That said, the service may still require the support of other Azure resources for the purposes of logging, monitoring, compute and storage. In this session we’ll will focus on exactly that point and explore the problem faced when structuring many integration pipelines in a highly scaled architecture.
Once coupled with other resources, we’ll look at one possible solution to this problem of pipeline organisation to create a dynamic, flexible, metadata driven processing framework that complements our existing solution pipelines. Furthermore, we will explore how to bootstrap multiple orchestrators (across tenants if needed), design for cost with nearly free Consumption Plans and deliver an operational abstraction over all our processing pipelines.
Finally, we'll explore delivering this framework within an enterprise and consider an architect’s perspective on a wider platform of ingestion/transformation workloads with multiple batches and execution stages.
Feedback Link https://sqlb.it/?7026
Starts: 17:10 10th Mar 2022
Ends: 18:00 10th Mar 2022
- We'll explore delivering this framework within an enterprise and consider an architect’s perspective on a wider platform of ingestion/transformation workloads with multiple batches and execution stages.
The SQL Bits Story
SQLBits was formed in 2007 by a group of volunteers who were passionate about the SQL Server product suite and wanted to provide much-needed community-driven education to the data community.
As one of the largest data platform conferences in the world, we offer more opportunities to a wider audience.
We’ve grown and expanded a lot since 2007.
SQLBits is the best place to meet fellow data professionals.
We welcome data professionals from all over the globe.
1140 recorded sessions
All the live sessions are recorded and offered for free, year round.
Experience the SQLBits Conference
Want to be part of the SQLBits community?
Attend the London conference in-person or virtually on