![lakehouse architecture lakehouse architecture](https://sqlofthenorth.files.wordpress.com/2021/03/elt.png)
This leads us on to… Pattern 2 – Extract, Load & Transform (Cloud Data Warehouse Variant) Extract & Load This pattern has been around for many, many years, and, despite the rise of “modern” patterns such as Data Lakes and Data Lakehouses, this ETL pattern is still perfectly fine for many analytic implementations. This allows for the creation of an elegant restart point in the event of failure, and prevents overloading the source systems. Such a semantic layer allows for querying logic to be abstracted away from business consumers, who, rather than writing SQL to create reports, they simply consume your semantic model and create reports with a drag and drop style interface.Ī common implementation variant of this includes the use of a staging area, often as an untyped “staging” schema in our target data store. You may also see a semantic layer on top of your reporting schema, such as Azure Analysis Services or Power BI. The transformed data is landed in a target database schema, such as an Azure SQL Database, often modelled in a star schema to allow cleaned, curated data to be presented to your reporting tools.
![lakehouse architecture lakehouse architecture](https://images.adsttc.com/BOTY/ava/2020/slideshow/1WAqQIBkxm-w-HHK87NlRhcfhCb10gHF9.jpg)
As the data volumes are relatively light, it allows the transformation to take place in memory inside the ETL engine itself, rather than having to push down compute to a specific data store and processing engine. In this ETL pattern there is typically no “data lake” but instead extracted data is transformed “in flight” by the aforementioned ETL Tool. Extract & Transformĭata volumes in this pattern generally don’t tend to be “big data” and often fall into the GB – TB range in size. This extraction, and subsequent transformations, are often done using an ETL tool such as SQL Server Integration Services. In this pattern – the traditional ETL pattern that has been around for decades – data is first extracted from line of business systems and files, such as SQL Server, PostgreSQL through to csv and text files.
![lakehouse architecture lakehouse architecture](https://architecturecompetitions.com/upload/02.06.2022/e8d47215faeaceedab0fa8a327a03007.png)
A minor change in wording, but a significant one. From a data perspective, nothing lends itself better to having both of these as ETL/ELT.įor those not familiar with the terms – they mean Extract, Transform & Load AND Extract, Load and Transform respectively.