Join 📚 Kevin's Highlights
A batch of the best highlights from what Kevin's read, .
When we unpack the common threads of how various people define data engineering, an obvious pattern emerges:
a **data engineer**
*gets data, stores it, and prepares it for consumption*
by **data scientists**, **analysts**, and others.
We define data engineering and data engineer as follows:
**Data engineering** is
the *development*, *implementation*, and *maintenance*
of **systems** and **processes** that take in raw data
and produce high-quality, consistent information
that supports downstream use cases,
such as analysis and machine learning.
**Data engineering** is
the intersection of
*security*,
*data management*,
*DataOps*,
*data architecture*,
*orchestration*, and
*software engineering*.
A **data engineer**
*manages the data engineering lifecycle*,
beginning with getting data from source systems and
ending with serving data for use cases,
such as analysis or machine learning.
Fundamentals of Data Engineering
Reis, Joe;Housley, Matt;
The core Azure technologies used to implement data engineering workloads include:
• Azure Synapse Analytics
• Azure Data Lake Storage Gen2
• Azure Stream Analytics
• Azure Data Factory
• Azure Databricks
Data engineering in Microsoft Azure - Training
wwlpublish
...catch up on these, and many more highlights
Readwise