Preface: Why Data Engineering, Why Now
There is a version of this preface that quotes a McKinsey report about data volumes doubling every two years and a LinkedIn report naming data engineer one of the fastest-growing jobs in the world. All of that is true. None of it is why we wrote this book.
We wrote it because the job changed, and the resources did not keep up.
What Changed
For most of the last decade, data engineering meant ETL. Extract data from a source, transform it into a usable shape, load it into a warehouse. The tools were straightforward. The career path was clear. The skills were teachable.
Then three things happened at once.
The cloud became the default. On-premises data infrastructure did not disappear, but the new work moved to managed cloud services. A data engineer in 2026 is expected to know S3, Redshift, Databricks, Snowflake, Azure Data Factory, and BigQuery at minimum, plus the infrastructure-as-code tooling that provisions them. The stack is wider than it has ever been.
Real-time became a requirement. Batch pipelines that ran at midnight were sufficient when dashboards were refreshed once a day. Businesses now expect near-real-time data for operations, fraud detection, personalization, and AI features. Kafka and streaming architectures moved from specialized knowledge to standard expectation.
AI joined the workflow. This is the shift that is still in progress. Data engineers are not just building pipelines for analysts anymore. They are building the data infrastructure that AI systems run on: the feature stores, vector databases, embedding pipelines, and retrieval systems that make models usable in production. Simultaneously, AI tools are becoming part of how engineers write code, debug pipelines, and design systems.
Why This Book Exists
The tutorials available online teach individual tools. Most courses stop before reaching production. The books that do cover the full stack are either years out of date or priced out of reach for engineers in markets where the salaries do not match the dollar-denominated course prices.
This book is free because the knowledge should not be a financial barrier to entering the profession.
It is practical because theory without production context does not build engineers. Every chapter connects the concept to how it appears in real work.
It is comprehensive because a data engineer who knows SQL but not orchestration, or Python but not cloud, is only partway there. The goal is to bring someone from zero to genuinely employable, or from junior to senior, in a single coherent path.
What This Book Is Not
It is not a preparation guide for a single certification. Certifications have value. This book aims at a broader target.
It is not a replacement for building things. The exercises are there for a reason. Reading about Airflow without ever deploying a DAG is like reading about swimming. The knowledge does not transfer to your hands.
It is not finished. Data engineering evolves. We update the book as the tools change and as we learn from people who use it and tell us what they needed that we did not cover.
A Note on the AI Era
Part 5 of this book covers agentic data engineering: using AI tools as active participants in the engineering workflow, not just as autocomplete. This is new enough that the practices are still settling. We cover what we know works in production today, while being honest about what is still emerging.
The more important point is this: learning data engineering in the AI era is not harder than it used to be. It is different. The tools are more powerful. The expectations are higher. The path from beginner to productive is actually shorter than it has ever been, because AI assistants reduce the friction on every step of the learning curve.
What still matters, and will always matter, is understanding what you are building, why it should be built that way, and what breaks when you get it wrong. That understanding is what this book is designed to give you.