Free Learning Roadmap

How to Become a Data Engineer

Design, build, and operate the data pipelines, warehouses, and infrastructure that power analytics, ML, and product decisions.

Topics
13
Resources
90
Cost
Free
Most resources are free or freemium
Format
Self-paced

1. Programming Foundations

core
Python

The default language for data engineering — pipelines, orchestration, glue code, and testing all live here.

Bash & the Unix Command Line

Every server, container, and CI runner speaks bash. Learn to move data with curl, jq, awk, sed, and pipes.

Git & Version Control

Data pipelines live in git. Branching, PR reviews, and code history matter as much for pipelines as for app code.

2. SQL & Relational Databases

core
SQL Fundamentals

Joins, aggregations, window functions, CTEs. The bread and butter of every data role — must be second nature.

PostgreSQL

The most feature-rich open-source RDBMS. Master its planner, indexes, JSONB, and extensions.

Data Modeling

3NF for OLTP, dimensional (Kimball) for OLAP. Knowing when to denormalize is a senior-level skill.

Query Optimization & Indexing

Reading EXPLAIN plans, picking the right index type, understanding join algorithms — table-stakes for scaling.

3. Data Warehousing & Modeling

core
Warehouse Fundamentals

OLAP vs OLTP, columnar storage, partitioning, clustering. Why warehouses are structurally different from OLTP databases.

Snowflake

The industry-leading cloud warehouse. Separation of storage and compute, virtual warehouses, and micro-partitions.

BigQuery

Google's serverless warehouse. Learn slot-based pricing, partitioning, clustering, and BQ ML.

dbt (data build tool)

SQL-first transformation framework. Testing, docs, lineage, and incremental models are what separates modern warehouses from stored-procedure hell.

4. Distributed Data Processing

core
Apache Spark

The default engine for batch data processing at scale. Learn DataFrames, Spark SQL, adaptive execution, and partitioning.

Batch Processing Concepts

MapReduce, shuffle, skew, spilling, and why joins fail at scale. Foundation for everything Spark/Flink/BigQuery.

5. Streaming Data

core
Apache Kafka

The de-facto event streaming platform. Master partitions, consumer groups, exactly-once, and schema registry.

Streaming Patterns

Event time vs processing time, watermarks, windowing, exactly-once semantics. Same concepts show up in Flink, Beam, Spark Structured Streaming.

Apache Flink

State-of-the-art streaming engine when you need low latency + strong state semantics.

6. Pipelines & Orchestration

core
ETL vs ELT

Modern data stack prefers ELT — load raw first, transform in the warehouse with dbt. Understand why.

Apache Airflow

The industry-standard orchestrator. DAGs, operators, sensors, TaskFlow API, and the pitfalls of using it as a data processor.

Dagster & Prefect

Modern alternatives to Airflow. Asset-based (Dagster) or flow-based (Prefect) — better ergonomics for pure-Python teams.

Idempotency, Retries, Backfills

Every production pipeline must handle re-runs, partial failures, and historical backfills gracefully. This is a design skill, not a tool.

7. Cloud Platforms

core
AWS Data Stack

S3 (storage), Glue (catalog + ETL), EMR (Spark), Athena (Presto SQL over S3), Redshift (warehouse), Kinesis (streaming).

GCP Data Stack

BigQuery, Dataflow (Beam), Pub/Sub, Cloud Storage, Composer (managed Airflow). GCP is arguably the most data-friendly cloud.

Azure Data Stack

Synapse Analytics, Data Factory, Fabric, Databricks integration. Especially important in enterprise/Microsoft shops.

8. Data Lakes & Lakehouse

recommended
Parquet, ORC, Avro

Columnar formats power modern data lakes. Understand compression, predicate pushdown, and schema evolution.

Delta Lake, Apache Iceberg, Hudi

Table formats that add ACID transactions, time travel, and schema evolution on top of object storage.

9. Data Quality & Testing

recommended
Great Expectations

Python-native declarative data validation. Assertions, docs, and profiling in one framework.

dbt Tests

Generic tests (unique, not_null, accepted_values, relationships) + custom SQL tests. Non-negotiable for any dbt project.

Data Contracts

Producer-consumer schema agreements enforced at pipeline boundaries. Emerging discipline — worth being early on.

10. Data Governance & Security

recommended
PII, PHI, & Privacy

GDPR, CCPA, DPDP (India), HIPAA. Pseudonymization, tokenization, right to be forgotten. Compliance is now table-stakes.

Data Catalogs — DataHub, Amundsen, OpenMetadata

Metadata management, lineage, discovery. Essential as your warehouse grows past 100 tables.

11. Observability & Monitoring

recommended
Data Observability

Freshness, volume, distribution, schema, lineage — the 5 pillars. Monte Carlo, Bigeye, Elementary, or roll-your-own.

Pipeline Monitoring

SLA breach alerts, retry visibility, cost tracking. Airflow's DAG-level views + Prometheus/Grafana or Datadog.

12. CI/CD & Infrastructure for Data

recommended
Docker & Containers

Every pipeline runs in a container. Understand images, layers, multi-stage builds, and volume mounts.

Terraform & IaC

Declarative infrastructure — warehouse permissions, S3 buckets, Airflow deployments. State management is where teams trip up.

GitHub Actions for Data Pipelines

CI for dbt tests, pipeline deploys, and scheduled runs. Cheaper than a dedicated CI stack for small teams.

13. Career & Soft Skills

optional
Stakeholder Management

Data engineers serve analysts, ML engineers, and PMs. Managing their contradictory asks is a huge share of the actual job.

Writing Technical Design Docs

Every non-trivial pipeline change deserves a design doc — problem, options, chosen approach, rollout plan.

Communities to Follow

Where working data engineers actually hang out — worth lurking in.

Take it further

Curated by WhatTNext Ai · methodology · all roadmaps · last updated 2026-07-29