Project archive

Project 12 / 16

Modern Data Engineering Guide

A 2026 first-principles data engineering guide — how data moves from raw source to trusted, query-ready tables: storage & file formats, SQL & query engines, modeling & warehousing, Spark, ingestion/CDC, dbt, orchestration, streaming, and the lakehouse.

Open live site

What I built

  1. 01

    Authored all 12 chapters — foundations, storage & formats, SQL & query engines, modeling & warehousing, Spark, ingestion & CDC, dbt, orchestration, streaming (Kafka), the lakehouse, data quality & governance, and career

  2. 02

    Curriculum mined from real 2026 data-engineering job postings and gap-checked by a completeness critic; authored concept-first (columnar storage, desired-state pipelines, exactly-once) then mapped to today's tools (Snowflake/BigQuery, dbt, Airflow, Kafka, Iceberg/Delta)

  3. 03

    Built on the shared `@throughline/guide-kit` design system with checkpoint quizzes; source-available (© 2026 To Yin Yu, All Rights Reserved), deployed on Vercel