DataHub in Practice: Metadata Platform Design, Ingestion, and Governance

"DataHub in Practice: Metadata Platform Design, Ingestion, and Governance"

Modern data organizations don’t fail because they lack data—they fail because they can’t trust, find, govern, or change it safely. This book is written for experienced data engineers, platform engineers, and data governance leaders who need more than a catalog walkthrough: they need a production-ready metadata system of record. It treats DataHub as an operational platform with real-world constraints, competing producers, and measurable outcomes.

You’ll learn how DataHub’s architecture works end-to-end (GMS, storage, indexing, and messaging), how its metadata model (URNs, entities, aspects) enables safe evolution, and how modern metadata change protocols (MCP/MCL vs legacy MCE/MAE) support replayable, idempotent, observable pipelines. The book goes deep on ingestion engineering—recipes, transformers, incremental strategies, failure isolation, and CI quality gates—then moves into extension patterns with custom connectors and API-driven automation for bulk updates and enrichment.

On the governance side, it shows how to build domains and glossary as durable organizational structure, implement stewardship workflows and documentation standards, roll out authorization policies at scale, and operationalize lineage for impact analysis and change management. Expect trade-offs, anti-patterns, and production guidance anchored in upgrade and version milestone realities.