Data Engineering Design Patterns

Data engineering has gained even more relevance than before, and data engineering patterns are key to the successful implementation of data engineering projects. This book enables a data engineer to not only become familiar with data engineering patterns but also understand their application in real world use cases.

This book presents a comprehensive collection of data engineering patterns, each illustrated with relevant enterprise use cases to highlight their value and simplicity. It showcases both open-source and cloud technologies, guiding readers in building data systems for on-premise and cloud environments. The book covers patterns for data ingestion, transformation, storage, and serving, while also offering insights into performance engineering for data pipelines. Once we understand fundamental data engineering patterns, we then shift focus to patterns that help us build high-performance low latency data systems. We cover data caching, partitioning, replication, and how to select the technology stack for building out the patterns in this book.

By the end of the book, readers will have a deep understanding of various data engineering use cases and will be able to map the appropriate patterns to address them. They will also be equipped to choose the right technical stack for implementing these patterns, enabling them to create robust and efficient data systems in a secure and a cost-effective manner.

What you will learn

● Key data engineering patterns.

● Data ingestion and processing patterns.

● Modern architectures like Lambda.

● Explore time-tested data patterns of ETL and ELT.

● Modern data systems like data lake and medallion architectures.

● Domain-specific patterns and also on data orchestration, observability, and security.

● Overcoming performance challenges in building complex data systems.

Who this book is for

This book is designed for data engineers with beginner to intermediate experience in building enterprise-grade data systems. ETL developers transitioning into data engineering roles will also find this book valuable for understanding essential data engineering patterns. The code snippets provided throughout the book are written in Python or Scala, so a basic understanding of either language will help readers more easily grasp the concepts presented.

Table of Contents

1. Understanding Data Engineering

2. Data Engineering Patterns, Terminologies, and Technical Stack

3. Batch Ingestion and Processing

4. Real-time Ingestion and Processing

5. Micro-batching

6. Lambda Architecture

7. ETL and ELT

8. Data Fundamentals

9. Databases and Transactional Data

10. Data Warehouse and Data Analytics

11. Data Lake and Medallion Architecture

12. Data Replication and Partitioning

13. Hot Versus Cold Data Storage

14. Data Caching and Low Latency Serving

15. Data Search Patterns

16. Domain Specific Patterns

17. Data Security Patterns

18. Data Observability and Monitoring Patterns

19. Idempotency and Deduplication Patterns

20. Data Orchestration Patterns

21. Common Performance Pitfalls

22. Technology and Infrastructure Selection

23. Recap and Next Steps