Items related to Data Pipelines with Apache Airflow
Synopsis
Data Pipelines with Apache Airflow teaches you how to build and maintain effective data pipelines.
Summary
A successful pipeline moves data efficiently, minimizing pauses and blockages between tasks, keeping every process along the way operational. Apache Airflow provides a single customizable environment for building and managing data pipelines, eliminating the need for a hodgepodge collection of tools, snowflake code, and homegrown processes. Using real-world scenarios and examples, Data Pipelines with Apache Airflow teaches you how to simplify and automate data pipelines, reduce operational overhead, and smoothly integrate all the technologies in your stack.
Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.
About the technology
Data pipelines manage the flow of data from initial collection through consolidation, cleaning, analysis, visualization, and more. Apache Airflow provides a single platform you can use to design, implement, monitor, and maintain your pipelines. Its easy-to-use UI, plug-and-play options, and flexible Python scripting make Airflow perfect for any data management task.
About the book
Data Pipelines with Apache Airflow teaches you how to build and maintain effective data pipelines. You’ll explore the most common usage patterns, including aggregating multiple data sources, connecting to and from data lakes, and cloud deployment. Part reference and part tutorial, this practical guide covers every aspect of the directed acyclic graphs (DAGs) that power Airflow, and how to customize them for your pipeline’s needs.
What's inside
Build, test, and deploy Airflow pipelines as DAGs
Automate moving and transforming data
Analyze historical datasets using backfilling
Develop custom components
Set up Airflow in production environments
About the reader
For DevOps, data engineers, machine learning engineers, and sysadmins with intermediate Python skills.
About the author
Bas Harenslak and Julian de Ruiter are data engineers with extensive experience using Airflow to develop pipelines for major companies. Bas is also an Airflow committer.
Table of Contents
PART 1 - GETTING STARTED
1 Meet Apache Airflow
2 Anatomy of an Airflow DAG
3 Scheduling in Airflow
4 Templating tasks using the Airflow context
5 Defining dependencies between tasks
PART 2 - BEYOND THE BASICS
6 Triggering workflows
7 Communicating with external systems
8 Building custom components
9 Testing
10 Running tasks in containers
PART 3 - AIRFLOW IN PRACTICE
11 Best practices
12 Operating Airflow in production
13 Securing Airflow
14 Project: Finding the fastest way to get around NYC
PART 4 - IN THE CLOUDS
15 Airflow in the clouds
16 Airflow on AWS
17 Airflow on Azure
18 Airflow in GCP
"synopsis" may belong to another edition of this title.
About the Author
Bas Harenslak and Julian de Ruiter are data engineers with extensive experience using Airflow to develop pipelines for major companies including Heineken, Unilever, and Booking.com. Bas is a committer, and both Bas and Julian are active contributors to Apache Airflow.
From the Back Cover
Pipelines can be challenging to manage, especially when your data has to flow through a collection of application components, servers, and cloud services. Airflow lets you schedule, restart, and backfill pipelines, and its easy-to-use UI and workflows with Python scripting has users praising its incredible flexibility. Data Pipelines with Apache Airflow takes you through best practices for creating pipelines for multiple tasks, including data lakes, cloud deployments, and data science.
Data Pipelines with Apache Airflow teaches you the ins-and-outs of the Directed Acyclic Graphs (DAGs) that power Airflow, and how to write your own DAGs to meet the needs of your projects. With complete coverage of both foundational and lesser-known features, when you're done you'll be set to start using Airflow for seamless data pipeline development and management.
Key Features
Framework foundation and best practices
Airflow's execution and dependency system
Testing Airflow DAGs
Running Airflow in production
For data-savvy developers, DevOps and data engineers, and system
administrators with intermediate Python skills.
About the technology
Data pipelines are used to extract, transform and load data to and from multiple sources, routing it wherever it's needed -- whether that's visualisation tools, business intelligence dashboards, or machine learning models. Airflow streamlines the whole process, giving you one tool for programmatically developing and monitoring batch data pipelines, and integrating all the pieces you use in your data stack.
Bas Harenslak and Julian de Ruiter are data engineers with extensive experience using Airflow to develop pipelines for major companies including Heineken, Unilever, and Booking.com. Bas is a committer, and both Bas and Julian are active contributors to Apache Airflow.
"About this title" may belong to another edition of this title.
Search results for Data Pipelines with Apache Airflow
Stock Image
Data Pipelines with Apache Airflow
Seller: HPB-Red, Dallas, TX, U.S.A.
Seller rating 5 out of 5 stars
paperback. Condition: Good. Connecting readers with great books since 1972! Used textbooks may not include companion materials such as access codes, etc. May have some wear or writing/highlighting. We ship orders daily and Customer Service is our top priority! Seller Inventory # S_403672461
Quantity: 1 available
Stock Image
Data Pipelines with Apache Airflow
Seller: thebookforest.com, San Rafael, CA, U.S.A.
Seller rating 5 out of 5 stars
Condition: LikeNew. Text block, wraps and binding are in like new condition, without markings of any kind. Extremely fine. Supporting Bay Area Friends of the Library since 2010. Well packaged and promptly shipped. Seller Inventory # 1LAUHV002ZVZ
Quantity: 1 available
Stock Image
Data Pipelines with Apache Airflow
Seller: Goodwill, Brooklyn Park, MN, U.S.A.
Seller rating 5 out of 5 stars
Condition: Good. Corners are bent. There is significant highlighting through out the book. Cover/Case has some rubbing and edgewear. Access codes, CDs, slipcovers and other accessories may not be included. Seller Inventory # 2Y6RVS0042XC_ns
Quantity: 1 available
Stock Image
Data Pipelines with Apache Airflow
Seller: Goodwill Books, Hillsboro, OR, U.S.A.
Seller rating 5 out of 5 stars
Condition: Good. Signs of wear and consistent use. Seller Inventory # 3IIT4R0045FL_ns
Quantity: 1 available
Seller Image
Data Pipelines with Apache Airflow
Seller: Goodwill of Greater Milwaukee and Chicago, Racine, WI, U.S.A.
Seller rating 5 out of 5 stars
Condition: acceptable. Book is considered to be in acceptable condition. The actual cover image may not match the stock photo. Book may have one or more of the following defects: noticeable wear on the cover dust jacket or spine; curved, dog eared or creased page s ; writing or highlighting inside or on the edges; sticker s or other adhesive on cover; CD DVD may not be included; and book may be a former library copy. Seller Inventory # SEWV.1617296902.A
Quantity: 1 available
Seller Image
Data Pipelines with Apache Airflow
Seller: Goodwill of Silicon Valley, SAN JOSE, CA, U.S.A.
Seller rating 4 out of 5 stars
Condition: acceptable. Supports Goodwill of Silicon Valley job training programs. The cover and pages are in Acceptable condition! Any other included accessories are also in Acceptable condition showing use. Use can include some highlighting and writing, page and cover creases as well as other types visible wear such as cover tears discoloration, staining, marks, scuffs, etc. All pages intact. Seller Inventory # GWSVV.1617296902.A
Quantity: 1 available
Stock Image
Data Pipelines with Apache Airflow
Seller: WorldofBooks, Goring-By-Sea, WS, United Kingdom
Seller rating 5 out of 5 stars
Paperback. Condition: Very Good. The book has been read, but is in excellent condition. Pages are intact and not marred by notes or highlighting. The spine remains undamaged. Seller Inventory # GOR012040616
Quantity: 1 available
Stock Image
Data Pipelines With Apache Airflow
Seller: medimops, Berlin, Germany
Seller rating 5 out of 5 stars
Condition: good. Befriedigend/Good: Durchschnittlich erhaltenes Buch bzw. Schutzumschlag mit Gebrauchsspuren, aber vollständigen Seiten. / Describes the average WORN book or dust jacket that has all the pages present. Seller Inventory # M01617296902-G
Quantity: 1 available
Stock Image
Data Pipelines with Apache Airflow
Seller: Goodvibes Books, STAFFORD, TX, U.S.A.
Seller rating 4 out of 5 stars
Condition: New. New Book. Seller Inventory # 1617296902-SBX
Quantity: 3 available
Stock Image
Data Pipelines with Apache Airflow Format: Paperback
Seller: INDOO, Avenel, NJ, U.S.A.
Seller rating 5 out of 5 stars
Condition: New. Seller Inventory # 9781617296901
Quantity: Over 20 available