extract_task >> process_task

default_args = 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime(2023, 3, 21), 'retries': 1, 'retry_delay': timedelta(minutes=5),

Apache Airflow is a popular open-source workflow management platform that programmatically schedules and monitors workflows. One of the key features of Airflow is its ability to manage and exchange data between tasks, which is achieved through a mechanism called XComs (short for "cross-communications").

You can explicitly push and pull data within your task, allowing you to use custom keys.

def process_data(**kwargs): # Pull the value from the upstream task ti = kwargs['ti'] # We use xcom_pull with the task_ids retrieved_id = ti.xcom_pull(task_ids='extract_task') print(f"Processing file: retrieved_id")

Tip: For passing large data, use external storage like S3, GCS, or an SQL table, and pass only the file path/reference through XCom. How to Use XComs: A Practical Guide There are two primary ways to work with XComs in Airflow. 1. The Automatic Way (Return Values)

You can manually push data with a specific key if you need to store multiple values from a single task.

There are two primary ways to interact with XComs: explicitly using the xcom_push / xcom_pull methods, or implicitly via the return value of a task.

Apache Airflow is the industry standard for workflow orchestration, allowing data engineers to build robust, DAG-driven data pipelines. By design, Airflow tasks are isolated units of work, often running on entirely different machines or containers. While this separation is essential for scalability, it introduces a challenge: Enter Airflow XComs (cross-communication).

from datetime import datetime, timedelta from airflow import DAG from airflow.operators.bash_operator import BashOperator

Because XComs are stored in the Airflow metadata database (PostgreSQL, MySQL), they are not suitable for passing large datasets (e.g., Pandas DataFrames, Spark partitions).

XComs are a powerful feature in Airflow that enable complex workflows by allowing tasks to exchange data with each other. By understanding how XComs work and using them judiciously, users can build more efficient and scalable workflows.