For more information on working with pools from the API, see the API documentation.īy default, all tasks in Airflow get assigned to the default_pool which has 128 slots. To create a pool, submit a POST request with the name and number of slots as the payload. The Airflow REST API: This option is available with Airflow version 2.0 and later. This can be useful if you have a large number of pools to define and doing so programmatically would be more efficient. With the Airflow CLI, you can also import pools from a JSON file with the import subcommand. See the Airflow CLI documentation for the full list of pool commands. The Airflow CLI: Run the airflow pools command with the set subcommand to create a new pool. You can define a name, the number of slots, and a description.
The Airflow UI: Go to Admin > Pools and add a new record. There are three ways you can create and manage pools in Airflow: To get the most out of this guide, you should have an understanding of: You'll also implement some sample DAGs that use pools to fulfill simple requirements.
#PYTHON QUEUE POOL EXAMPLE HOW TO#
In this guide, you'll learn basic Airflow pool concepts, how to create and assign pools, and what you can and can't do with pools. For example, tasks that make requests to the same API or database, or tasks that run on a GPU node of a Kubernetes cluster. They are often used in cases where you want to limit the number of parallel tasks that do a certain thing. Pools allow you to limit parallelism for an arbitrary set of tasks, allowing you to control when your tasks are run.
Airflow pools are designed for exactly this use case. For example, you might have many tasks that interact with the same source system, such as an API or database, that you don't want to overwhelm with requests. Unfortunately, horizontal scalability also necessitates some guardrails. With the right supporting infrastructure, you can run many tasks in parallel seamlessly. One of the benefits of Apache Airflow is that it is built to scale.
0 kommentar(er)
