The documentation you're currently reading is for version 3.9dev. Click here to view documentation for the latest stable version.

Tuning Action Runner Dispatcher Pool Size

When the action runner executes actions it uses two internal green thread pools - one for regular actions and one for workflows. Both of those pools have a hard limit on the maximum size. By default, it is 40 green threads for workflows and 60 green threads for non-workflow actions.

This hard limit is there to prevent resource exhaustion and to allow for a fair distribution across multiple action runner processes.

In some situations, such as if you have a lot of action-chain workflows with a delay policy, this limit can cause a deadlock to occur. (NB this deadlock can be reset by restarting action runner process which has dead locked).

To prevent such issues from occurring and to allow for a good server and action runner process resource utilization, you should adjust those pool sizes based on your configuration and workload requirements.

These settings are very hardware- and workload-specific (i.e. how many action runner processes are used, on how many servers, the type of actions being executed - long running vs short running, CPU intensive vs non-CPU intensive, etc.).

To effectively tune those limits you should monitor the following parameters:

  • Sizes of the RabbitMQ queues relating to the action executions

  • Action runner process resource usage (CPU and memory)

Once you have established baselines and have a good idea of your resource utilization and workloads, you can increase resource utilization and prevent deadlocks by tuning pool sizes and/or increasing/decreasing the number of action runner processes.

You can tune pool sizes by changing these settings in st2.conf:

# Internal pool size for dispatcher used by workflow actions.
workflows_pool_size = 40

# Internal pool size for dispatcher used by regular actions.
actions_pool_size = 60