Nginx and WSGI

Production StackStorm installations use nginx for SSL termination, serving Web UI static content, and running st2auth and st2api as WSGI apps via gunicorn/uwsgi. StackStorm nginx configurations can be found at /etc/nginx/sites-enabled/st2*.conf.

st2auth and st2api can also run using a built-in simple Python server. This is used for development and strongly discouraged for any production. Be aware that some settings in /etc/st2.conf are only effective when running in development mode, and don’t apply when running under WSGI servers. Refer to the comments in st2.conf.sample.

Configure MongoDB

StackStorm requires a connection to MongoDB to operate.

In /etc/st2/st2.conf include the following section:

[database]
host = <MongoDB host>
port = <MongoDB server port>
db_name = <User define database name, usually st2>
username = <username for db login>
password = <password for db login>

The username and password properties are optional.

StackStorm also supports MongoDB replica sets using MongoDB URI string.

In /etc/st2/st2.conf include the following section:

[database]
host = mongodb://<#MDB_NODE_1>,<#MDB_NODE_2>,<#MDB_NODE_3>/?replicaSet=<#MDB_REPLICA_SET_NAME>

You can also add ports, usernames and passwords, etc to your connection string. See https://docs.mongodb.com/manual/reference/connection-string/
To understand more about setting up a MongoDB replica set, see https://docs.mongodb.com/manual/replication/

StackStorm also supports SSL/TLS to encrypt MongoDB connections. A few extra properties need be added to the configuration apart from the ones outlined above.

In /etc/st2/st2.conf include the following section:

[database]
...
ssl = <True or False>
ssl_keyfile = <Path to key file>
ssl_certfile = <Path to certificate>
ssl_cert_reqs = <One of none, optional or required>
ssl_ca_certs = <Path to certificate form mongod>
ssl_match_hostname = <True or False>

ssl - Enable or disable connection over TLS/SSL or not. Default is False.
ssl_keyfile - Private keyfile used to identify the local connection against MongoDB. If specified ssl is assumed to be True.
ssl_certfile - Certificate file used to identify the local connection. If specified ssl is assumed to be True.
ssl_cert_reqs - Specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided.
ssl_ca_certs - Certificates file containing a set of concatenated CA certificates, which are used to validate certificates passed from MongoDB.
ssl_match_hostname - Enable or disable hostname matching. Not recommended to disable and defaults to True.

Note

Only certain distributions of MongoDB support SSL/TLS:

MongoDB enterprise versions have SSL/TLS support.
Build MongoDB from source to enable SSL/TLS support. See https://github.com/mongodb/mongo/wiki/Build-Mongodb-From-Source for more information.

StackStorm also supports transport / network level compression since StackStorm v3.5.0. Example below shows which configuration options need to be set to enable compression:

In /etc/st2/st2.conf include the following section:

[database]
...
compressors = <zstd|zlib>
zlib_compression_level = 6

compressors - A comma delimited list of compression algorithm client supports for network / transport level compression which should be advertised to the server. Actual algorithm used will then be determined based algorithm which is supported both by the client and the server.
zlib_compression_level - Compression level to use (possible values are from -1 to 9) when compressors is set to zlib.

Note

Zstandard (zstd) compression is only supported by MongoDB server version 4.0 and above.

Compression is always a trade off - CPU cycles for compression and decompression operation are exchanged for less bytes transfered over the network.

How compression affects actual database operation durations and throughput is very much workload and deployment specific (spare CPU cycles, network setup, available bandwidth, is the NIC being saturated, is MongoDB hosted on the same server / AZ, etc).

Our micro benchmarks showed that enabling zstandard compression seems to have no perceived impact on database read and write operation duration, but your milleage may vary.

Compression may come especially handy when working with large Trigger Instance and Execution objects (aka executions which produce large textual results). Executions and Trigger Instances usually contain large textual result which compresses very well and can result in substantial reduction in network traffic.

Configure RabbitMQ

StackStorm uses RabbitMQ for messaging between its services.

In /etc/st2/st2.conf include the following section:

[messaging]
url = amqp://#RMQ_USER:#RMQ_PASSWD@#RMQ_HOST:#RMQ_PORT/#RMQ_VHOST

The #RMQ_VHOST property is optional and can be left blank.

StackStorm also supports SSL/TLS to encrypt RabbitMQ connections. To enable SSL, you need to set messaging.ssl config option to True or pass ?ssl=true query parameter at the end of the connection URL string.

For example:

[messaging]
url = amqp://guest:[email protected]:5671/
ssl = True

or

[messaging]
url = amqp://guest:[email protected]:5671/?ssl=true

Note

Keep in mind that if you want to use custom ssl options (e.g. use a key and cert file or use a custom ca cert), you need to enable ssl via messaging.ssl option and not via connection URL string. When SSL is enabled via connection URL string default SSL options will be used which can’t be changed.

In addition to encrypted connection to RabbitMQ, some other SSL related options which are documented below are also supported:

[messaging]
...
ssl = <True or False>
ssl_keyfile = <Path to key file>
ssl_certfile = <Path to certificate>
ssl_cert_reqs = <One of none, optional or required>
ssl_ca_certs = <Path to CA certificate>
login_method = <One of PLAIN, AMQPLAIN or EXTERNAL>

ssl - Enable or disable connection over TLS/SSL or not. Default is False.
ssl_keyfile - Private keyfile used to identify the local connection against RabbitMQ. If specified ssl is assumed to be True.
ssl_certfile - Certificate file used to identify the local connection. If specified ssl is assumed to be True.
ssl_cert_reqs - Specifies whether a certificate is required from the other side of the connection, and whether it will be validated if provided.
ssl_ca_certs - Certificates file containing a set of concatenated CA certificates, which are used to validate certificates passed from RabbitMQ.
login_method - Login method to use. Default is PLAIN. Other possible options are AMQPLAIN and EXTERNAL.

Note

If you want to use custom SSL settings (e.g. using a different ca bundle or similar) you should specify all those options as part of the st2.conf and also do the same for enabling ssl using messaging.ssl option. Combining URL and config parameters for SSL doesn’t work - if you enable ssl as part of the URL it will use default SSL settings, but you won’t be able to specify a custom value for cert_reqs ca_certs and other SSL related options.

Note

RabbitMQ doesn’t expose an SSL / TLS listener by default and needs to be configured to enable TLS support. For more information, refer to the official documentation - Enabling TLS Support in RabbitMQ.

StackStorm also supports compressing payloads which are sent over the message bus since StackStorm v3.5.0. Example below shows which configuration options need to be set to enable compression:

[messaging]
...
compression = <zstd|lzma|bz2|gzip>

Note

Compression is always a trade off - CPU cycles for compression and decompression operation are exchanged for less bytes transfered over the network.

How compression affects actual message bus operation durations and throughput is very much workload and deployment specific (spare CPU cycles, network setup, available bandwidth, is the NIC being saturated, is RabbitMQ hosted on the same server / AZ, etc).

StackStorm also supports RabbitMQ cluster.

In /etc/st2/st2.conf include the following section:

[messaging]
cluster_urls = amqp://#RMQ_USER:#RMQ_PASSWD@#RMQ_NODE_1:#RMQ_PORT/#RMQ_VHOST,
               amqp://#RMQ_USER:#RMQ_PASSWD@#RMQ_NODE_2:#RMQ_PORT/#RMQ_VHOST,
               amqp://#RMQ_USER:#RMQ_PASSWD@#RMQ_NODE_3:#RMQ_PORT/#RMQ_VHOST

To understand more about setting up a RabbitMQ cluster, see https://www.rabbitmq.com/clustering.html
RabbitMQ HA guide - https://www.rabbitmq.com/ha.html

Configure SSH

To run actions on remote hosts, StackStorm uses SSH. We recommend using public key-based based SSH access on all remote hosts.

The StackStorm ssh user and path to SSH key are set in /etc/st2/st2.conf. During installation,, the one-line install script configures ssh on the local box for the user stanley.

Follow these steps to configure a stanley user on remote sytems:

useradd stanley
mkdir -p /home/stanley/.ssh
chmod 0700 /home/stanley/.ssh

# generate ssh keys and copy over public key to remote box.
ssh-keygen -f /home/stanley/.ssh/stanley_rsa -P ""
cp ${KEY_LOCATION}/stanley_rsa.pub /home/stanley/.ssh/stanley_rsa.pub

# authorize key-based access.
cat /home/stanley/.ssh/stanley_rsa.pub >> /home/stanley/.ssh/authorized_keys
chmod 0600 /home/stanley/.ssh/authorized_keys
chown -R stanley:stanley /home/stanley
echo "stanley    ALL=(ALL)       NOPASSWD: SETENV: ALL" >> /etc/sudoers.d/st2

# ensure requiretty is not set to default in the /etc/sudoers file.
sudo sed -i -r "s/^Defaults\s+\+requiretty/# Defaults +requiretty/g" /etc/sudoers

To verify, run this from your StackStorm system:

# ssh should not require a password since the key is already provided
ssh -i /home/stanley/.ssh/stanley_rsa [email protected]

# make sure that no password is required
sudo su

SSH Troubleshooting

Validate that passwordless SSH configuration works for the destination. Assuming the default user stanley:
```
sudo ssh -i /home/stanley/.ssh/stanley_rsa -t [email protected] uname -a
```

Using SSH config

StackStorm allows loading an SSH config file local to the system user. This is a configurable option. To enable, add the following to /etc/st2/st2.conf

[ssh_runner]
use_ssh_config = True
...

SUDO Access

StackStorm’s shell actions - local-shell-cmd, local-shell-script, remote-shell-cmd, remote-shell-script- are performed by a special user. By default, this user is named stanley. This is configurable via st2.conf.

Note

the stanley user requires the following access:

Sudo access to all boxes on which the script action will run.
SETENV option needs to be set for all the commands. This way environment variables which are available to the local runner actions will also be available when the user executes local runner actions under a different user or with root privileges.
As some actions require sudo privileges, password-less sudo access to all boxes.

One way of setting up passwordless sudo is perform the below operation on each remote box:

echo "stanley    ALL=(ALL)       NOPASSWD: SETENV: ALL" >> /etc/sudoers.d/st2

Configure Logging

By default, the logs can be found in /var/log/st2.

With the standard logging setup you will see files like st2*.log and st2*.audit.log in the log folder.
Per-component logging configuration can be found in /etc/st2/logging.<component>.conf. Those files use Python logging configuration format. Log file location and other settings can be modified in these configuration files, e.g. to change the output to use syslog instead.
StackStorm ships with example configuration files to show how to use syslog - these are at /etc/st2/syslog.<component>.conf. To use them, edit /etc/st2/st2.conf, and change the logging = lines to point to the syslog configuration file. You can also see more instructions and example configurations at exchange-misc/syslog.
By default, log rotation is handled via logrotate. Default log rotation config (logrotate.conf) is included with all package-based installations. Note that handlers.RotatingFileHandler is used by default in /etc/st2/logging.*.conf, but the maxBytes and backupCount args are not specified so no rotation is performed by default which then lets logrotate handle the rotation. If you want Python services instead of logrotate to handle the log rotation, update the logging configs as shown below:
```
[handler_fileHandler]
class=handlers.RotatingFileHandler
level=DEBUG
formatter=verboseConsoleFormatter
args=("logs/st2api.log", "a", 100000000, 5)
```
In this case the log file will be rotated when it reaches 100000000 bytes (100MB) and a maximum of 5 old log files will be kept. For more information, see RotatingFileHandler docs.

Keep in mind that log level names need to be uppercase (e.g. DEBUG, INFO, etc.).

Sensors run in their own process so it is recommended to not allow sensors to share the same RotatingFileHandler. To configure a separate handler per sensor /etc/st2/logging.sensorcontainer.conf can be updated as follows, where MySensor is the sensor in the mypack pack that will have its own log file:

[loggers]
keys=root,MySensor

[handlers]
keys=consoleHandler, fileHandler, auditHandler, MySensorFileHandler, MySensorAuditHandler

[logger_MySensor]
level=INFO
handlers=consoleHandler, MySensorFileHandler, MySensorAuditHandler
propagate=0
qualname=st2.SensorWrapper.mypack.MySensor

[handler_MySensorFileHandler]
class=handlers.RotatingFileHandler
level=INFO
formatter=verboseConsoleFormatter
args=("logs/mysensor.log",)

[handler_vSphereEventSensorAuditHandler]
class=handlers.RotatingFileHandler
level=AUDIT
formatter=gelfFormatter
args=("logs/mysensor.audit.log",)

Check out LogStash configuration and Kibana dashboard for pretty logging and audit at exchange-misc/logstash

Configure The Number of Action Runner Workers

In CentOS/RockyLinux/RHEL the number of action workers defaults to 10. In Ubuntu the number of workers defaults to the number of CPU cores the machine has. You may wish to increase the number of workers in an HA setup or on system with plenty of resources.

The number of workers can be increased by modifying the environment variable WORKERS. To persist the number of st2actionrunner workers, create or edit the environment variable file for your distribution and add the number of workers, eg. 25: WORKERS=25. On RHEL/RockyLinux/CentOS we use the /etc/sysconfig/st2actionrunner file and on Ubuntu use the /etc/default/st2actionrunner file to pass custom environment variables to the st2actionrunner service/unit:

WORKERS=25

Authentication

Please refer to Authentication to learn details of authentication, integrations with various identity providers, and managing API tokens.

Configure ChatOps

StackStorm brings native two-way ChatOps support. To learn more about ChatOps, and how to configure it manually, please refer to Configuration section under ChatOps.

Configure secrets masking

In order to manage secrets masking on a system-wide basis you can also modify /etc/st2/st2.conf and control secrets masking at 2 levels i.e. API and logs. Note that this feature only controls external visibility of secrets and does not control how secrets are stored as well as managed by StackStorm.

To mask secrets in API response. This is enabled on a per API basis and only available to admin users.
```
[api]
...
mask_secrets = True
```
To mask secrets in logs:
```
[log]
...
mask_secrets = True
```

For more information and limitations on secrets masking please refer to Secrets Masking.