Traces are tracking entities that serve to gather all StackStorm entities like ActionExecution, TriggerInstance and Rule that originate from an event. In the StackStorm context an event could be one of the following:

  • Events from an external system sent to StackStorm via a Sensor or Webhook.
  • Action executed via UI, CLI or API.
  • Action executed via ChatOps.


Let us walk through a few canonical examples:

External events

Sensors dispatch TriggerInstances into StackStorm and Webhooks are also translated to TriggerInstances when posted to StackStorm. Rules are written to match specific Triggers and compared against TriggerInstances.

In the canonical case, TriggerInstance(ti1) dispatched by Sensor to StackStorm, matches a Rule(r1) leading to an ActionExecution(ae1). On completion of ae1 an ActionTrigger TriggerInstance(ti2) is dispatched by StackStorm.

The trace created in this case contains all the entities from above since they originate from the same event i.e. TriggerInstance(ti1) dispatched into StackStorm.

  |- ti1
  |- r1
  |- ae1
  |- ti2

Connected flows

StackStorm raises an internal Trigger called the ActionTrigger. It is possible for rules to be used in conjunction with those on completion of executions.

ActionExecution(ae1) started by user, on completion of ae1 an ActionTrigger TriggerInstance(ti1) is dispatched, Rule(r1) matches and leads to ActionExecution(ae2) another ActionTrigger TriggerInstance(ti2) is dispatched but no rule matched.

The trace created in this case contains all the entities from above since they cascade from the same origin i.e. ActionExecution(ae1) dispatched into the system.

  |- ae1
  |- ti1
  |- r1
  |- ae2
  |- ti2

Tracing Triggers and Executions

It is possible for users to define identifying information for a Trace at event injection points. The injection points for StackStorm where a Trace can start are:

  • Dispatch a Trigger (more precisely this is dispatching a TriggerInstance) by a Sensor.
  • Webhook posted to StackStorm.
  • Execute an Action (aka creation of an ActionExecution) via UI, CLI, API or Chat.

What is a trace_tag and trace_id?

  • trace-tag : User specified and therefore friendly way to tag or identify a Trace. There is no requirement for this value to be unique and StackStorm will not enforce this either. Whenever only a trace-tag is provided at one of the injection points a new Trace is started if one does not already exist.
  • trace-id : This is a StackStorm defined value and is guaranteed to be unique. Users can specify this value at the injection points as all but a Trace with the specified trace-id must already exist.

Dispatch a Trigger

TriggerInstance dispatch most often happens from a Sensor. The authoring a sensor page contains information on how to introduce a Trace.

A brief snippet is included here to explain some trace specific constructs. A sensor would inject such triggers by using the sensor_service passed into the sensor on instantiation:

self.sensor_service.dispatch(trigger=trigger, payload=payload, trace_tag=trace_tag)

Here the Sensor is expected to supply a meaningful value for trace_tag e.g.:

  • Commit SHA of a git commit for a git commit hook trigger.
  • ID of the event from a monitoring system, like Sensu or Nagios, that is relayed to StackStorm.


Both custom webhooks and generic StackStorm webhooks support supplying a trace-tag via a header.

  • Header : St2-Trace-Tag

In case of a custom webhook the curl command will be

curl -X POST -H "X-Auth-Token: matoken" -H "Content-Type: application/json" -H "St2-Trace-Tag: webhook-1" --data '{"key1": "value1"}'

Execute an Action

Execution of an Action can also be associated with a Trace. Here is how this could be done from the CLI:

To start a new trace use trace-tag:

$ st2 run core.local date --trace-tag TraceDateAction

To associate with an existing trace use trace-id:

$ st2 run core.local uname --trace-id 55d505fd32ed35711522c4c8

Viewing Traces

StackStorm CLI provides the ability to list and get traces.


  • All traces in the system:
$ st2 trace list
  • Filter by trace-id:
$ st2 trace list --trace-tag <trace-tag>
  • Filter by execution:
$ st2 trace list --execution 55d505fd32ed35711522c4c7
  • Filter by rule:
$ st2 trace list --rule 55d5064432ed35711522c4ca
  • Filter by trigger-instance:
$ st2 trace list --trigger-instance 55d5069832ed35711cc4b08e


  • Get a specific trace:
$ st2 trace get <trace-id>
  • View the causation chain in a trace for an action execution. Similarly for rule and trigger-instance:
$ st2 trace get <trace-id> -e
  • View specific type in a trace:
$ st2 trace get <trace-id> --show-executions
  • Hide no-op trigger instances. These are trigger instances which do not lead to a rule enforcement:
$ st2 trace get <trace-id> --hide-noop-triggers

Is everything traced?

By default all ActionExecutions and TriggerInstances are traced. If no trace-tag is provided by a user then StackStorm automatically generate a trace-tag to provide tracking.