Traces

Traces are tracking entities that serve to gather all StackStorm entities like ActionExecution, TriggerInstance and Rule that originate from an event. In the StackStorm context an event could be one of the following:

  • Events from an external system sent to StackStorm via a Sensor or Webhook.

  • Action executed via UI, CLI or API.

  • Action executed via ChatOps.

Examples

Let’s walk through a few canonical examples:

External events

Sensors dispatch TriggerInstances into StackStorm and Webhooks are also translated to TriggerInstances when posted to StackStorm. Rules are written to match specific Triggers and compared against TriggerInstances.

In the canonical case, a TriggerInstance ti1 is dispatched by Sensor to StackStorm, matches a Rule r1 leading to an ActionExecution ae1. On completion of ae1 an ActionTrigger TriggerInstance ti2 is dispatched by StackStorm.

The trace created in this case contains all the entities from above, since they originate from the same event i.e. TriggerInstance ti1.

Trace
  |- ti1
  |- r1
  |- ae1
  |- ti2

Connected Flows

StackStorm raises an internal Trigger called the ActionTrigger. Rules can be used in conjunction with those, on completion of executions.

An ActionExecution ae1 is started by a user. On completion of ae1 an ActionTrigger TriggerInstance ti1 is dispatched. Rule r1 matches it, and leads to ActionExecution ae2. Another ActionTrigger TriggerInstance ti2 is dispatched but no rule is matched.

The trace created in this case contains all the entities from above since they cascade from the same origin i.e. ActionExecution ae1.

Trace
  |- ae1
  |- ti1
  |- r1
  |- ae2
  |- ti2

Tracing Triggers and Executions

Users can define their own identifying information for a Trace at event injection points. The injection points for StackStorm where a Trace can start are:

  • Dispatch a Trigger (more precisely this is dispatching a TriggerInstance) by a Sensor.

  • Webhook posted to StackStorm.

  • Execute an Action (aka creation of an ActionExecution) via UI, CLI, API or Chat.

What is trace_tag and trace_id?

  • trace-tag: User specified and therefore friendly way to tag or identify a Trace. There is no requirement for this value to be unique, and StackStorm will not enforce this. Whenever only a trace-tag is provided at one of the injection points a new Trace is started if one does not already exist.

  • trace-id: This is a StackStorm-defined value and is guaranteed to be unique. Users can specify this value at the injection points as well, but a Trace with the specified trace-id must already exist.

Dispatch a Trigger

TriggerInstance dispatch usually happens from a Sensor. The authoring a sensor page contains information on how to introduce a Trace.

A brief snippet is included here to explain some trace-specific constructs. A sensor would inject such triggers by using the sensor_service passed into the sensor on instantiation:

self.sensor_service.dispatch(trigger=trigger, payload=payload, trace_tag=trace_tag)

Here the Sensor is expected to supply a meaningful value for trace_tag e.g.:

  • Commit SHA of a git commit for a git commit hook trigger.

  • ID of the event from a monitoring system, e.g. Sensu or Nagios.

Webhook

Both custom webhooks and generic StackStorm webhooks support supplying a trace-tag via the optional header St2-Trace-Tag.

For a custom webhook, the equivalent curl command is:

curl -X POST http://127.0.0.1:9101/v1/webhooks/sample -H "X-Auth-Token: matoken" -H "Content-Type: application/json" -H "St2-Trace-Tag: webhook-1" --data '{"key1": "value1"}'

Execute an Action

Execution of an Action can also be associated with a Trace. Here’s how to do that via CLI:

  • To start a new trace use trace-tag:

    $ st2 run core.local date --trace-tag TraceDateAction

  • To associate with an existing trace use trace-id:

    $ st2 run core.local uname --trace-id 55d505fd32ed35711522c4c8

Viewing Traces

StackStorm CLI provides the ability to list and get traces.

List

  • All traces in the system:

    $ st2 trace list

  • Filter by trace-id:

    $ st2 trace list --trace-tag <trace-tag>

  • Filter by execution:

    $ st2 trace list --execution 55d505fd32ed35711522c4c7

  • Filter by rule:

    $ st2 trace list --rule 55d5064432ed35711522c4ca

  • Filter by trigger-instance:

    $ st2 trace list --trigger-instance 55d5069832ed35711cc4b08e

Get

  • Get a specific trace:

    $ st2 trace get <trace-id>

  • View the causation chain in a trace for an action execution. Similarly for rule and trigger-instance:

    $ st2 trace get <trace-id> -e

  • View specific type in a trace:

    $ st2 trace get <trace-id> --show-executions

  • Hide no-op trigger instances. These are trigger instances which do not lead to a rule enforcement:

    $ st2 trace get <trace-id> --hide-noop-triggers

Is Everything Traced?

By default all ActionExecutions and TriggerInstances are traced. If no trace-tag is provided by a user then StackStorm automatically generate a trace-tag to provide tracking.