Mistral is an OpenStack project that manages and executes workflows as a service. Mistral is installed as a separate service named “mistral” along with StackStorm. A Mistral workflow can be defined as a StackStorm action in a Mistral workbook using the v2 DSL. Expression languages such as YAQL are used for formatting variables and condition evaluations. Starting in StackStorm v2.2, Jinja2 is also supported where YAQL expressions are accepted. Both workbook and workflow definitions are supported. On action execution, StackStorm writes the definition to Mistral and executes the workflow. A workflow can invoke other StackStorm actions natively as subtasks. StackStorm handles the translations and calls transparently in Mistral and actively polls Mistral for execution results. StackStorm actions in the workflow can be traced back to the original parent action that invoked the workflow.
Essential Mistral Links:
- Mistral workflow definition language, aka v2 DSL
- YAQL documentation and YAQL online evaluator
- Jinja2 template engine
Workflow examples in this documentation will use YAQL expressions unless otherwise stated.
Let’s start with a very basic workflow that calls a StackStorm action and notifies StackStorm when the workflow is done. The files used in this example are also located under /usr/share/doc/st2/examples if StackStorm is already installed (and you can deploy examples). The first task is named run-cmd that executes a shell command on the local server where st2 is installed. A task can reference any registered StackStorm action directly. In this example, the run-cmd task is calling core.local and passing the cmd as input. core.local is an action that comes installed with StackStorm. When the workflow is invoked, StackStorm will translate the workflow definition appropriately before sending it to Mistral. Let’s save this as mistral-basic.yaml at /opt/stackstorm/packs/examples/actions/workflows where StackStorm is installed.
version: '2.0' examples.mistral-basic: description: A basic workflow that runs an arbitrary linux command. type: direct input: - cmd output: stdout: <% $.stdout %> tasks: task1: action: core.local cmd=<% $.cmd %> publish: stdout: <% task(task1).result.stdout %> stderr: <% task(task1).result.stderr %>
The following is the corresponding StackStorm action metadata for example above. The StackStorm pack for this workflow action is named “examples”. Please note that the workflow is named fully qualified as “<pack>.<action>” in the definition above. The StackStorm action runner is “mistral-v2”. The entry point for the StackStorm action refers to the YAML file of the workflow definition. Let’s save this metadata as mistral-basic.yaml at /opt/stackstorm/packs/examples/actions/.
--- description: Run a local linux command enabled: true entry_point: workflows/mistral-basic.yaml name: mistral-basic pack: examples parameters: cmd: required: true type: string runner_type: mistral-v2
The following table list optional parameters that can be defined in the workflow action. In the example, these optional parameters are set to immutable. It is good practice to set them to immutable even if they are empty since these are Mistral specific parameters for the workflow author.
|workflow||If definition is a workbook containing many workflows, this specifies the main workflow to execute.|
|task||If the type of workflow is “reverse”, this specifies the task to invoke.|
|context||A dictionary containing additional workflow start up parameters.|
st2 action create /opt/stackstorm/packs/examples/actions/mistral-basic.yaml to create this workflow action. This will register the workflow as examples.mistral-basic in StackStorm. Then to execute the workflow, run
st2 run examples.mistral-basic cmd=date -a where
-a tells the command to return and not wait for the workflow to complete. If the workflow completed successfully, both the workflow examples.mistral-basic and the action core.local would have a succeeded status in the StackStorm action execution list. By default,
st2 execution list only returns top level executions. This means subtasks are not displayed.
|54ee54c61e2e24152b769a47||examples .mistral- basic||stanley||succeeded||Wed, 25 Feb 2015 23:03:34 UTC||Wed, 25 Feb 2015 23:03:34 UTC|
To display subtasks, run
st2 execution get <execution-id> --tasks.
|54ee54c91e2e24152b769a49||core.local||stanley||succeeded||Wed, 25 Feb 2015 23:03:37 UTC||Wed, 25 Feb 2015 23:03:37 UTC|
The following is a simple extension of the previous workflow definition. In this example, we have a second task named task2. It might be natural to think that task2 will be executed after task1, i.e, in sequential order. However, when no tasks attributes like
on-error are defined, tasks are run in parallel. This is possible with mistral because it provides a join flow control which allows to synchronize multiple parallel workflow branches and aggregate their data.
version: '2.0' examples.mistral-basic-two-tasks-with-notifications: description: A basic workflow that runs two Linux commands (one in each task). type: direct output: stdout: <% $.stdout %> tasks: task1: action: core.local cmd="echo task1" publish: stdout: <% task(task1).result.stdout %> stderr: <% task(task1).result.stderr %> task2: action: core.local cmd="echo task2" publish: stdout: <% task(task2).result.stdout %> stderr: <% task(task2).result.stderr %>
Canceling Workflow Execution¶
An execution of a Mistral workflow can be cancelled by running
st2 execution cancel <execution-id>. Workflow tasks that are still running will not be canceled and will run to completion. No new tasks for the workflow will be scheduled.
Rerunning Workflow Execution¶
An execution of a Mistral workflow can be rerun on error. The execution either can be rerun from the
beginning or from the task(s) that failed. The latter is useful for long running workflows with
temporary service or network outages. Rerunning the workflow execution from the beginning is
exactly like rerunning any StackStorm execution with the command
st2 execution re-run <execution-id>.
The rerun is a completely separate execution with a new execution ID in both StackStorm and Mistral.
Rerunning the workflow from where it errored is slightly different. To retain context, the original
workflow execution is reuse in Mistral but a new StackStorm execution will be created to stay consistent
in StackStorm. The re-run command has a new
--tasks option that takes a list of task names to re-run.
For example, given a workflow that fails at task3 and task4 on separate parallel branches, the
st2 execution re-run <execution-id> --tasks task3 task4 will resume the Mistral workflow
execution and rerun both task3 and task4 using original inputs. Both the workflow and task execution
in Mistral has to be in an
errored state for rerun.
If using a Mistral workbook, task of subworkflows can also be rerun. For example, if the main
workflow has a task1 that calls subflow1, then to rerun subtask1 of subflow1, the syntax for the st2
execution re-run command would be
st2 execution re-run <execution-id> --tasks task1.subtask1.
If the task to rerun is a with-items task, there is an option to rerun only failed iterations. For
example, task1 is a with-items task with 5 items. Let’s say 2 of the items failed. By specifying
st2 execution re-run --tasks task1 task2 --no-reset task1 option, task1 will only re-run
the 2 items that failed. If the
--no-reset option is not provided, then all 5 items will be
Rerunning workflow execution from the task(s) that failed is currently an experimental feature and subject to bug(s) and change(s). Please also note that rerunning a subtask nested in another StackStorm action is not currently supported.
Publishing variables in Mistral workflows¶
A Mistral task can publish results from a task as variables that can be consumed in other tasks.
A simple example is shown below:
tasks: get_hostname: action: core.local input: cmd: "hostname" publish: hostname: <% task(get_hostname).result.stdout %>
In the above example, get_hostname is a core.local action which runs the command
core.local action produces an output consisting of fields
We just want to publish the variable
stdout from it so rest of tasks can consume. To reference
the result of the task, use the task function, which returns a dictionary containing attributes for
the task such as id, state, result, and additional info.
Another example is shown below:
tasks: create_new_node: action: rackspace.create_vm input: name: <% $.hostname %> flavor_id: <% $.vm_size_id %> image_id: <% $.vm_image_id %> key_material: <% $.ssh_pub_key %> metadata: asg: <% $.asg %> publish: ipv4_address: '<% task(create_new_node).result.result.public_ips %>' ipv6_address: '<% task(create_new_node).result.result.public_ips %>'
In the above example, action rackspace.create_vm is a Python action that produces a result object.
We just want to publish the IP addresses from
public_ips list field from the result object.
Please note that
result.result is not a typo. The Python action posts output to a key named
result for the st2 action execution and the Mistral task function puts the result of the Python
result of its output dictionary.
Such published variables are accessible as input parameters to other tasks in the workflow. An
example of using
ipv4_address from the above example in another task is shown below:
tasks: # ... <snap> setup_ipv4_dns: action: rackspace.create_dns_record wait-before: 1 # delay, in seconds input: name: '<% $.hostname %>.<% $.asg %>.<% $.domain %>' zone_id: <% $.dns_zone_id %> type: 'A' data: <% $.ipv4_address %> # .... </snap>
Stitching together a more Complex Workflow¶
The following is a mock up of a more complex workflow. In this mock up running simple printf and sleep commands, the workflow demonstrates nested workflows, fork, and join.
version: "2.0" name: examples.mistral-workbook-complex description: A sample workflow that demonstrates nested workflows, forks, and join. workflows: main: type: direct input: - vm_name - cpu_cores - memory_mb output: vm_id: <% $.vm_id %> ip: <% $.ip %> tasks: register_dns: action: core.local input: cmd: "sleep 1; printf 'Registering <% $.vm_name %>...'" publish: ip: "10.1.23.99" status_message: "DNS for <% $.vm_name %> is registered." on-success: - configure_vm - notify create_vm: wait-before: 1 workflow: create_vm input: name: <% $.vm_name %> cpu_cores: <% $.cpu_cores %> memory_mb: <% $.memory_mb %> publish: vm_id: <% task(create_vm).result.vm_id %> status_message: "VM <% $.vm_name %> is created." on-success: - configure_vm - notify configure_vm: join: all workflow: configure_vm input: vm_id: <% $.vm_id %> ip: <% $.ip %> publish: status_message: "VM <% $.vm_name %> is reconfigured." on-success: - close_request - notify close_request: action: std.noop publish: status_message: "VM request is fulfilled." on-success: - notify notify: action: core.local input: cmd: "printf '<% $.status_message %>'" create_vm: type: direct input: - name - cpu_cores - memory_mb output: vm_id: <% $.vm_id %> tasks: create: action: core.local input: cmd: "printf 'vm1234'; sleep 5" publish: vm_id: <% task(create).result.stdout %> configure_vm: type: direct input: - vm_id - ip tasks: add_disks: action: core.local input: cmd: "sleep 1; printf 'disks created'" add_nics: action: core.local input: cmd: "sleep 1; printf 'nics created'" install_apps: action: core.local input: cmd: "sleep 1; printf 'apps installed'"
Since there are multiple workflows defined in this workbook, the workflow author has to specify which workflow to execute in the metadata as shown in the workflow parameters below.
--- description: Run a series of simulated actions. enabled: true entry_point: workflows/mistral-workbook-complex.yaml name: mistral-workbook-complex pack: examples parameters: cpu_cores: default: 1 type: integer memory_mb: default: 1024 type: integer vm_name: required: true type: string workflow: default: examples.mistral-workbook-complex.main immutable: true type: string runner_type: mistral-v2
To test out this workflow, save the metadata file to /opt/stackstorm/packs/examples/actions/ and the workflow file to /opt/stackstorm/packs/examples/actions/workflows. Run
st2 action create /opt/stackstorm/packs/examples/actions/mistral-workbook-complex.yaml to create the action and run
st2 run examples.mistral-workbook-complex vm_name="vmtest1" -a to test.
There are more workflow examples under /usr/share/doc/st2/examples such as error handling, repeat, and retries.
Check out this step-by-step tutorial on building a workflow in StackStorm http://stackstorm.com/2015/07/08/automating-with-mistral-workflow/
More details about Mistral can be found at http://docs.openstack.org/developer/mistral/.