Mistral¶
Note
Our new workflow engine Orquesta is now GA. We recommend writing all new workflows in Orquesta. Mistral is still supported, but will be removed in a future release.
Mistral is an OpenStack project that manages and executes workflows as a service. Mistral is automatically installed as a separate service named “mistral” along with StackStorm. A Mistral workflow can be defined as a StackStorm action in a Mistral workbook using the v2 Workflow Language.
Expression languages such as YAQL are used for formatting variables and condition evaluations. Starting with StackStorm v2.2, Jinja2 is also supported where YAQL expressions are accepted. Both workbook and workflow definitions are supported.
On action execution, StackStorm writes the definition to Mistral and executes the workflow. A workflow can invoke other StackStorm actions natively as subtasks. StackStorm handles the translations and calls transparently in Mistral and actively polls Mistral for execution results. StackStorm actions in the workflow can be traced back to the original parent action that invoked the workflow.
Essential Mistral Links:
Note
The following tools and resources are not owned by StackStorm. Please use at your own risk.
Mistral workflow definition language, aka v2 WorkFlow Language
Jinja2 template engine documentation and Jinja2 online evaluator
Note
Workflow examples in this documentation use YAQL expressions unless otherwise stated.
Basic Workflow¶
Similarly to ActionChains, Mistral workflows have an action metadata file in
/opt/stackstorm/packs/<mypack>/actions, and the workflow definition itself in
/opt/stackstorm/packs/<mypack>/actions/workflows.
Let’s start with a very basic workflow that calls a StackStorm action and notifies StackStorm when the workflow is done. The files used in this example are also located under /usr/share/doc/st2/examples if StackStorm is already installed (see also deploy examples).
The first task is named run-cmd. It executes a shell command on the server where StackStorm is
installed. A task can reference any registered StackStorm action directly. In this example, the
run-cmd task calls core.local and passing the cmd as input. core.local is an action
that comes installed with StackStorm. When the workflow is invoked, StackStorm will translate the workflow
definition appropriately before sending it to Mistral. Let’s save this as
/opt/stackstorm/packs/examples/actions/workflows/mistral-basic.yaml on our StackStorm server.
version: '2.0'
examples.mistral-basic:
description: A basic workflow that runs an arbitrary linux command.
type: direct
input:
- cmd
- timeout
output:
stdout: <% $.stdout %>
tasks:
task1:
action: core.local cmd=<% $.cmd %> timeout=<% $.timeout %>
publish:
stdout: <% task(task1).result.stdout %>
stderr: <% task(task1).result.stderr %>
This is the corresponding StackStorm action metadata for the example above. The StackStorm pack for this
workflow action is named “examples”. Note that the workflow is named fully qualified as
<pack>.<action> in the definition above. The StackStorm action runner is mistral-v2. The entry
point for the StackStorm action refers to the YAML file of the workflow definition. Let’s save this
metadata as /opt/stackstorm/packs/examples/actions/mistral-basic.yaml:
---
description: Run a local linux command
enabled: true
runner_type: mistral-v2
entry_point: workflows/mistral-basic.yaml
name: mistral-basic
pack: examples
parameters:
cmd:
required: true
type: string
timeout:
type: integer
default: 60
The following table list optional parameters that can be defined in the workflow action. In the example, these optional parameters are set to immutable. It is good practice to set them to immutable even if they are empty since these are Mistral-specific parameters for the workflow author.
options |
description |
|---|---|
workflow |
If definition is a workbook containing many workflows, this specifies the main workflow to execute. |
task |
If the type of workflow is “reverse”, this specifies the task to invoke. |
context |
A dictionary containing additional workflow start up parameters. |
Next, run st2 action create /opt/stackstorm/packs/examples/actions/mistral-basic.yaml to
create this workflow action. This will register the workflow as examples.mistral-basic in
StackStorm. To execute the workflow, run st2 run examples.mistral-basic cmd=date -a where -a
tells the command to return and not wait for the workflow to complete.
If the workflow completed successfully, both the workflow examples.mistral-basic and the
action core.local should have a succeeded status in the StackStorm action execution list. By
default, st2 execution list only returns top level executions. This means subtasks are not
displayed.
+--------------------------+--------------+--------------+-----------+-----------------+---------------+
| id | action.ref | context.user | status | start_timestamp | end_timestamp |
+--------------------------+--------------+--------------+-----------+-----------------+---------------+
| 54ee54c61e2e24152b769a47 | examples | stanley | succeeded | Wed, 25 Feb | Wed, 25 Feb |
| | .mistral- | | | 2015 23:03:34 | 2015 23:03:34 |
| | basic | | | UTC | UTC |
+--------------------------+--------------+--------------+-----------+-----------------+---------------+
To display subtasks, run st2 execution get <execution-id> --show-tasks:
+--------------------------+------------+--------------+-----------+------------------------------+------------------------------+
| id | action.ref | context.user | status | start_timestamp | end_timestamp |
+--------------------------+------------+--------------+-----------+------------------------------+------------------------------+
| 54ee54c91e2e24152b769a49 | core.local | stanley | succeeded | Wed, 25 Feb 2015 23:03:37 | Wed, 25 Feb 2015 23:03:37 |
| | | | | UTC | UTC |
+--------------------------+------------+--------------+-----------+------------------------------+------------------------------+
The following is a simple extension of the previous workflow definition. In this example, we have
a second task named “task2”. It might be natural to think that “task2” will be executed after
“task1”, i.e, in sequential order. However, when no tasks attributes like on-complete,
on-success and on-error are defined, tasks are run in parallel. This is possible with
Mistral because it provides a join flow control which allows us to synchronize multiple parallel
workflow branches and aggregate their data.
version: '2.0'
examples.mistral-basic-two-tasks-with-notifications:
description: A basic workflow that runs two Linux commands (one in each task).
type: direct
output:
stdout: <% $.stdout %>
tasks:
task1:
action: core.local cmd="echo task1"
publish:
stdout: <% task(task1).result.stdout %>
stderr: <% task(task1).result.stderr %>
task2:
action: core.local cmd="echo task2"
publish:
stdout: <% task(task2).result.stdout %>
stderr: <% task(task2).result.stderr %>
Publishing Variables¶
A Mistral task can publish results from a task as variables that can be consumed in other tasks:
tasks:
get_hostname:
action: core.local
input:
cmd: "hostname"
publish:
hostname: <% task(get_hostname).result.stdout %>
In the above example, get_hostname is a core.local action which runs the command
hostname. The core.local action produces output consisting of the fields stdout,
stderr, exit_code etc.
We just want to publish the variable stdout from it, for the rest of tasks to consume. To
reference the result of the task, use the task function, which returns a dictionary containing
attributes for the task such as id, state, result, and additional info.
Another example is shown below:
tasks:
create_new_node:
action: rackspace.create_vm
input:
name: <% $.hostname %>
flavor_id: <% $.vm_size_id %>
image_id: <% $.vm_image_id %>
key_material: <% $.ssh_pub_key %>
metadata:
asg: <% $.asg %>
publish:
ipv4_address: '<% task(create_new_node).result.result.public_ips[1] %>'
ipv6_address: '<% task(create_new_node).result.result.public_ips[0] %>'
In the above example, the action rackspace.create_vm is a Python action that produces a result
object. We just want to publish the IP addresses from the public_ips list field from the
result object.
Please note that result.result is not a typo. The Python action posts output to a key named
result for the st2 action execution and the Mistral task function puts the result of the Python
action in result of its output dictionary.
Such published variables are accessible as input parameters to other tasks in the workflow. An
example of using ipv4_address from the above example in another task is shown below:
tasks:
# ... <snap>
setup_ipv4_dns:
action: rackspace.create_dns_record
wait-before: 1 # delay, in seconds
input:
name: '<% $.hostname %>.<% $.asg %>.<% $.domain %>'
zone_id: <% $.dns_zone_id %>
type: 'A'
data: <% $.ipv4_address %>
# .... </snap>
Stitching Together a More Complex Workflow¶
The following is a mock up of a more complex workflow. In this mock up running simple printf
and sleep commands, the workflow demonstrates nested workflows, fork, and join:
version: "2.0"
name: examples.mistral-workbook-complex
description: A sample workflow that demonstrates nested workflows, forks, and join.
workflows:
main:
type: direct
input:
- vm_name
- cpu_cores
- memory_mb
output:
vm_id: <% $.vm_id %>
ip: <% $.ip %>
tasks:
register_dns:
action: core.local
input:
cmd: "sleep 1; printf 'Registering <% $.vm_name %>...'"
publish:
ip: "10.1.23.99"
status_message: "DNS for <% $.vm_name %> is registered."
on-success:
- configure_vm
- notify
create_vm:
wait-before: 1
workflow: create_vm
input:
name: <% $.vm_name %>
cpu_cores: <% $.cpu_cores %>
memory_mb: <% $.memory_mb %>
publish:
vm_id: <% task(create_vm).result.vm_id %>
status_message: "VM <% $.vm_name %> is created."
on-success:
- configure_vm
- notify
configure_vm:
join: all
workflow: configure_vm
input:
vm_id: <% $.vm_id %>
ip: <% $.ip %>
publish:
status_message: "VM <% $.vm_name %> is reconfigured."
on-success:
- close_request
- notify
close_request:
action: std.noop
publish:
status_message: "VM request is fulfilled."
on-success:
- notify
notify:
action: core.local
input:
cmd: "printf '<% $.status_message %>'"
create_vm:
type: direct
input:
- name
- cpu_cores
- memory_mb
output:
vm_id: <% $.vm_id %>
tasks:
create:
action: core.local
input:
cmd: "printf 'vm1234'; sleep 5"
publish:
vm_id: <% task(create).result.stdout %>
configure_vm:
type: direct
input:
- vm_id
- ip
tasks:
add_disks:
action: core.local
input:
cmd: "sleep 1; printf 'disks created'"
add_nics:
action: core.local
input:
cmd: "sleep 1; printf 'nics created'"
install_apps:
action: core.local
input:
cmd: "sleep 1; printf 'apps installed'"
Since there are multiple workflows defined in this workbook, the workflow author has to specify which workflow to execute in the metadata:
---
description: Run a series of simulated actions.
enabled: true
entry_point: workflows/mistral-workbook-complex.yaml
name: mistral-workbook-complex
pack: examples
parameters:
cpu_cores:
default: 1
type: integer
memory_mb:
default: 1024
type: integer
vm_name:
required: true
type: string
workflow:
default: examples.mistral-workbook-complex.main
immutable: true
type: string
runner_type: mistral-v2
To test out this workflow, save the metadata file to /opt/stackstorm/packs/examples/actions/
and the workflow file to /opt/stackstorm/packs/examples/actions/workflows. Run
st2 action create /opt/stackstorm/packs/examples/actions/mistral-workbook-complex.yaml to
create the action and run st2 run examples.mistral-workbook-complex vm_name="vmtest1" -a to
test.
Validation¶
The Mistral CLI includes tools for performing high-level sanity checks of Mistral workflow YAML files:
# Validate a workflow
mistral workflow-validate /path/to/workflow.yaml
# Validate a workbook
mistral workbook-validate /path/to/workbook.yaml
Note
These sanity checks simply provide a test against the Mistral DSL schema. They do NOT test YAQL or Jinja2 expressions.
More Examples¶
There are more workflow examples under /usr/share/doc/st2/examples. These include error handling, repeat, and retries.
Check out this step-by-step tutorial on building a workflow in StackStorm https://stackstorm.com/2015/07/08/automating-with-mistral-workflow/
More details about Mistral can be found at https://docs.openstack.org/mistral/latest/.
More Topics¶
The following sections go into more details on specific topics.