Building Pipelines

Learn about building pipelines using Sophos Factory.

Step Modules

Sophos Factory provides a drag-and-drop pipeline builder that accelerates the process of creating pipelines.

The action performed at each step of a pipeline is determined by a step module. You can browse or search for step modules in the left sidebar of the pipeline builder. Drag them into the design area to add steps to a pipeline.

For a guided tutorial, see Your First Pipeline.

Each step module has its own settings and properties. To see and edit these fields, drag a step into the design area and then select it.

For a full list of step modules and their properties, see the step module reference.

Pipelines Can Be Steps Too

In addition to the built-in step modules, a pipeline can include another pipeline as one of its steps. For each included pipeline, a new scope is created. The variables from the inner pipeline are exposed to the outer pipeline as step properties, which facilitates passing data from one pipeline to the next.

Reusable component pipelines are a powerful tool for creating pipelines that act as reusable modules.

Using Dynamic Values in Steps

Step properties and some fields can contain expressions. Expressions can be used to compute values dynamically at runtime, as well as read and transform data from previous steps.

In order to evaluate expressions, the code needs to be delimited with curly braces and pipes, like this:

{| (vars.jsonString | parse_json).description |}

Some fields only support expressions, and will have the delimiters wrapping the field, like this:

Other fields can support both Inline Mode (which is just plain text) or Expression Mode. Those fields have an expression toggle button on the right side. Toggling to Expression Mode switches the input to a code editor where an expression can be used.

For more information about expressions, see the expression reference.

Pipeline Variables

Pipeline variables are inputs that the pipeline receives at runtime.

To add an input variable to a pipeline, open the pipeline builder and select Variables at the top right. Click the Add Variable button.

The variable type, a display name, and a unique key must be specified.

The description of the variable will be displayed to your pipeline’s users in two places:

  1. In the Run Pipeline form, where users may edit the variable value.
  2. In the Create Job form, when creating a job for the pipeline.

Pipeline variables may be marked as required, which means that a value must be supplied when running the pipeline.

Pipeline variables may also be marked as hidden, which has two effects:

  1. The variables are not displayed as form fields in run dialogs. Users running your pipeline will not see these variables.
  2. When a run is created from Sophos Factory or via the API, hidden variable values are merged into the final variables for the run.

This allows you to provide default values for variables that don’t need to be provided when the pipline is run.

Accessing Variables From Steps

To access a variable from an expression, assuming the variable key is hostname, use vars.hostname.

The vars.<var key> syntax is available from any expression, and contains all variables available in the current scope. For more about variables and scopes, read Core Concepts: Variables.

You cannot access parent variables directly from an included pipeline. Instead, you must define input variables for the child pipeline, and pass values through from the parent pipeline’s step properties.

Passing Data Between Steps

Expressions evaluating within step properties have access to all other steps in the pipeline. To get another step, use steps.<step id>. This expression evaluates to an object.

All steps have a common set of fields, such as name, tags, and properties. The properties fields differs for each step module. Properties fields for each step module can be seen in the Step Module Reference.

For example, to retrieve the method of an HTTP Request step module with the ID http, we can use the this expression:

steps.http.properties.method

After a step executes, it has an additional result field that contains data about the execution result. The fields available on the result object differ for each step module, however, all step modules will have a status field that indicates whether the step execution succeeded or failed.

For example, the following expression evaluates to true if the step with ID http succeeded:

steps.http.result.status == 'Succeeded'

Reusable component pipeline steps have one additional field on their result object containing the evaluated outputs of the pipeline. For example, to retrieve an output with key ipaddress on an included pipeline step with the ID my_include, use expression:

steps.my_include.result.outputs.ipaddress

Accessing Credentials

Credentials can be retrieved dynamically by expressions. This is useful in certain cases, for example, when using the HTTP Request module with custom Authorization token headers.

To get credential data, pass the credential ID to the credential() helper function, like this:

credential('my_cred_id')

This expression evaluates to an object containing one or more data fields associated with the credential type. For example, to retrieve the password field of a Username/Password type of credential, use the following expression:

credential('my_cred_id').password

Because credentials are also a built-in variable type, it’s common to create a top-level variable in a pipeline and then pull the credential data out from an expression, which would look like this:

credential(vars.my_cred_var).password

Run Steps Conditionally

All steps have a condition field which must be an expression. If this field evaluates to false, then the step will be skipped.

Step conditions combined with reusable component pipelines can be very powerful for advanced pipeline control flow. When an included pipeline step is skipped, all steps within that pipeline are also skipped.

After a step executes, a result field is added. This contains data about the execution result. Using the result field, subsequent steps can be conditionally run based on whether the step succeeded or failed. Other result fields are also available, such as the stdout and stderr of script steps.

Ignoring Step Errors

Errors in step execution can be ignored by setting the ignore_errors field to true.

Step Looping

Most step modules and pipeline include steps support looping which will execute the step multiple times in a “loop”. There are three types of loops – each, while and until. Only one type of looping can be selected for a step.

The loop Context Variable

In a step loop, a special context variable is created for each iteration. The name of this variable is loop. The loop context variable has the following structure:

{
    "index": Number,
    "prev_result": Object
}
  • index is a zero-based counter specifying the current loop iteration. The value is always zero on the first iteration
  • prev_result is an object containing the value of the step result field after the previous iteration completed. prev_result is not available during the first iteration of the step.

NOTE: In previous versions of Factory, the context variable for an each loop was called each. In order to support existing pipelines still using each looping, the each variable is still supported as a synonym for loop. However, the context variable each has been deprecated and will be removed in a future version of the product. It is recommended that existing pipelines be updated to use loop as the context variable name.

each Loops

To use an each loop, the each field of the step must contain an expression that evaluates to an array. For each element of the array, the step is executed, and the loop context variable is available. In an each loop, the loop context variable documented above has two additional fields which gives it the following structure:

{
  "index": Number,
  "item": Any,
  "items": [Any],
  "prev_result": {
    "status": String, // Succeeded | Skipped | Failed
    "outputs": Object // See outputs defined for step module or child pipeline
  }
}
  • index - as noted above. Note that for each loops, index also points to the location of item in items.
  • prev_result - as noted above.
  • item - the current item at items[index].
  • items - a read-only reference to the array created by the each field expression.

Here is a simple example of the loop context variable’s value. Suppose the step each field evaluates to an array – ['a', 'b', 'c'] and the first iteration of the loop succeeds producing one output, myOutput with the value myValue. On the second iteration, the loop context variable will contain the following values:

{
  "index": 1,
  "prev_result": {
    "status": "Succeeded",
    "outputs": [
      { "key": "myOutput", "value": "myValue" }
    ]
  },
  "item": "b",
  "items": [ "a", "b", "c" ]
}

while Loops

To use a while loop, the while field of the step must contain an expression that evaluates to a boolean value. Before each iteration, the expression is evaluated. If the result is true, the iteration is executed. If the result is false, no further iterations are executed.

NOTE: If the while expression evaluates to false before the first iteration, no iterations will be executed.

For each iteration, the loop context variable is available and contains the fields noted in The loop Context Variable above.

Here is a simple example of the loop context variable’s value. Suppose the step while field contains the expression (loop.index | default(0)) < 2 and the first iteration of the loop succeeds producing one output, myOutput with the value myValue. On the second iteration, the loop context variable will contain the following values:

{
  "index": 1,
  "prev_result": {
    "status": "Succeeded",
    "outputs": [
      { "key": "myOutput", "value": "myValue" }
    ]
  }
}

until Loops

To use an until loop, the until field of the step must contain an expression that evaluates to a boolean value. After each iteration has exectued, the expression is evaluated. If the result is false, the next iteration is executed. If the result is true, no further iterations are executed.

NOTE: An until step loop always executes the first iteration and then checks the until condition for the first time.

For each iteration, the loop context variable is available and contains the fields noted in The loop Context Variable above.

Here is a simple example of the loop context variable’s value. Suppose the step until field contains the expression (loop.index | default(0)) > 2 and the first iteration of the loop succeeds producing one output, myOutput with the value myValue. On the second iteration, the loop context variable will contain the following values:

{
  "index": 1,
  "prev_result": {
    "status": "Succeeded",
    "outputs": [
      { "key": "myOutput", "value": "myValue" }
    ]
  }
}

while vs. until

Step loops using while and until might seem interchangeable, but there are some important differences.

Question while until
When is the condition evaluated? Before each iteration, including the first After each iteration, including the first
What is the minimum number of iterations? If the condition is false before the first iteration, no iterations will be executed The first iteration will always execute regardless of the condition value
When does iteration stop? When the condition becomes false When the condition becomes true

Step Results in Loops

Each iteration of a step loop is executed serially. After all iterations have been executed, the step will contain two fields with results.

  • result (singular) - an object containing the result of the last iteration of the loop.
  • results (plural) - an array which contains the result object for each iteration of the step execution. The results in this array will be in the same order that the iterations were executed. For an each loop, this means that the results entries will be in the same order as their corresponding entries in the each array.

Error Handling in Loops

The ignore_errors field affects the entire step, not individual iterations. Thus, if an error occurs in any iteration of a step loop, the step loop will be halted and no more iterations will execute, regardless of the ignore_errors setting. As with any step, pipeline execution may continue if the step’s ignore_errors field is true.

Creating Step Loops in the Builder

To configure a step loop from the pipeline builder, see the Looping Settings section of any step that supports loops.

each Loop Settings

while Loop Settings

until Loop Settings

Restoring Previous Revisions

When you save a pipeline, a new revision is created. Revisions are a linear history, and each revision is given a number. The pipeline revision history allows you to go back in time and restore previous states of a pipeline, as long as the pipeline has not been deleted. The revision history is limited to the pipeline steps and does not include changes to the pipeline name or image.

To restore an old version of a pipeline, open the pipeline in the builder and click the Revisions tab in the right sidebar. Find your desired revision and open it. Then save the pipeline again, bringing that revision to the top of the history.

It’s also sometimes useful to clear a pipeline’s revision history. To do this, clone the pipeline into the same project and then delete the old pipeline.

Installing Tools for Step Modules

Many built-in step modules require an underlying tool, such as a programming language runtime or a CLI program. Supported tools can be automatically installed at a specified version during pipeline execution.

There are two ways to install a tool from a pipeline:

  1. Using the Tool Version field on a supported pipeline step. Using this method, the tool is only installed for the subprocess run by the step, and will not propagate to other operating system sessions.
  2. Using a tool installer step module. The step modules install the tool globally on the runner machine by modifying the PATH and environment.

Tool installations are cached on the runner for each version of the tool.