Multi-instance variable aggregation

Multi-instance workflow executions allow a task or subprocess to be executed repeatedly, either sequentially or in parallel, for a dynamic number of items in a collection. A common use case is collecting input from multiple participants, such as reviewers providing feedback on a document or application.

By default, variables created or updated in each individual instance of a multi-instance execution are confined to a local scope. This prevents data clashes but makes it difficult to consolidate outputs from multiple instances into a single variable for subsequent use. The Variable Aggregations feature automates this process, enabling the collection, consolidation, and transformation of data from multi-instance executions efficiently.

Configuring variable aggregations

The Variable Aggregations feature is available in the Multi instance properties group of process elements that support multi-instance executions. It allows you to configure one or more aggregation definitions, mapping variables from individual executions to a single, structured target variable.

The configuration includes the following options:

  • Target: The name of the final variable that contains the aggregated data or an expression that resolves to a variable name at runtime. The aggregated variable is typically a JSON array. All variable definitions from the same aggregation contribute to a single target variable.
  • Type: The method used for aggregation:
    • Default: Uses the out-of-the-box aggregation logic, which is suitable for most use cases.
    • Custom: Allows for advanced aggregation logic by specifying a Delegate Expression or a Class that implements the VariableAggregator interface. This option provides an extension point for highly specific requirements.
  • Target variable creation: A predefined list of options that determine when and how the target variable is created and persisted:
    • Default: Creates the target variable as a normal, persisted process variable only after all multi-instance executions are completed. This is the standard behavior for a "map-reduce" pattern.
    • Create overview variable: Creates the variable at the start of the multi-instance loop and is continuously updates it as each child instance is completed. This allows for real-time access to the aggregated data by any running process instance. The final variable created is a normal, persisted JSON variable.
    • Store as transient variable: Creates the target variable upon completion of all instances, similar to the Default option, but as a non-persisted or transient variable. This data is not saved to the process history and is suitable for temporary or intermediate data that is not needed for auditing or long-term storage.
  • Definitions: A list of definitions that map source variables from each multi-instance execution to a target variable key in the final aggregated collection. For each definition, you specify a key-value pair that to be included in the JSON object representing the aggregated data:
    • Source: The input data, as variable name or expression.
    • Target: The key under which the source data is stored in the final JSON array. If the target name is identical to the source variable, the target is optional.

Important considerations

The options for Target variable creation and the overarching variable retention policy represent important aspects of how data is handled in multi-instance loops.

The options for creating the target variable offer distinct behavioral patterns with significant implications for process design:

  • Default behavior: The Default option is straightforward and mirrors a common "map-reduce" pattern. The data from all individual instances is collected, but the final, aggregated variable is not created until the entire multi-instance loop has completed. This behavior is ideal for scenarios where the aggregated data is only needed for a subsequent task that requires a complete dataset, such as making a final decision after all review feedback has been collected.
  • Enabling dynamic and real-time workflows with the Create overview variable option: Unlike the Default option, the Create overview variable option creates an overview variable at the very start of the multi-instance loop. This variable is continuously updated in real-time as each individual execution completes, allowing a parent process or even other concurrent child executions to access the current, partial state of the aggregated data. For example, a subprocess might use a completion condition that dynamically inspects the overview variable and terminates the loop as soon as a certain threshold is met, such as reaching a combined score of 250 or receiving three Approved comments. This capability transforms the multi-instance pattern from a static data collection mechanism into a dynamic coordination tool that allows the process to react to partial results, leading to more adaptive and efficient workflows.

When variable aggregation is enabled, only data specified in the aggregation is retained after the multi-instance execution is completed. All other locally-scoped variables are discarded. This policy promotes a cleaner, more efficient process instance variable space by preventing clutter. To prevent data loss, ensure that all required data is explicitly defined in the variable Definitions.

Expressions in aggregation

Variable definitions support expressions, such as ${my_expression}, allowing for data transformations during aggregation. This centralizes data transformation logic in the process element, reducing the need for separate script or service tasks.

Example of aggregating feedback from multiple reviewers

Consider a review process with a user task that collects the input of reviewers, each with a name and a score weight. The task is executed for each reviewer in the collection, allowing them to provide a score and a comment.

  1. Multi-instance setup:
    • Multi instance type: Parallel.
    • Collection: This expression points to the collection variable created in a previous task, for example ${reviewers}.
    • Element variable: A temporary variable referencing the current item in the collection, such as reviewer.
  2. Variable Aggregations:
    • Target: reviews.
    • Type: Default.
    • Target variable creation: Default.
  3. Definition that maps the comment variable from the local instance to the comment key in the final JSON object.:
    • Source: comment.
  4. Definition that uses an expression to transform data. It multiplies the locally-scoped score by the scoreWeight from the reviewer element variable and divides by 100 to get a weighted score. This calculated value is then stored under the score key in the final JSON object:
    • Source: ${score * reviewer.scoreWeight / 100}.
    • Target: score.
  5. Definition that maps the name of the reviewer from the reviewer element variable to the reviewer key in the final JSON object:
    • Source: ${reviewer.name}.
    • Target: reviewer.

After all executions are completed, the aggregated variable reviews contains a JSON array with the consolidated data:

[
    { "comment": "Application looks good", "score": 90.0, "reviewer": "Jane" },
    { "comment": "Application looks perfect", "score": 80.0, "reviewer": "Jake" }
]

Conclusion

The Variable Aggregations feature simplifies process models by automating data consolidation and transformation. It eliminates the need for manual workarounds, reduces development effort, and minimizes errors. By centralizing logic in the process element, it promotes readability, maintainability, and scalability in workflows.