Change Pipeline

The Change Pipeline step redirects the data that it receives into a different Pipeline Workflow.

Table of Contents

Overview

The Change Pipeline step enables the creation of pipeline stages: chaining multiple pipeline workflows, where each workflow indexes the data into Squirro, and then passes them to the next workflow for further processing.

For example, it can be used as the last step of a lightweight workflow in order to index the data as fast as possible, and then redirect them to a more heavyweight workflow which includes computationally expensive steps.

The picture above depicts two workflows:

• the Light Workflow, which includes a small set of steps that they are generally fast to execute, and as its last step, the Change Pipeline step which is configured to redirect the data into the Heavy Workflow.

• the Heavy Workflow, which includes a fictional “Heavy Pipelet” step, which represents a pipelet that is computationally expensive and may take a lot of time to execute.

In this scenario, the data will undergo a basic enrichment and get indexed into Squirro by the Light Workflow, and then continue to the Heavy Workflow, which will further enrich the items and finally index them into Squirro again in order for the new enrichment to become available.

Configuration