Change Pipeline

The Change Pipeline step redirects the data that it receives into a different Pipeline Workflow.

Table of Contents

Overview

The Change Pipeline step enables the creation of pipeline stages: chaining multiple pipeline workflows, where each workflow indexes the data into Squirro, and then passes them to the next workflow for further processing.

For example, it can be used as the last step of a lightweight workflow in order to index the data as fast as possible, and then redirect them to a more heavyweight workflow which includes computationally expensive steps.

Additionally this step can change the priority of the redirected items. This can be done to ensure that the important pipeline steps such as initial data indexing happens more quickly and the slower components do not block the ingestion of new data. See https://squirro.atlassian.net/wiki/pages/resumedraft.action?draftId=2713813031 for further information.

The picture above depicts two workflows:

• the Light Workflow, which includes a small set of steps that they are generally fast to execute, and as its last step, the Change Pipeline step which is configured to redirect the data into the Heavy Workflow.

• the Heavy Workflow, which includes a fictional “Heavy Pipelet” step, which represents a pipelet that is computationally expensive and may take a lot of time to execute.

In this scenario, the data will undergo a basic enrichment and get indexed into Squirro by the Light Workflow, and then continue to the Heavy Workflow, which will further enrich the items and finally index them into Squirro again in order for the new enrichment to become available.

Configuration