...

New Ingestion Pipeline Editor

The pipeline editor has been completely recreated. The new editor is more visual, and provides a much easier overview of the various pipelines in a project.

• Add new built-in pipeline step “Transform Input” which does the item fields and facets mapping. This used to be done before in the dataloader.

• Control whether item transformation happens during the extraction phase from the dataloader or inside a pipeline workflow from the “Transform Input” step using the configuration option item_transformation_in_pipeline. It is disabled by default, and should be considered experimental for this release.

• Modify the Ingester to move the input data to a processed directory after executing the pipeline. This behavior is controlled by the configuration option keep_processed_data, which is disabled by default.

• Extend the Ingester to automatically remove the processed input data after a certain time period or disk space threshold. Controlled by the configuration options days_to_retain_processed_batchesand hours_to_retain_processed_batches. This mechanism kicks in when processed data are configured to be retained.

• Offer three Pipeline Workflow presets, a set of pre-made Pipeline Workflows with steps for covering various use cases.

• Built-in steps have the possibility to use the config_options to render steps specific settings.

• Error when configuring source with pipeline steps that come from a DL plugin

• Some steps which are already part of a workflow are missing values for certain properties

• Pipelet implementing getArguments() is not rendered as 1st class widget -- throws error instead

• Pipeline Editor: Displaying issue with the config options of the Near-Duplicate Detection

• "Pipeline editor: On using scroll arrows in the top left of the editor, the pipelines should scroll instead of jump

• Pipeline Editor: Cannot delete workflow which was recently created and saved without reentering the editor

• [Pipeline Editor] Step facet properties not being saved after changing value

• "Pipeline editor - The link under Related in the left panel ""Add new Relate steps in the AI Studio"" links a user to the 'dashboards' space"

• [Pipeline] Pipeline edit middle draggable section frontend implementation

• Unable to rename step in pipeline workflow

• [backend] Enable modification of pipeline step names

• Ability to rerun whole pipeline for already processed data

• Pre-populate pipeline with default steps

• Enable triggering the rerunning of a pipeline workflow on the already processed raw data

• Data loader frontend config - booleans break frontend when mappings.json provided

• Data loader frontend config - Defaults don't show up

• Pipelets which hint in their names that they perform Known Entity Extraction are categorized in the “Relate” section of the Pipeline Editor.

...

• Fonts are now downloaded from Salesforce instead of Squirro.

• Styles no longer bleed from Squirro to Salesforce.

• Fixed an issue where we were unable to open items.

Breaking Changes

With the introduction of the new pipeline editor, the navigation structure in the Setup space has changed. To make room for the new pipeline editor to use the full width, all the options that were previously under the Enrich tab have now moved to the new AI Studio tab. Rerunning of enrichments will soon disappear from there and then move into the new pipeline editor properly. As a result, any custom studio plugins under these sections have to be re-uploaded to either one of the existing sections or to a new section called “AI Studio”. This Can be achieved by specifying the ”location”: “dss” in the studio_plugin.json file

Fresh Installation Instructions

...