Page Comparison

...

The pipeline editor has been completely recreated. The new editor is more visual, and provides a much easier overview of the various pipelines in a project.

• Add In addition to that we have also laid down a lot of ground-work to allow for the re-running of pipeline workflows (for datasources running in the Frontend only) which allows for easier experimentation during the project setup process. For the more technical audience, this is enabled through the following underlying configurations. This will be included in the frontend in the upcoming releases.

  • We have added a new built-in pipeline step “Transform Input” which does the item fields and facets mapping. This

  used to be

  • was previously done

  before

  • in the dataloader

  .Control whether item transformation happens during the extraction phase from the dataloader or inside a pipeline workflow from the “Transform Input” step

  • itself but can now be handled in the pipeline itself. This step is controlled using the configuration option item_transformation_in_pipeline. It is disabled by default, and should be considered

  experimental

  • a beta feature for this release.

  Modify

  • We have introduced a new processed directory in the Ingester to

  move

  • store the input data to

  a processed directory after executing the pipeline

  • this directory before performing pipeline steps. This enables us to keep a copy of the raw data to re-run the pipeline without fetching the data from the original source. This behavior is controlled by the configuration option keep_processed_data, which is also disabled by default

  . Useful for re-running

  • .

  Not recommended for production setup.Extend

  • We have also extend the Ingester to automatically remove the

  processed

  • input data after a certain time period or disk space threshold

  . Controlled

  • to avoid disk over-filling. This is controlled by the configuration options days_to_retain_processed_batchesand hours_to_retain_processed_batches. This mechanism kicks in when the keep_processed

  data are configured to be retained.

• Offer three Pipeline Workflow presets, a set of pre-made Pipeline Workflows with steps for covering various use cases.

  • Minimal

  • Standard

  • Binary Document

• We have

  • _data is enabled.

• In addition, we now offer three different Pipeline Workflow presets designed for various use-cases.

...

• We have also added the functionality to rename all of the steps in pipeline workflow to your liking.

• Pipelets which hint in their names that they perform Known Entity Extraction are now by default categorized in the “Relate” section of the Pipeline Editor.

...

• Projects names are visible again in the project sidebar.

• Adapt the SQ Dataloader behavior to not kill jobs automatically every 30 mins as long as we can fetch new data.

• Cannot Fixed an issue where user cannot change avatar picture - because save button missing

• Project selector (name) is missing if there's only one dashboard

• Item detail: x button is misaligned.

• "[Data Sources] ""Add feed source URL"" button has a weird circle animation".

• Add animation for “Add feed sources” button on the feed dataloader plugin.

• Fixed an issue where Startup of topic service would fail to install saml2 plugin because of failing to install the pysaml2 dependency.

• Multiple Visual bug fixes around Cards widget and item detail view.

• Improve exception handing in the feed plugin.

• Scan endpoint now is not limited to 1000 entities per call/iteration.

• Fixed the creation of favorites on the dashboard.

Salesforce SDK

• Fonts are now downloaded from Salesforce instead of Squirro.

• Styles no longer bleed from Squirro to Salesforce.

• Fixed an issue where we were unable to open items.

...