Squirro 2.6.3 - Release Notes
- Moritz Müller
- Saurabh Jain
- Ethan Brown (Deactivated)
Owned by Moritz Müller
We're excited to announce Squirro 2.6.3, released on January 22nd 2019.
Known upgrade limitations:
- If you are using custom widgets based on the horizontal result widget in one of your projects, do not update to 2.6.3 yet, but please contact support. to clarify required steps.
Contents -What's in the release?
New Features
Data Sources
- New Overview of existing Data Sources:
- See scheduling of data loading with information about "Last Run" and "Next Run"
- See the Number of items that exist for this data source.
- Source Status added: Queued, Running, Error, Complete
- New action buttons: Run, Stop, Reset, Explore added. Explore allows you to explore the items coming from a data source on the Explore dashboard.
- Data Source Types:
- Support of new File Importer Plugin:
- Upload of a ZIP File containing multiple PDF and Microsoft Office documents
- Support of Frontend data loading for PDF and Microsoft Office documents.
- Removal of old data source types that are no longer supported.
- (Previous data loader plugins are still available as legacy versions and will be supported on request.)
- Support of new File Importer Plugin:
- Replacement of Providers by Data Loader Plugins (more information below)
Entities
- With 2.6.3 the use of entities is significantly improved throughout Squirro.
- Entities are now visible in the Explore screen.
- In the Explore screen the item display can be switched between showing the abstract of the item or the related entities.
- Addition of entity filters to Explore screen sidebar and Dashboard sidebar.
- Addition of new Entities widget.
- Entity Highlighting: Jump to entity extract within the item view when selecting respective entity.
Machine Learning
- Major improvement of Annotation Workbench for Model training (typeahead, keyword support, etc.)
- Numerous enhancements of machine learning service to simplify and extend functionality (see below in section: New libNLP steps and features.)
- Placement of ML tasks under TRAIN tab. (Previously in Project Settings).
- Addition of "Run Now" Button to ML Job Page.
- New squirro_client method: get_machinelearning_workflow_assets (See documentation here.)
Widgets & Dashboard
New Widgets
- Divider: Show a line to separate areas of the dashboard and add section titles
- Sample Text Labeller:The Sample Text Labeller widget allows to do an inference test to assert the performance of a given Machine Learning Workflow.
- Item Detail: The item detail widget shows the contents of the active Squirro item. It can also be controlled by the dashboard store to display the item whose ID is stored there.
- Entities: The entities widget display all the entities available in the project, in a dropdown fashion similar to Facets widget.
Widget Improvements and Changes
Merge of Horizontal Result List into Result List Widget:
- Option to switch between horizontal and vertical mode in result list widget.
- Widget Static State / Drilldown Disabled State
- We have an option to disable drilldown on some widgets, in which case, widget does not react to any query except its own. In such case, widgets visually adopt a static state which highlights only the value selected on them.
- Result List Widget
- New option to change between vertical (default) and horizontal mode (previously horizontal result widget)
- Open item detail in place. Designed for efficient use of screen real estate, it opens the item detail in place of the result list. Cross button allows you to navigate back to the list in a single click.
- Config. option to hide item metadata and abstract. Now display only the titles of the item for a quick scan through the list.
- All Result displaying Widgets (Result List, Cards, Facet Table, Kanban Lane)
- Selection of Admin vs. End User mode for the visibility of facets. In Admin Mode all facets are visible and editable; whereas in End User Mode only the pre-defined facets are shown to the user, offering a cleaner look for the item detail.
- New config. option to toggle filtering by item tags.
- Display only the value from a key value pair inside the item tag UI element.
- Enjoy seamless infinite scroll on mobile dashboards. If the bottom-most widget on a mobile dashboard is a scrollable widget, namely Result List, Cards, Table and Facet Table widget, the infinite scroll is activated.
Dashboard & Layer
- Option to copy & paste widgets across all layers and dashboards within a project for both desktop and mobile mode of the dashboard editor.
- Option to copy & paste layers across all dashboards within a project or both desktop and mobile version of the dashboard editor.
- Numerous bug fixes of dashboard & layer functionalities.
Improvements
New libNLP steps and features
- Query multiple Squirro instances in Squirro query loader
- Threshold filter step for predicted facet
- Squirro entity filter to create Squirro entities
- Tracking facet to record which model has been applied
- Smartfilter classifier
- Regex filter
Return of "Import Project" Functionality
- "Import Project" was integrated into Project Template in previous release causing a loss of certain functionalities, so we brought the "Import Project" functionality back that allows you to import selected features.
Refactoring of Services
- Topic Service
- Dataloader Service
- Update of Ingester Service to account for refactoring of Topic & Dataloader Service.
Major Code Cleanup
- Removal of significant amount of legacy code
- Removal of deprecated assets
Continued integration of project template functionality
- Simpler and more enhanced project template functionality.
- Significantly reduced setup time of projects.
- Possibility to ship project templates (predefined data sources, dashboards, classification jobs (ML tasks), etc.)
Other Improvements
- Update of language detection module (Now language detection with ML-based library: fastText)
- Change of internal data schema of MySQL database (removal of unnecessary fields, etc.)
- Addition of squirro_status CLI Tool
- Update of Highcharts library to version v6.2.0
- Update of several other libraries to the latest versions
- Capability to use user-specific parameters for machine learning & training.
Bug Fixes & Improvements
Deprecations
Changes of Python client
Deprecated Services
- API Sourcer
- API Bulk
- Fileimport
- Squirrel
Deprecated Providers
All providers have been removed and will be replaced by dataloader plugins instead.
| Provider | Note |
|---|---|
| bing | To be replaced with the Newsriver Plugin |
| dataloader | Replaced with the new datasource service |
| diffbot | Removed, no replacement planned |
| Removed | |
| feed | To be replaced by a dataloader plugin |
| Removed | |
| To be replaced with a dataloader plugin | |
| youtube | Removed |
Added on January 31, 2019 (build 212 - patch 4)
- Fix broken run of pre 2.6.3 data sources.
- Fix typeahead cursor position
- Remove support of
processing_configfromItemUploader. - Add proximity filter to lib NLP.
- Fix result list in hidden layer.
Added on February 18, 2019 (build 232 - patch 5-11)
- Fix load more in result list.
- Fix metric widget cutoff.
- Fix HTTP method for pipeline workflow moving.
- Force validation via docstring Parameters in libNLP.
- Fix result list in explore screen.
- Ensure detail event on cards widget.
- Fix name scoping of keys in KV cache/store in DL plugins.
- Don't open edit mode when passing dash params.
- Speed up weighted keyword aggregations.
- Detach frozen widgets collections.
- Allow disabling of job locking by cli dataloader.
- Tune scheduler settings for datasource.
- Add force_cache_clear for item updates.
- Move ML job runs into a separate process (sqml).
- Optional mode selection for libNLP steps. Allow renaming of entity property values.
- Remove linkedinservice.
- Improve group members loading performance by improving sql joins.
- Fix ES migration for empty projects and Squirro's filter index.
- More fair scheduling of sources by Ingester (Processes one batch of each source at a time).
- Added a copy filter in libNLP for name spacing.
- Migrate
typestotypein ML workflows. Catch validation errors before ML job runs. - Fix bug where entities are no longer showing in explore view.
- Annotation workbench fixes.
- Scheduler fixes.
- Add tracking property to entities.
- Fixes for parallel sections in ML workflows.
Added on February 21, 2019 (build 234 - patch 12)
- Fix force_cache_clear for modify_items
Added on March 4, 2019 (build 236 - patch 13)
- Add config options to rename axis labels
- Item detail widget title aligned with star icon
- Cards widget: Apply card style only to the header
- Item Detail widget: Have option to configure pre-defined tags
- Show item title in two lines if needed
- Ability to sort facet values
- More robust scheduling policies for ingester
- Fix dashboard save
- Improve dashboards print
- Fix mobile dashboard bottom widget cut off
- Fix item detail widget padding on mobile
- Open in place for cards widget
- Hide space changer until permissions load
- Breadcrumbs widget doesnt display query from tab (bookmark) widget
- Fixing typeahead overflow
- Improved default time bucketing for line charts and time aggregations
Added on March 5, 2019 (build 237 - patch 14)
- Add feed plugin by default to globally shipped plugins
Added on March 11, 2019 (build 239 - patch 15)
- Fix broken sdk on Squirro servers.
- Fix ingester bug where sources with empty pipelet config will not be processed.
- Support select on non created-at facets in timeline.
- Isolate materialize navs and subnavs.
Added on March 12, 2019 (build 240 - patch 16)
- Fix ingester bug where sources with empty pipelet config will not be processed.
Added on March 12, 2019 (build 242 - patch 17)
- Fix an issue with dashboards throwing occasional error
Added on March 13, 2019 (build 243 - patch 18)
- Fix an issue with smartfilter upload
Added on March 15, 2019 (build 244 - patch 19)
- Item detail widget store controlled fails to show item if not in collection
- Do not export data checkpoints when exporting ML Models
Added on March 20, 2019 (build 245 - patch 20)
- Fix an issue config file encryption in a cluster setup.
Added on March 25, 2019 (build 247 - patch 21)
- Fix an issue with deduplicating items caused by the source refactoring.
Added on March 26, 2019 (build 248 - patch 22)
- Fix another issue with deduplicating items caused by the source refactoring.
Added on March 31, 2019 (build 249 - patch 23)
- Update link for downloading fasttext language detection model
- Fix project import/export on IE
- Improve ES query for many duplicates
Added on March 31, 2019 (build 249 - patch 23)
- Typeahead fixes
- Facet name suggestions in typeahead results are now alphabetically sorted.
- Properly quote typeahead suggestions to generate a valid Squirro query if the typeahead suggestions contain special characters (like parantheses)
- Alphabetical sorting of facet values on dashboards.
Installation & Upgrade
Fresh Installation Instructions
Please follow the regular installation steps
Upgrade Instructions
Please ensure that your current version is 2.6.2. If you are on a version older than 2.6.2, please contact support.
Due to the source and provider framework refactoring the update instructions are more involved than usual. Please make sure you follow each step in the update instructions carefully.
During the upgrade some of the data in elasticsearch will be modified. This leads to a temporary increase of disk space use. Please ensure that you have enough free disks space available before you commence.
The default location of the index files is: /var/lib/elasticsearch
In this example the overall index size is 28 GB:
# du -sch /var/lib/elasticsearch/ 28G /var/lib/elasticsearch/ 28G total
For the migration to succeed, we recommend you ensure you have at least another 28 GB of free disk space available.