New Features

Editable e-mail templates

The E-mail Templates used for alert and trend notification are now configurable per tenant.

Monitoring Plugin

We're releasing a Studio Plugin that offers a overview of the health of the System to establish the system health without the need of logging into the system and any sysadmin skills.

• Status of all services on all cluster nodes
• Status of all Elasticsearch instances
• Overview of all Elasticsearch Indices, including health state, size, number of documents etc.
• Status of the Pipeline/Ingester with current activity and errors etc.

OpenSearch API

We added a Studio Plugin that exposes an OpenSearch API compatible endpoint of the Squirro search interface. This endpoint can for example be used to integrate Squirro search results into Salesforce via federated search. See Salesforce Federated Search for further instructions on how to do this.

Contextual Help

In order to make the Squirro experience simpler, we have added contextual help to nearly every page inside the Setup and Server spaces. To access this help, all you need to do is click on the question mark icon on the top right corner and choose the "Explain this page" option. Relevant links will guide you directly to documentation where you can discover all about the page you're on.

We are working towards making Squirro more intuitive and easy to use, shaping the learning curve as steep as possible to ensure that users can quickly grasp the tools and techniques in a very short time.

Improvements

Machine learning service

2.6.1. brings improved stability and robustness to the Machine Learning Service. For better vision into how different jobs run (or fail), we've added API access to run logs. If you notice a problem, we've also added the ability to kill jobs. For model tuning, we're now recording model performance per training batch, and for model validation, we've adjusted the Heatmap widget to be used effectively as a confusion matrix. Finally, as we are trying to make the process of creating and deploying machine learning models more efficient, we're providing a simple example pipelet that can tag ingested documents with a pre-trained model. To see it in action, we recommend following the provided walkthroughs. 

Recommendations

Recommendations get a new look in 2.6.1. We've converted our custom widget into a standard product widget with several UI improvements both in creation and display. These include more accurate feature counts, tighter integration with the Machine Learning Service, and pagination on entities result lists. NOTE: This feature is still considered in ALPHA, so please speak with our team before deploying.

Mobile dashboards

• The Mobile Dashboard Editor is a new addition to the dashboard editing tool. It allows you to edit and maintain a mobile version for each dashboard. The web and mobile version of a dashboard share the same URL. While the web mode of a dashboard renders on all web-like and tablet screens, the mobile mode of a dashboard renders on all mobile-like screens. While editing a dashboard, user is offered a switcher to switch between the two modes. The web and the mobile dashboards are synced by default. The sync can be turned on/off. As the desktop dashboard is the leading dashboard, re-syncing is uni-directional, i.e., it copies the layers and widgets from the web mode to the mobile and not the other way around.
• The mobile version of the dashboard is now launched with improved functionality and interaction patterns.
• Mobile dashboard widgets occupy full width and are stacked automatically. 
• Dashboard now has vertical scroll instead of horizontal swipe.
• Custom widgets are supported. 
• Some widgets might have reduced config. options based on optimisation for mobile.

Widget Browser

• The Dashboard Editor gets a new look.
• The widget selector dropdown is now replaced by a widget browser window which is easy to use and drastically cuts short the time and clicks required to build dashboards.
• Easy to find: Widgets browser now has built-in search functionality.
• Easy to locate: All widgets are listed in a tile view with representative icons.
• Easy to glance: A quick scroll allows you to glance over the entire widget library.
• Easy to build: Widgets can be dragged and dropped to the desired position on the dashboard.
• Time Saving: Multiple widgets can be added in one go, allowing you to put together a decent dashboard within seconds.
• Logical listing: All widgets are listed under functional categories: items, navigation, filters, over-time, distribution, geographic, special, others and custom.
• Descriptive: Every widget has a description displayed on hover over the widget icon. Custom widgets upload tool now supports description.

Project Browser

In the new Project Browser, the project dropdown is replaced by a slide in navigation drawer. The active project is highlighted. Add Project button is on the top of the projects list, thus eliminating the need to scroll to the bottom of the list to create a new project.

Dashboard and widgets related improvements

• Every new dashboard has a few widgets put together to give you a head start.
• Widget resize from all directions.
• Widget duplicate is of the same size as the original.
• Each widget now comes with its own custom size appropriated to match their commonly used sizes.
• Dashboard has grey background color to pop out the cards.
• Widgets can have transparency on turning the card off.
• Finer grids (now 48 columns).
• Bookmark widget has a chip mode to allow you to toggle the selection made.
• Link widget can now be controlled by dashboard store value.
• Free Placement: Option to freely place the widgets on the dashboard.
• Dropdown mode for smartfilters, trends and favourite widget.
• Drill the dashboard by metric widget. 
• Option to freeze widget state by disabling drilldown on the widget by its own query.
• Read more about widgets

Bug Fixes

Pipeline - less memory hunger for noise removal

• The noise removal step requires now much less memory and can process also complex documents in large batches.
• Time selection widget toggle buttons mode now displays all options

Added on July 13, 2018 (build 2457 – patch 1)

• SQ-9715 - Table widget does not react to query changes.
• SQ-9711 - Item detail has wrong white-space on pre tag
• SQ-9712 - Topic cluster has wrong probability label
• SQ-9695 - Fix repeated options on time dropdown
• SQ-9589 - CTA is displayed in the center of the card
• SQ-9716 - Share dashboard does not have enough grants to look at the entities
• SQ-9584 - Fix overflow issues on Metric widget
• SQ-9619 - Fixing hide top values in pie chart
• SQ-9698 - Fixing broken label select on store change
• SQ-9414 - Avoid source creation if default workflow is chosen for an existing source
• SQ-9706 - Editing of layer conditions is not working in some dashboards
• SQ-9704 - Facets list dropdown mode is lost for old dashboards
• Allow scheduler options for ML jobs
• Default to running only once for ML training jobs. Check for existence before deleting ML run directory
• Migrate facetlists property
• Update libNLP documentation. Making sure Squirro item conversation only allows Squirro types.
• Added separate training and inference storage for clustered setups. Fixed bug where error status was not being reported in ML jobs.

Added on July 17, 2018 (build 2457 – patch 2)

• Added projector steps to allow dimensionality reduction.
• Fix path in topic modeling analyzer.
• Fix facets factory initialisation
• SQ-9669 - Adding fix for bar chart on drill down
• Fix parameter name translation for last_n_log_lines in ML jobs

Added on July 17, 2018 (build 2458 – patch 3)

• Fix drag and drop in widgets

Added on July 26, 2018 (build 2460 – patch 4)

• SQ-9784  - Rerun studio plugin fails to add keywords to items

• Fix in the Topic Clusters Widget: all topics are now show

Added on July 26, 2018 (build 2461 – patch 5)

• Fix for killing machine learning jobs
• Monit file for Mysql no longer shipped for centos7
• Properly handle the Not-Found error while killing a ML job
• Improvements to the Monitoring Studio plugin
  • No scroll bar in the tabs
  • Add a loading indicator

Added on July 26, 2018 (build 2463 – patch 6)

• Fix null return features in recommendations when return features is not the target feature.

Added on July 30, 2018 (build 2465 – patch 7)

• Improved "ML Jobs" plugins with more jobs information and actions like killing the job & fetching the run logs. More documentation here
• Add fasttext library for Machine Learning.
• Fixed end-to-end embedding dictionary dimensions. Allow labels with spaces in fasttext classifier. Auto-weight imbalanced labels in seq2one classifiers.

Added on August 8, 2018 (build 2466 – patch 8)

• Fix a bug where editing a dataloader subscription configuration will leave the orphaned sources with the old configuration behind.
• More robust scheduling handling for dataloader & machine learning jobs.

Added on August 9, 2018 (build 2467 – patch 9).

...

New Features

Editable e-mail templates

The E-mail Templates used for alert and trend notification are now configurable per tenant.

Monitoring Plugin

We're releasing a Studio Plugin that offers a overview of the health of the System to establish the system health without the need of logging into the system and any sysadmin skills.

• Status of all services on all cluster nodes
• Status of all Elasticsearch instances
• Overview of all Elasticsearch Indices, including health state, size, number of documents etc.
• Status of the Pipeline/Ingester with current activity and errors etc.

OpenSearch API

We added a Studio Plugin that exposes an OpenSearch API compatible endpoint of the Squirro search interface. This endpoint can for example be used to integrate Squirro search results into Salesforce via federated search. See Salesforce Federated Search for further instructions on how to do this.

Contextual Help

In order to make the Squirro experience simpler, we have added contextual help to nearly every page inside the Setup and Server spaces. To access this help, all you need to do is click on the question mark icon on the top right corner and choose the "Explain this page" option. Relevant links will guide you directly to documentation where you can discover all about the page you're on.

We are working towards making Squirro more intuitive and easy to use, shaping the learning curve as steep as possible to ensure that users can quickly grasp the tools and techniques in a very short time.

Improvements

Machine learning service

2.6.1. brings improved stability and robustness to the Machine Learning Service. For better vision into how different jobs run (or fail), we've added API access to run logs. If you notice a problem, we've also added the ability to kill jobs. For model tuning, we're now recording model performance per training batch, and for model validation, we've adjusted the Heatmap widget to be used effectively as a confusion matrix. Finally, as we are trying to make the process of creating and deploying machine learning models more efficient, we're providing a simple example pipelet that can tag ingested documents with a pre-trained model. To see it in action, we recommend following the provided walkthroughs. 

Recommendations

Recommendations get a new look in 2.6.1. We've converted our custom widget into a standard product widget with several UI improvements both in creation and display. These include more accurate feature counts, tighter integration with the Machine Learning Service, and pagination on entities result lists. NOTE: This feature is still considered in ALPHA, so please speak with our team before deploying.

Mobile dashboards

• The Mobile Dashboard Editor is a new addition to the dashboard editing tool. It allows you to edit and maintain a mobile version for each dashboard. The web and mobile version of a dashboard share the same URL. While the web mode of a dashboard renders on all web-like and tablet screens, the mobile mode of a dashboard renders on all mobile-like screens. While editing a dashboard, user is offered a switcher to switch between the two modes. The web and the mobile dashboards are synced by default. The sync can be turned on/off. As the desktop dashboard is the leading dashboard, re-syncing is uni-directional, i.e., it copies the layers and widgets from the web mode to the mobile and not the other way around.
• The mobile version of the dashboard is now launched with improved functionality and interaction patterns.
• Mobile dashboard widgets occupy full width and are stacked automatically. 
• Dashboard now has vertical scroll instead of horizontal swipe.
• Custom widgets are supported. 
• Some widgets might have reduced config. options based on optimisation for mobile.

Widget Browser

• The Dashboard Editor gets a new look.
• The widget selector dropdown is now replaced by a widget browser window which is easy to use and drastically cuts short the time and clicks required to build dashboards.
• Easy to find: Widgets browser now has built-in search functionality.
• Easy to locate: All widgets are listed in a tile view with representative icons.
• Easy to glance: A quick scroll allows you to glance over the entire widget library.
• Easy to build: Widgets can be dragged and dropped to the desired position on the dashboard.
• Time Saving: Multiple widgets can be added in one go, allowing you to put together a decent dashboard within seconds.
• Logical listing: All widgets are listed under functional categories: items, navigation, filters, over-time, distribution, geographic, special, others and custom.
• Descriptive: Every widget has a description displayed on hover over the widget icon. Custom widgets upload tool now supports description.

Project Browser

In the new Project Browser, the project dropdown is replaced by a slide in navigation drawer. The active project is highlighted. Add Project button is on the top of the projects list, thus eliminating the need to scroll to the bottom of the list to create a new project.

Dashboard and widgets related improvements

• Every new dashboard has a few widgets put together to give you a head start.
• Widget resize from all directions.
• Widget duplicate is of the same size as the original.
• Each widget now comes with its own custom size appropriated to match their commonly used sizes.
• Dashboard has grey background color to pop out the cards.
• Widgets can have transparency on turning the card off.
• Finer grids (now 48 columns).
• Bookmark widget has a chip mode to allow you to toggle the selection made.
• Link widget can now be controlled by dashboard store value.
• Free Placement: Option to freely place the widgets on the dashboard.
• Dropdown mode for smartfilters, trends and favourite widget.
• Drill the dashboard by metric widget. 
• Option to freeze widget state by disabling drilldown on the widget by its own query.
• Read more about widgets

Bug Fixes

Pipeline - less memory hunger for noise removal

• The noise removal step requires now much less memory and can process also complex documents in large batches.
• Time selection widget toggle buttons mode now displays all options

Added on July 13, 2018 (build 2457 – patch 1)

• SQ-9715 - Table widget does not react to query changes.
• SQ-9711 - Item detail has wrong white-space on pre tag
• SQ-9712 - Topic cluster has wrong probability label
• SQ-9695 - Fix repeated options on time dropdown
• SQ-9589 - CTA is displayed in the center of the card
• SQ-9716 - Share dashboard does not have enough grants to look at the entities
• SQ-9584 - Fix overflow issues on Metric widget
• SQ-9619 - Fixing hide top values in pie chart
• SQ-9698 - Fixing broken label select on store change
• SQ-9414 - Avoid source creation if default workflow is chosen for an existing source
• SQ-9706 - Editing of layer conditions is not working in some dashboards
• SQ-9704 - Facets list dropdown mode is lost for old dashboards
• Allow scheduler options for ML jobs
• Default to running only once for ML training jobs. Check for existence before deleting ML run directory
• Migrate facetlists property
• Update libNLP documentation. Making sure Squirro item conversation only allows Squirro types.
• Added separate training and inference storage for clustered setups. Fixed bug where error status was not being reported in ML jobs.

Added on July 17, 2018 (build 2457 – patch 2)

• Added projector steps to allow dimensionality reduction.
• Fix path in topic modeling analyzer.
• Fix facets factory initialisation
• SQ-9669 - Adding fix for bar chart on drill down
• Fix parameter name translation for last_n_log_lines in ML jobs

Added on July 17, 2018 (build 2458 – patch 3)

• Fix drag and drop in widgets

Added on July 26, 2018 (build 2460 – patch 4)

• SQ-9784  - Rerun studio plugin fails to add keywords to items

• Fix in the Topic Clusters Widget: all topics are now show

Added on July 26, 2018 (build 2461 – patch 5)

• Fix for killing machine learning jobs
• Monit file for Mysql no longer shipped for centos7
• Properly handle the Not-Found error while killing a ML job
• Improvements to the Monitoring Studio plugin
  • No scroll bar in the tabs
  • Add a loading indicator

Added on July 26, 2018 (build 2463 – patch 6)

• Fix null return features in recommendations when return features is not the target feature.

Added on July 30, 2018 (build 2465 – patch 7)

• Improved "ML Jobs" plugins with more jobs information and actions like killing the job & fetching the run logs. More documentation here
• Add fasttext library for Machine Learning.
• Fixed end-to-end embedding dictionary dimensions. Allow labels with spaces in fasttext classifier. Auto-weight imbalanced labels in seq2one classifiers.

Added on August 8, 2018 (build 2466 – patch 8)

• Fix a bug where editing a dataloader subscription configuration will leave the orphaned sources with the old configuration behind.
• More robust scheduling handling for dataloader & machine learning jobs.

Added on August 9, 2018 (build 2467 – patch 9)

• "Rerun Enrichment" now takes care of sending the enrichment config properly when rerunning an enrichment.

Added on August 10, 2018 (build 2468 – patch 10)

• Fixed nginx file permission issues due to selinux policies.
• Hardened query checks in bookmark widgets.

Added on August 13, 2018 (build 2470 – patch 11)

• Support for custom steps in machine learning workflows.
• Backend support for entity highlighting.
• Batch rerunning of pipelets with support for entities.

Added on August 13, 2018 (build 2472 – patch 12)

• Entity highlighting fixes.
• Frontend service support for entity highlighting.

Added on August 15, 2018 (build 2473 – patch 13)

• Fix for enrichment rerun studio plugin.

Added on September 10, 2018 (build 2476 – patch 14)

• Fix Heat Map widget update.
• Widgets API: Add support for 'detail' event on Result List with modal.
• Map Widget: Satellite mode not working on IE11.
• Map Widget: Switching from satellite to map, does not respect coordinates.

Added on September 12, 2018 (build 2477 – patch 15)

• Fixed field selection in pipelet rerun.

Added on September 14, 2018 (build 2478 – patch 16)

• Add support for custom splash screen.

Added on October 2, 2018 (build 2479 – patch 17, 18)

• Bump bulk ES timeout to 60 seconds.
• Fix scrollbar on tabs component.
• Stop emitting selections from bookmarks in certain modes.

Added on November 6, 2018 (build 2482 – patch 19)

• SB-120 - Disable Highlight not working on Facet Table Widget
• SB-140 - Authentication isn't renewed after the 10 minute default validity interval with SAML
• SB-150 - 1000's Separator in Dashboard
• SB-155 - Disable Highlight not working on Facet Table Widget

Installing and Upgrading

Fresh Installation Instructions

yum update jre
yum install jre # if the `yum update` command returns this message `no packages marked for update`
yum update squirro-storage-node-users
yum update squirro-storage-node
yum update squirro-cluster-node-users
yum update squirro-*

yum update jre
yum install jre # if the `yum update` command returns this message `no packages marked for update`
yum update squirro-storage-node-users
yum update squirro-storage-node

yum update jre
yum install jre # if the `yum update` command returns this message `no packages marked for update`
yum update squirro-cluster-node-users
yum update squirro-*

...