Squirro 3.1.0 - Release Notes

Owned by Moritz Müller
Last updated: Nov 06, 2020 by Wiktor Starczewski
8 min read


We're excited to announce Squirro 3.1.0, released on 20th of July 2020.

Contents - What's in the release?

New Features

Widget Editor 

  • It is now possible to permanently associate widgets with GitHub repositories for a deeper level of integration
  • The GitHub flow has been reworked, allowing quickly detecting changes between the widget and the repo

Layout Master in Squirro Dashboards

We have added a new concept "Layout Master" to this release. Please refer to Layout Master Dashboard for more details.

  • Added option to replicate the precise master section setup in any dashboard through the Sync functionality
  • Added option to align dashboard tabs with the center section


Synonyms  

Starting with Squirro 3.1.0 release, we have made the support for Synonyms for your search requests even more powerful. While it has already been possible to use synonyms with specific facets using our APIs for a few years, we have also made it possible now to use them on the Title, Body as well as the Abstract of your Squirro Items. There is more, you can do all of that now finally over the Frontend with just a few clicks. If you are interested in knowing more about it, please read our Using Synonyms documentation page.

Spellcheck 

  • Squirro 3.1.0 exposes the ability to spellcheck user queries based on the stored data in Squirro.
  • For further information please see the documentation page Spellchecking BE.

Significant Terms Extractor

Given a set of documents, the goal of significant terms extractor is to identify in the text the set of words that can potentially describe a document. It is an unsupervised technique that constantly adapts itself at every call to maintain the model always up to date.

For further information please refer to Significant Terms Extraction in Squirro

ElasticSearch 7.7

There has been a major migration of the Elasticsearch engine that is supported by Squirro. We have moved from using the version 6.2.2 to 7.7.0.

We were careful not to introduce new features per-se during the migration. It did, however, facilitate the deployment of extra functionality to our platform like encrypted connectivity with ES (see below)

Security Improvements (ES, MariaDB) 

With Squirro 3.1.0 you have the possibility to encrypt Squirro communication with MariaDB and Elasticsearch.

For further information please refer to Securing Elasticsearch and MariaDB over TLS

Improved Entities Support 

Starting with Squirro 3.1.0 release, Squirro entity data-model has become even more powerful. It now allows saving for more than one value in a specific entity property. What does all of that even mean (wink). It means that now you can finally add multiple companies to a single entity. In addition to that, we have also made it possible to add more than one entities to a single sentence. This will help you catch all those sentences which do not quite fit one single concept in your business. 

Please note that this is a breaking change. If you have code which directly creates entities using our APIs, please contact Squirro Support.

Layer arrangement (composite layers)  

  • Allowed to change the default widget arrangement on a layer
  • Three widget arrangements available: Grid (default for Squirro 3.0 and below), Accordion and Carousel
  • Accordion arrangement packs all widgets on a layer into an accordion style control
    • Allows for creation of menu-style navigation as found on Squirro Explore page
  • Carousel arrangement packs all widgets on a layer into a carousel style control
    • Allows displaying multiple widgets in the same visual space without the need for a layer-based implementation

Profile Avatar

  • Users can now customise their avatar (profile picture)
  • Exposed the user avatar into Custom Widget API

Squirro Dashboard SDK

  • The SDK now parses well under strict mode, allowing for use in environments that require it (like Salesforce API v40+)
  • Optimised several techniques used in the Dashboard component, allowing usage under Salesforce Lightning

Restrict Dashboard width

  • For better UX, config. option under app n navbar settings to select max. width for dashboard resize, beyond which white space grows on either side.
  • Options: 1024, 1366, 1440 and 1680 to select from so that they make conscious decisions wrt. the devices they use.


Improvements

  • Sentence Splitting improvement 
  • Card shadows, font sizes, border radius improvements
  • Table widget now configurable by multiple facets
  • Bigger searchbar, tabs, buttons, etc. as 1 unit height restriction on certain widgets removed
  • Configurable tooltips for the icons displayed inside link widget and facets widget
  • Facet dropdown multi-select: The buttons are always visible now.


Bug Fixes

 Click here to see most relevant Bug Fixes & Improvements
  • SB-136 - Rich text formatting issues
  • SB-313 - Can't type 's' on RTE widget when having an item detail widget
  • SB-349 - Small language change for email validation
  • SB-370 - Support array on userPref
  • SB-395 - RTE widget content is not scrollable
  • SB-396 - Breadcrumb widget action chips always visible
  • SB-413 - Users page number of results dropdown doesn't always work
  • SB-414 - Request demo url is wrong
  • SB-415 - Catalyst mapping in incorrect classification feedback popup is not working correctly!
  • SB-416 - Selection tagging is removing consecutive spaces from sentence
  • SQ-10748 - Force sending empty array when all tags are removed from the single facet
  • SQ-10751 - Cards Widget: Padding on CTA is wrong
  • SQ-10880 - Entities highlights not working on result widget with only entities enabled
  • SQ-10886 - implement-multi-value-selection-for-facets-list
  • SQ-11172 - Replacing long URLS with only the domain name or source name in case of PDF
  • SQ-11219 - Users List filter dropdown doesn't always work
  • SQ-11261 - Facets List Widget: Facet List Order config does not work
  • SQ-11265 - Tag Cloud widget: Legend is always empty
  • SQ-11266 - Trend Widget: Y axis label is in the wrong side
  • SQ-11277 - Direct access to search tagging edit url fails
  • SQ-11283 - Sort loaders alphabetically
  • SQ-11311 - Line height for share item is wrong
  • SQ-11319 - Link widget delete icon issues
  • SQ-11370 - Do not limit Link widget to 4 columns
  • SQ-11376 - Smartfilter widget does not render in edit mode
  • SQ-11410 - Topic cluster filtering is not working
  • SQ-11571 - Prevent fetching recommendation catalysts on item detail if not needed
  • Only allow master dashboard promotion and demotion for admins


Added on July 24, 2020 (build 36 - Patch 1)

  • Fix loading of themes

Added on August 11, 2020 (build 38 - Patch 2)

  • Added multi-value properties for entities and multiple entities per sentence.
  • Adjusted entity UI to allow User or Admin mode for displaying of entity popups.

Added on October 8, 2020 (build 41 - Patch 4)

  • Increased file uploader timeout from 30 seconds to 10 minutes.

Added on October 12, 2020 (build 42 - Patch 5)

  • SB-487- Fix broken search after a project reset in a project with synonyms
  • SB-467 - Error re-running the pipelets with entities
  • SSO & SAML improvements

  • SB-472 Terms extractor fixes

    • Python 3 compatibility

    • Performance improvements

    • NLTK language models are downloaded if not available

  • Add support for german language

Added on October 15, 2020 (build 43 - Patch 6)

  • SB-484 - Fixed wrong highlighting for PDF (duplicated words)
  • Exposed base Collections and Models globals to custom widgets
  • Prevent against code breakage due to scenarios where a project has None synonym_Config

Added on October 19, 2020 (build 44 - Patch 7)

  • Added and documented new Widget API method, getCustomEvents()

Added on November 5, 2020 (build 45 - Patch 8)

  • Facets Bar Chart: Do not cache models to allow collection tweaking

Added on November 6, 2020 (build 46 - Patch 9)

  • Fix aggregation collections caching in Pie Chart

Fresh Installation Instructions

Please follow the regular installation steps

Upgrade Instructions

Please ensure that your current version is 3.0.0 If you are on a version older than 3.0.0, please contact support.

With this & every subsequent upgrade, all services will run in python3.6.

If you have not migrated & tested your custom dataloader plugins, studio plugins or piplets to python3.6 compatible code, you can still manually revert to run specific services in python2.7 by following the steps outlined here.

This is the last release which continues python 2.7 support. Therefore, we strongly recommend that you migrate any custom code now and proceed with the new default python 3.6.



 1. Upgrade Storage Nodes and Cluster Nodes collocated on the same machine/VM
RHEL6 / CentOS6 / RHEL7 / CentOS7
# stop squirro
squirro_stop

yum install java-1.8.0-openjdk
yum update java-1.8.0-openjdk
yum update squirro-storage-node-users

# Ensure that `path.data` variable is set in ES. If you have used Ansible in the past to deploy Squirro, you will be affected by this bug.
grep -q "^path.data" /etc/elasticsearch/elasticsearch.yml ||  echo "path.data: /usr/share/elasticsearch/data/" >> /etc/elasticsearch/elasticsearch.yml

# update elasticsearch
# Make sure that you delete any indices which have been created with squirro_v8 template. 
# curl http://localhost:9200/_cat/indices -> then inspect for occurrences of squirro_v8 in the output.
# If your cluster has been created with Squirro 2.6.0, released on April 3rd 2018 or later, 
# you do not have to worry about this.
# To delete: curl -XDELETE http://localhost:9200/squirro_v8*
# To delete old jaeger indices: curl -XDELETE http://localhost:9200/jaeger*

systemctl stop elasticsearch
yum update elasticsearch

# Disregard the ES rpmnew warnings as they will be resolved by `squirro-storage-node` cluster

# update storage node
yum update squirro-storage-node
# wait for elasticsearch status to become green, only then
# update cluster node
yum update squirro-cluster-node-users
yum update squirro-*
squirro_start

Resolve all the `.rpmnew` files in `/etc/squirro`. This process involves merging the changes between the `.ini` (e.g., `storage.ini`) and '.ini.rpmnew`(e.g., `storage.ini.rpmnew`) files and then eventually deleting the `.ini.rpmnew` files. Finally, restart the services with merged `.ini` files (or just use `squirro_restart` to restart all services).

 2. Upgrade Storage and Cluster Nodes when they are on different servers (and there is only one storage node and one cluster node)

Stop squirro on the cluster node

RHEL6 / CentOS6 / RHEL7 / CentOS7
squirro_stop

Upgrade the storage node by running:

RHEL6 / CentOS6 / RHEL7 / CentOS7
yum install java-1.8.0-openjdk
yum update java-1.8.0-openjdk
yum update squirro-storage-node-users

# Ensure that `path.data` variable is set in ES. If you have used Ansible in the past to deploy Squirro, you will be affected by this bug.
grep -q "^path.data" /etc/elasticsearch/elasticsearch.yml ||  echo "path.data: /usr/share/elasticsearch/data/" >> /etc/elasticsearch/elasticsearch.yml

# update elasticsearch
# Make sure that you delete any indices which have been created with squirro_v8 template. 
# curl http://localhost:9200/_cat/indices -> then inspect for occurrences of squirro_v8 in the output.
# If your cluster has been created with Squirro 2.6.0, released on April 3rd 2018 or later, 
# you do not have to worry about this.
# To delete: curl -XDELETE http://localhost:9200/squirro_v8*
# To delete old jaeger indices: curl -XDELETE http://localhost:9200/jaeger*

systemctl stop elasticsearch
yum update elasticsearch

# Disregard the ES rpmnew warnings as they will be resolved by `squirro-storage-node` cluster


#update storage node
yum update squirro-storage-node

# wait for elasticsearch status to become green, only then update the clusternode

Upgrade the cluster node by running:

RHEL6 / CentOS6 / RHEL7 / CentOS7
yum install java-1.8.0-openjdk
yum update java-1.8.0-openjdk
yum update squirro-cluster-node-users
yum update squirro-*
squirro_start

Resolve all the `.rpmnew` files in `/etc/squirro`. This process involves merging the changes between the `.ini` (e.g., `storage.ini`) and '.ini.rpmnew`(e.g., `storage.ini.rpmnew`) files and then eventually deleting the `.ini.rpmnew` files. Finally, restart the services with merged `.ini` files (or just use `squirro_restart` to restart all services).

 3. Upgrade multi-node clusters (multiple Storage Nodes and/or multiple Cluster Nodes)

Stop squirro on all cluster nodes

RHEL6 / CentOS6 / RHEL7 / CentOS7
squirro_stop

Stop elasticsearch on all storage nodes

RHEL6 / CentOS6 / RHEL7 / CentOS7
# Make sure that you delete any indices which have been created with squirro_v8 template. 
# curl http://localhost:9200/_cat/indices -> then inspect for occurrences of squirro_v8 in the output.
# If your cluster has been created with Squirro 2.6.0, released on April 3rd 2018 or later, 
# you do not have to worry about this.
# To delete: curl -XDELETE http://localhost:9200/squirro_v8*
# To delete old jaeger indices: curl -XDELETE http://localhost:9200/jaeger*

systemctl stop elasticsearch

Upgrade every storage nodes (one by one) by running:

RHEL6 / CentOS6 / RHEL7 / CentOS7
yum install java-1.8.0-openjdk
yum update java-1.8.0-openjdk
yum update squirro-storage-node-users

# Ensure that `path.data` variable is set in ES. If you have used Ansible in the past to deploy Squirro, you will be affected by this bug.
grep -q "^path.data" /etc/elasticsearch/elasticsearch.yml ||  echo "path.data: /usr/share/elasticsearch/data/" >> /etc/elasticsearch/elasticsearch.yml

# update elasticsearch
yum update elasticsearch

# Disregard the ES rpmnew warnings as they will be resolved by `squirro-storage-node` cluster

# update storage node
yum update squirro-storage-node

# wait for elasticsearch status to be green before continuing with next storagenode

Upgrade each cluster nodes by running:

RHEL6 / CentOS6 / RHEL7 / CentOS7
yum install java-1.8.0-openjdk
yum update java-1.8.0-openjdk
yum update squirro-cluster-node-users
yum update squirro-*
squirro_start

Resolve all the `.rpmnew` files in `/etc/squirro`. This process involves merging the changes between the `.ini` (e.g., `storage.ini`) and '.ini.rpmnew`(e.g., `storage.ini.rpmnew`) files and then eventually deleting the `.ini.rpmnew` files. Finally, restart the services with merged `.ini` files (or just use `squirro_restart` to restart all services).