Squirro 2.5.2 - Birch - Release Notes

Released on October 17th, 2017

Introducing Squirro 2.5.2 - Birch

We're happy to release Squirro version 2.5.2, using elasticsearch 5.6.0!

Updates

  • This release mainly updates the Elasticsearch core component from version 2.2 to version 5.6.

Improvements

  • Exact match on queries containing stopwords for phrase searches
  • More relevant search results: we are using BM25 (which is supposed to work better for short fields) instead of classic TF-IDF for ranking.
  • Improved Pipeline 2.0 concurrency on large sources by removing source-wide lock (added with build 2.5.2-4024 on October 25, 2017)
  • Provided Squirro in a Boxes with OS user interface. The same *.ova files can now be used for both VMware and VirtualBox (added on December 11, 2017)

Bug Fixes

  • Numerous bug fixes and improvements to UI based on feedback to the 2.5.1 release

  • Added guidance on behalf of Pipeline 2.0 on how to "opt in" to additional built-in enrichments to /etc/squirro/provider.ini (added with build 2.5.2-4024 on October 25, 2017)
  • Offline installation fix for Redhat 6 upgrading python to 2.7.13 on our mirror (added with build 2.5.2-4024 on October 25, 2017) and added additional missing dependency packages for Redhat 6 offline installation (added on December 12, 2017)

Fresh Installation Instructions

Please follow the regular installation steps

Upgrade Instructions

Please ensure that your current version is 2.5.1. If you are on a version older than 2.5.1, please contact support.

Upgrading will need to reindex old elasticsearch indexes to new indexes with template version v8. This will need few hours in case you have an old big index in template v7 (A reindex on an index of 1'337'000 documents , 16GB in size took 57 minutes)

Additionally if you are using Squirro in a Box, additional steps are involved. In this case we also ask you to contact support.

From earlier versions 2.5.2-4031 or lower

If you have already upgraded to 2.5.2 before 9th November 2017 and you are intending to apply a patch release of storage node, after normal steps of upgrading storage node, you please run this command:

bash /opt/squirro/elasticsearch/update/migrate-storage-node.sh

From version 2.5.1

1. Upgrade Storage Nodes and Cluster Nodes collocated on the same machine/VM

CentOS 6 / RHEL 6CentOS 7
for service in `ls /etc/monit.d/sq*d | sed -e "s|^.*/||" | grep -v "sqclusterd"`; do monit stop $service; done
# wait for `monit summary` to indicate that all but 5 services are stopped
yum update squirro-storage-node-users
yum update elasticsearch
yum update squirro-storage-node
bash /opt/squirro/elasticsearch/update/migrate-storage-node.sh
# wait until migrate-storage-node.sh finished
yum update squirro-cluster-node-users
yum update squirro-*
monit monitor all
cd /lib/systemd/system
for service in $(ls sq*d.service); do echo "Stopping $service"; systemctl stop $service; done
yum update squirro-storage-node-users
yum update elasticsearch
yum update squirro-storage-node
systemctl daemon-reload
bash /opt/squirro/elasticsearch/update/migrate-storage-node.sh
# wait until migrate-storage-node.sh finished
yum update squirro-cluster-node-users
yum update squirro-*
for service in $(ls sq*d.service); do echo "Starting $service"; systemctl start $service; done


2. Upgrade Storage and Cluster Nodes when they are on different servers (and there is only one storage node and one cluster node)

On the one cluster node, shut down most of the Squirro services like so:

CentOS 6 / RHEL 6CentOS 7
for service in `ls /etc/monit.d/sq*d | sed -e "s|^.*/||" | grep -v "sqclusterd"`; do monit stop $service; done
# wait for `monit summary` to indicate that all but 5 services are stopped
cd /lib/systemd/system
for service in $(ls sq*d.service); do echo "Stopping $service"; systemctl stop $service; done
# the output of following statement should indicate that all sq*d services are stopped:
for service in $(ls sq*d.service); do echo "Status of $service"; systemctl status $service; done

Upgrade the one storage nodes by running:

CentOS 6 / RHEL 6CentOS 7
yum update squirro-storage-node-users
yum update elasticsearch
yum update squirro-storage-node
bash /opt/squirro/elasticsearch/update/migrate-storage-node.sh
# wait until update-storage-node.sh finished
yum update squirro-storage-node-users
yum update elasticsearch
yum update squirro-storage-node
systemctl daemon-reload
bash /opt/squirro/elasticsearch/update/migrate-storage-node.sh
# wait until migrate-storage-node.sh finished

Upgrade the one cluster node by running:

CentOS 6 / RHEL 6CentOS 7
yum update squirro-cluster-node-users
yum update squirro-*
monit monitor all
yum update squirro-cluster-node-users
yum update squirro-*
cd /lib/systemd/system
for service in $(ls sq*d.service); do echo "Starting $service"; systemctl start $service; done
# wait for the following statement to indicate that all sq*d services are started
for service in $(ls sq*d.service); do echo "Status of $service"; systemctl status $service; done

3. Upgrade multi-node clusters (multiple Storage Nodes and/or multiple Cluster Nodes)

Upgrading clusters of Squirro nodes to release 2.5.2 is very involved. Please contact Squirro support for assistance.

On each cluster node, shut down most of the Squirro services like so:

CentOS 6 / RHEL 6CentOS 7
for service in `ls /etc/monit.d/sq*d | sed -e "s|^.*/||" | grep -v "sqclusterd"`; do monit stop $service; done
# wait for `monit summary` to indicate that all but 5 services are stopped
cd /lib/systemd/system
for service in $(ls sq*d.service); do echo "Stopping $service"; systemctl stop $service; done
# wait for the following statement to indicate that all but 5 services are stopped
for service in $(ls sq*d.service); do echo "Status of $service"; systemctl status $service; done

Stop elasticsearch service on every storage nodes, check that no elasticsearch service running:


CentOS 6 / RHEL 6CentOS 7
service elasticsearch stop
ps -ef | grep elasticsearch
systemctl stop elasticsearch
ps -ef | grep elasticsearch

Upgrade every storage nodes by running:

CentOS 6 / RHEL 6CentOS 7
yum update squirro-storage-node-users
yum update elasticsearch
yum update squirro-storage-node-users
yum update elasticsearch

and then on every storage node:

CentOS 6 / RHEL 6CentOS 7
yum update squirro-storage-node
systemctl daemon-reload
yum update squirro-storage-node



Migrate data from old elasticsearch template v7 index to new elasticsearch index template v8. Run this on ONLY ONE storage node.

CentOS 6 / RHEL 6CentOS 7
bash /opt/squirro/elasticsearch/update/migrate-storage-node.sh
# wait until update-storage-node.sh finished
bash /opt/squirro/elasticsearch/update/migrate-storage-node.sh
# wait until migrate-storage-node.sh finished

New ES does not allow to run some plugins likes head or kopf anymore, therefore you need to start them on your local machine and use port forward to a storage node to connect  to ES. We recommend you to use an addon to see status of ES cluster, index and shards during upgrade when you have multiple nodes setup. For example if you use  cerebro:

Installation:

If you use port 9500 as port forward to ES node, then use: ssh -L 9500:storagehost:9200. After that you can connect cerebro to http://localhost:9500

Upgrade each cluster nodes by running:

CentOS 6 / RHEL 6CentOS 7

First run the following on all cluster nodes one at a time:

yum update squirro-cluster-node-users
yum update squirro-python-squirro.service.cluster

Followed by running the following on all cluster nodes one at a time:

yum update squirro-*
monit monitor all
yum update squirro-cluster-node-users
yum update squirro-*
for service in $(ls sq*d.service); do echo "Starting $service"; systemctl start $service; done

4. Elasticsearch settings

  • ElasticSearch memory is set in /etc/elasticsearch/jvm.options, not in /etc/sysconfig/elasticsearch anymore.

  • To change number of shards and replicas for your cluster, you have to update settings in the templates before creating index:
 # edit /etc/elasticsearch/templates/squirro_v8.json, change values:

"settings": {
   "index": {
       "number_of_shards": 6,
       "number_of_replicas": 0,
       ...
   }
}

$ ./etc/elasticsearch/templates/ensure_templates.sh
# check that templates is set correctly:
$ curl 'http://localhost:9200/_template?pretty'

# after finishing running 
# check that templates is set correctly:
$ curl 'http://localhost:9200/_template?pretty'