Squirro 2.3.2 - Release Notes

Owned by Felix Hürlimann (Deactivated)
Last updated: Jun 13, 2016

Released on April 6, 2016.

New Features

Improvements

  • Search bar alignment and buttons improved

  • Better HTTP and object cache management

  • Generate a stable item identifier to improve bulk loading

  • Remove Save option from Search Bar widget

  • Remove foreground limit for significant terms widget

Bug Fixes

  • Fix sub items view in dashboard modal view

  • Set correct item creation date for bulk loading

  • Fix double quotes handling in facet values

  • Fix highlighting for proximity searches

  • Fix field search for language specific fields (e.g. title or body)

  • Use correct facets for heat-map facet value formatting

Updates

  • Updated Elasticsearch to version 2.2

Fresh Installation Instructions

When you follow the installation steps, please perform the following while you install the storage node. After you install the storage node, but before you bring up Elastic Search, do the following:

For a fresh installation on a single node, the number_of_shards and number_of_replicas need to be adjusted to the /etc/elasticsearch/elasticsearch.yml configuration file:

# set number of shards to two
root$ echo index.number_of_shards: 2 >> /etc/elasticsearch/elasticsearch.yml

# set number of replicas to zero for single-node squirro setup
root$ echo index.number_of_replicas: 0 >> /etc/elasticsearch/elasticsearch.yml

In addition for fresh installations the following four elasticsearch related permissions need to be set manually like so:

root$ chmod 644 /etc/elasticsearch/elasticsearch.yml
root$ chmod 644 /etc/elasticsearch/logging.yml
root$ chown elastic:elastic /var/lib/elasticsearch
root$ chown elastic:elastic /var/log/elasticsearch
root$ chown elastic:elastic /var/run/elasticsearch

These issues will be addressed with the upcoming release 2.3.3 .

Upgrade Instructions

As this version contains an update to a new major version of Elasticsearch, it is only possible to directly upgrade from a Squirro version bigger than 2.3. If you are currently running a version smaller than 2.3 please contact support. 

This is not the latest version of Squirro. To upgrade to this version, please ensure that you point the squirro yum repository to version '2.3.2' and not to 'latest'.

If you are using Squirro in a Box, then there are additional steps involved. In this case, please contact support.

 

Step 1: Prepare the upgrade

It is not possible to update to this version without a service interruption.

The upgrade will re-index your data, therefore you need at least 50% of the disk space available on your storage nodes.

The order of the following steps is important, do not skip any step. If a step fails, do not continue before resolving the issue.

Step 2: Stop Squirro

On each cluster node, run:

root$ monit stop sqclusterd

Step 3: Update the Elasticsearch templates

On each storage node, one after the other, run:

root$ rpm -Uvh --nodeps $(repoquery --location squirro-elasticsearch-templates)

This will update the templates and re-index all indices with the new mapping v6. This may take a while.  

Warning: Do not use yum update squirro-elasticsearch-templates as this tries to update Elasticsearch at the same time.

Step 4: Update Elasticsearch

 

root$ rpm -Uvh --nodeps $(repoquery --location elasticsearch)

Elasticsearch will not start as there are configuration conflicts between the two versions that need to be fixed manually:

 

warning: /etc/elasticsearch/elasticsearch.yml created as /etc/elasticsearch/elasticsearch.yml.rpmnew
warning: /etc/init.d/elasticsearch created as /etc/init.d/elasticsearch.rpmnew
warning: /etc/sysconfig/elasticsearch created as /etc/sysconfig/elasticsearch.rpmnew
warning: /usr/lib/systemd/system/elasticsearch.service created as /usr/lib/systemd/system/elasticsearch.service.rpmnew

 

Networking behaviour changed with Elasticsearch 2. In a multi-storagenode setup, make sure that the elasticsearch cluster network configuration is set up correctly. In a standard setup, adding the following to the /etc/elasticsearch/elasticsearch.yml file should work:

 

network.bind_host: _site_,_local_
network.publish_host: _site_

For further help, please consult https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

File permissions:

  • Ensure /etc/elasticsearch/* is readable by the elastic user. 
  • Ensure that /var/log/elasticsearch/ is readable and writable by the elastic user.

Resolve these issues.

Remove the ES plugins as they are no longer compatible (new ones will be installed by yum update):

root$ rm -rf /usr/share/elasticsearch/plugins/*

And last, index the percolation queries that could not be migrated automatically (only required if the file /tmp/elasticsearch/squirro_v5_filter.out exists):

root$ curl -s -XPOST http://localhost:9200/_bulk --data-binary "@/tmp/elasticsearch/squirro_v5_filter.out"

 

Step 5: Update Elasticsearch plugins and all other packages

root$ yum update 

Depending on your configuration, Elasticsearch might not come up until you updated a second storage node as it requires at least two nodes to be present.
Depending on your index size, the initial start of elasticsearch service will take a while as it migrates the index internally. 

Step 6: Upgrade Cluster Nodes

On each cluster node, run:

root$ yum update
root$ yum reinstall squirro-python-squirro.api.topic

Resolve /etc/squirro/topic.ini.rpmnew

 

If you run Squirro in a multi-cluster-node environment you need to do the following additional steps on each cluster node:
root$ mkdir -p /mnt/gv0/widgets
root$ chown -R sqtopic:squirro /mnt/gv0/widgets
root$ sed -e 's|^custom_widgets_directory = .*|custom_widgets_directory = /mnt/gv0/widgets/|' -i /etc/squirro/topic.ini
root$ sed -i -e 's/db_endpoint_discovery = false/db_endpoint_discovery = true/' /etc/squirro/topic.ini
root$ if [[ ! -L "/var/lib/squirro/topic/widgets" && -d "/var/lib/squirro/topic/widgets" ]]; then rm -ir /var/lib/squirro/topic/widgets; fi
root$ ln -s /mnt/gv0/widgets /var/lib/squirro/topic/widgets
root$ service sqtopicd restart

 

For the cluster node (or in a multi-cluster-node environment all cluster nodes), please run the following steps needed to exercise the new custom widgets extension with release 2.3.2

root$ CUSTOM_WIDGET_SYMLINK="/opt/squirro/virtualenv/lib/python2.7/site-packages/squirro/integration/frontend/userapp/static/min/custom"
root$ if [[ -d "$CUSTOM_WIDGET_SYMLINK" && ! -L "$CUSTOM_WIDGET_SYMLINK" ]]; then rm -rf "$CUSTOM_WIDGET_SYMLINK"; fi
root$ ln -s /var/lib/squirro/topic/widgets "$CUSTOM_WIDGET_SYMLINK"

Then start Squirro again:

root$ monit start sqclusterd
root$ monit restart sqfrontendd