Squirro 2.6.3 - Release Notes

Owned by Moritz Müller
Last updated: Apr 30, 2019 by Saurabh Jain
13 min read

We're excited to announce Squirro 2.6.3, released on January 22nd 2019.

Known upgrade limitations:

  • If you are using custom widgets based on the horizontal result widget in one of your projects, do not update to 2.6.3 yet, but please contact support. to clarify required steps.

Contents -What's in the release?

New Features

Data Sources

  • New Overview of existing Data Sources:
    • See scheduling of data loading with information about "Last Run" and "Next Run"
    • See the Number of items that exist for this data source.
    • Source Status added: Queued, Running, Error, Complete
    • New action buttons: Run, Stop, Reset, Explore added. Explore allows you to explore the items coming from a data source on the Explore dashboard.
  • Data Source Types:
    • Support of new File Importer Plugin:
      • Upload of a ZIP File containing multiple PDF and Microsoft Office documents
      • Support of Frontend data loading for PDF and Microsoft Office documents.
    • Removal of old data source types that are no longer supported.
      • (Previous data loader plugins are still available as legacy versions and will be supported on request.)
  • Replacement of Providers by Data Loader Plugins (more information below)

Entities

  • With 2.6.3 the use of entities is significantly improved throughout Squirro.
  • Entities are now visible in the Explore screen.
  • In the Explore screen the item display can be switched between showing the abstract of the item or the related entities.
  • Addition of entity filters to Explore screen sidebar and Dashboard sidebar.
  • Addition of new Entities widget.
  • Entity Highlighting: Jump to entity extract within the item view when selecting respective entity.

Machine Learning

  • Major improvement of Annotation Workbench for Model training (typeahead, keyword support, etc.)
  • Numerous enhancements of machine learning service to simplify and extend functionality (see below in section: New libNLP steps and features.)
  • Placement of ML tasks under TRAIN tab. (Previously in Project Settings).
  • Addition of "Run Now" Button to ML Job Page.
  • New squirro_client method: get_machinelearning_workflow_assets (See documentation here.)

Widgets & Dashboard

New Widgets

  • Divider: Show a line to separate areas of the dashboard and add section titles
  • Sample Text Labeller:The Sample Text Labeller widget allows to do an inference test to assert the performance of a given Machine Learning Workflow.
  • Item Detail: The item detail widget shows the contents of the active Squirro item. It can also be controlled by the dashboard store to display the item whose ID is stored there.
  • Entities: The entities widget display all the entities available in the project, in a dropdown fashion similar to Facets widget.

Widget Improvements and Changes

  • Merge of Horizontal Result List into Result List Widget:

    • Option to switch between horizontal and vertical mode in result list widget.
  • Widget Static State / Drilldown Disabled State
    • We have an option to disable drilldown on some widgets, in which case, widget does not react to any query except its own.  In such case, widgets visually adopt a static state which highlights only the value selected on them.
  • Result List Widget
    • New option to change between vertical (default) and horizontal mode (previously horizontal result widget)
    • Open item detail in place. Designed for efficient use of screen real estate, it opens the item detail in place of the result list. Cross button allows you to navigate back to the list in a single click.
    • Config. option to hide item metadata and abstract. Now display only the titles of the item for a quick scan through the list.
  • All Result displaying Widgets (Result List, Cards, Facet Table, Kanban Lane)
    • Selection of Admin vs. End User mode for the visibility of facets. In Admin Mode all facets are visible and editable; whereas in End User Mode only the pre-defined facets are shown to the user, offering a cleaner look for the item detail.
    • New config. option to toggle filtering by item tags.
    • Display only the value from a key value pair inside the item tag UI element.
    • Enjoy seamless infinite scroll on mobile dashboards. If the bottom-most widget on a mobile dashboard is a scrollable widget, namely Result List, Cards, Table and Facet Table widget, the infinite scroll is activated.

Dashboard & Layer

  • Option to copy & paste widgets across all layers and dashboards within a project for both desktop and mobile mode of the dashboard editor.
  • Option to copy & paste layers across all dashboards within a project or both desktop and mobile version of the dashboard editor.
  • Numerous bug fixes of dashboard & layer functionalities.

Improvements

New libNLP steps and features

  • Query multiple Squirro instances in Squirro query loader
  • Threshold filter step for predicted facet
  • Squirro entity filter to create Squirro entities
  • Tracking facet to record which model has been applied
  • Smartfilter classifier
  • Regex filter

Return of "Import Project" Functionality

  • "Import Project" was integrated into Project Template in previous release causing a loss of certain functionalities, so we brought the "Import Project" functionality back that allows you to import selected features.

Refactoring of Services

  • Topic Service
  • Dataloader Service
  • Update of Ingester Service to account for refactoring of Topic & Dataloader Service.

Major Code Cleanup

  • Removal of significant amount of legacy code
  • Removal of deprecated assets

Continued integration of project template functionality

  • Simpler and more enhanced project template functionality.
  • Significantly reduced setup time of projects.
  • Possibility to ship project templates (predefined data sources, dashboards, classification jobs (ML tasks), etc.)

Other Improvements

  • Update of language detection module (Now language detection with ML-based library: fastText)
  • Change of internal data schema of MySQL database (removal of unnecessary fields, etc.)
  • Addition of squirro_status CLI Tool
  • Update of Highcharts library to version v6.2.0
  • Update of several other libraries to the latest versions
  • Capability to use user-specific parameters for machine learning & training.

Bug Fixes & Improvements

 Click here to see most relevant Bug Fixes & Improvements
  • SB-30 - Delimiter field disappears and created facet does not show up.
  • SB-49 - No error message(s) when using an incorrect password during login on a mobile device.
  • SB-67 - Single layer autosize to fit frame: Widgets do not resize on browser window resize
  • SB-69 - Dashboard edit: Adding a new widget shows the properties of the last selected widget
  • SB-79 - Warning message appears on navigating away from a dashboard, even when no changes are made
  • SB-107 - Result list widgets highlighting SF even when they shouldn't
  • SB-116 - Timeline widget selection state doesn't go away after removing the filter.
  • SB-132 - Thumbnail position is wrong
  • SB-135 - Breadcrumb in hidden layer is always collapsed
  • SB-137 - The default "Home" chip in the breadcrumb widget is not configurable.
  • SB-140 - Authentication isn't renewed after the 10 minute default validity interval with SAML
  • SB-146 - Typeahead facet display issue
  • SB-150 - 1000's Separator issue in Dashboard
  • SB-164 - Unable to restart walkthrough using the side bar right after finishing it
  • SB-165 - Slow loading of data on Table Widget onClick
  • SB-166 - Weird scrollbars on Table Widget when columns are small
  • SB-168 - Improvements in Recommendation widget explanation and fix of displaying issues.
  • SB-170 - Breadcrumb (Chips) not matching the query terms
  • SB-179 - "Matching pages" column shown without content if highlighting is disabled
  • SB-193 - Facet widget selections not cleared on Reset "Clear all"
  • SB-197 - Create project sidebar closes itself when I resize browser
  • SB-204 - Entity list window does not continue to bottom of item
  • SB-205 - Data labeling lane widget is overflowing
  • SB-206 - Replace overflow-y:scroll with overflow-y:auto so the scrollbars are not visible all the time
  • SB-220 - Deleting a layer says 'section' instead of layer in confirmation modal
  • SB-221 - Link widget: Custom frame input field should not be visible when the widget is controlled by the dashboard store
  • SQ-8280 - Result list widgets: Have config options to enable / disable metadata info and abstract.
  • SQ-9126 - 'Clear All' button should appear only when some filter is applied
  • SQ-9244 - Add more data loader mapping options
  • SQ-9313 - Trend widget config: When no trend is setup, show helper text inside the dropdown
  • SQ-9325 - Map widget config: When no geo-point facets are created, show helper text inside the dropdown
  • SQ-9329 - Barchart widget: Rename 'Facet-X Axis' label to 'Facet'
  • SQ-9566 - Item detail: Icons list jumps on click on tag, body icons
  • SQ-9668 - IE11: In widget browser, overflowing widget names should be displayed in two line
  • SQ-9728 - Have option to duplicate widget across layers for both desktop and mobile version of a dashboard
  • SQ-9735 - Wide modal view breaks on resize
  • SQ-9877 - Implement Walkthrough for CSV Data Loader
  • SQ-9966 - Have option to hide the keywords from the item detail
  • SQ-9988 - Add Entities Filter to Sidebar and Dashboard Sidebar
  • SQ-10007 - Append user_id to email of deleted user (so that a new user can be made with the same email)
  • SQ-10004 - Data labeling lane is showing empty pill when there are no tags
  • SQ-10020 - Implement Walkthrough for Excel Data Loader
  • SQ-10036 - Filter list on dashboard sidebar has wrong paddings
  • SQ-10038 - Filters keep reloading on expanding/collapse
  • SQ-10041 - Allow to define the maximum file size for uploads in configuration.ini
  • SQ-10043 - Dashboard widgets' containers have wrong size until they are fully loaded
  • SQ-10048 - Widgets in disable drilldown mode keep appending filters to history
  • SQ-10062 - Create Adhoc Inference Widget
  • SQ-10070 - Show number of items loaded per source
  • SQ-10072 - Add icon to item detail toolbar to generate direct link to items
  • SQ-10073 - Add link to explore items loaded by a given source
  • SQ-10077 - Project Template, manual refreshing of the browser is not acceptable
  • SQ-10082 - Keywords translation is gone
  • SQ-10083 - Tags should highlight before being removed (double backspace)
  • SQ-10084 - Heatmap values are overflowing its cell space
  • SQ-10085 - Item detail arrows move while the item loads
  • SQ-10086 - Item detail content is using full width, should keep the 700px limit
  • SQ-10087 - Paddings in side panel lists are not balanced
  • SQ-10088 - Hover color on disabled filter list makes no sense
  • SQ-10089 - Show upload progress (0-100%) while uploading templates
  • SQ-10092 - Remove tag by clicking in the small cross require two clicks
  • SQ-10093 - Enable keywords navigation for item detail endpoint
  • SQ-10094 - Item detail modal jumps when set to wide-modal
  • SQ-10096 - Weighted keywords only show weights in Annotation lanes
  • SQ-10097 - Side panel refreshes every time we open/close an item
  • SQ-10104 - Tags highlighting UX review
  • SQ-10135 - Explain smartfilter is currently being displayed on top of the list (regression)
  • SQ-10136 - Show progress when creating a project from a template
  • SQ-10143 - Bookmarks widget in dropdown mode has wrong padding
  • SQ-10150 - Fixing issue with Widget browser regression
  • SQ-10175 - Add Option to Card Widgets to hide Title & Metadata
  • SQ-10180 - Show active state of card in Annotation workbench
  • SQ-10184 - Scroll to Abstract / Keywords no longer works on explore screen
  • SQ-10198 - 'Copy to clipboard' action for token generation copied the old token
  • SQ-10202 - Layer with breadcrumb remained visible when query removed via searchbar
  • SQ-10212 - Rename-the-reset-prompt-to-not-include-the-delete-part

Deprecations

Changes of Python client

 Click here to see all deprecated and modified python client methods ...
Class / Method Signature 2.6.2 (old)Signature 2.6.3 (new)Note
ItemUploader(..., processing_config=None, ... , object_id=None, ... , source_secret=None, ..., )ItemUploader(...)
  • object_id is deprecated and no longer needed
  • source_secret is deprecated and will be ignored
  • processing_config is deprecated and will be ignored
DocumentUploader(..., processing_config=None, ... )DocumentUploader(...)
  • processing_config is deprecated and will be ignored
delete_item(project_id, item_id, object_ids=None)delete_item(project_id, item_id)object_ids is no longer supported as input parameter
get_enrichment(project_id, enrichment_id, type=None)get_enrichment(project_id, enrichment_id)type is no longer supported as input parameter
update_enrichment(project_id, enrichment_id, type, name, config)update_enrichment(project_id, enrichment_id, name, config)type is no longer supported as input parameter
get_preview(project_id, provider, config)get_preview(project_id, config)provider is no longer supported as input parameter
get_entities_properties(project_id)get_entities_properties(project_id, query=None)New parameter query to filter entities
n.a.get_machinelearning_workflow_assets(project_id, ml_workflow_id,New method to return all binary assets like trained models associated with a ML workflow.
n.a.run_machinelearning_job(project_id, ml_workflow_id, ml_job_id)New method to schedule a machine learning job to run now
get_user_objects(...)n.a.deprecated
get_object(project_id, object_id)n.a.deprecated
new_object(...)n.a.deprecated
modify_object(...)n.a.deprecated
delete_object(...)n.a.deprecated
pause_object(...)n.a.deprecated
resume_object(...)n.a.deprecated
get_object_signals(...)n.a.deprecated
update_object_signals(...)n.a.deprecated
export_project(..., export_sources_dataloaders=True, ...)export_project(..., export_sources_dataloader_plugins=True, ...)parameter export_sources_dataloaders is now called export_sources_dataloader_plugins
n.a.import_project(...)New method to import parts or all configuration or metadata of a project
get_project_smartfilter_scores(project_id, tags, object_id=None, fields=None)get_project_smartfilter_scores(project_id, tags, fields=None)Parameter object_id is deprecated
get_object_subscriptions(...)get_sources(project_id, include_config=None, include_run_stats=None)deprecated, use get_sources instead
get_subscription(...)get_source(project_id, source_id, include_config=None, include_run_stats=None)deprecated, use get_source instead
new_subscription(...)new_source(project_id, name, config, scheduling_options, pipeline_workflow_id=None, source_id=None)deprecated, use new_source instead
modify_subscription(...)modify_source(project_id, source_id, name=None, config=None, scheduling_options=None, pipeline_workflow_id=None)deprecated, use modify_source instead
delete_subscription(...)delete_source(project_id, source_id)deprecated, use delete_source instead
pause_subscription(...)pause_source(project_id, source_id)deprecated, use pause_source instead
resume_subscription(...)resume_source(project_id, source_id)deprecated, use resume_source instead
n.a.run_source(project_id, source_id)New method to run a source now
n.a.reset_source(project_id, source_id, delete_source_data=None)New method to reset and run a source now

Deprecated Services

  • API Sourcer
  • API Bulk
  • Fileimport
  • Squirrel

Deprecated Providers

All providers have been removed and will be replaced by dataloader plugins instead.

ProviderNote
bingTo be replaced with the Newsriver Plugin
dataloaderReplaced with the new datasource service
diffbotRemoved, no replacement planned
facebookRemoved
feedTo be replaced by a dataloader plugin 
instagramRemoved
redditTo be replaced with a dataloader plugin
youtubeRemoved

Added on January 31, 2019 (build 212 - patch 4)

  • Fix broken run of pre 2.6.3 data sources.
  • Fix typeahead cursor position
  • Remove support of processing_config from ItemUploader.
  • Add proximity filter to lib NLP.
  • Fix result list in hidden layer.

Added on February 18, 2019 (build 232 - patch 5-11)

  • Fix load more in result list.
  • Fix metric widget cutoff.
  • Fix HTTP method for pipeline workflow moving.
  • Force validation via docstring Parameters in libNLP.
  • Fix result list in explore screen.
  • Ensure detail event on cards widget.
  • Fix name scoping of keys in KV cache/store in DL plugins.
  • Don't open edit mode when passing dash params.
  • Speed up weighted keyword aggregations.
  • Detach frozen widgets collections.
  • Allow disabling of job locking by cli dataloader.
  • Tune scheduler settings for datasource.
  • Add force_cache_clear for item updates.
  • Move ML job runs into a separate process (sqml).
  • Optional mode selection for libNLP steps. Allow renaming of entity property values.
  • Remove linkedinservice.
  • Improve group members loading performance by improving sql joins.
  • Fix ES migration for empty projects and Squirro's filter index.
  • More fair scheduling of sources by Ingester (Processes one batch of each source at a time).
  • Added a copy filter in libNLP for name spacing.
  • Migrate types to type in ML workflows. Catch validation errors before ML job runs.
  • Fix bug where entities are no longer showing in explore view.
  • Annotation workbench fixes.
  • Scheduler fixes.
  • Add tracking property to entities.
  • Fixes for parallel sections in ML workflows.

Added on February 21, 2019 (build 234 - patch 12)

  • Fix force_cache_clear for modify_items

Added on March 4, 2019 (build 236 - patch 13)

  • Add config options to rename axis labels
  • Item detail widget title aligned with star icon
  • Cards widget: Apply card style only to the header
  • Item Detail widget: Have option to configure pre-defined tags
  • Show item title in two lines if needed
  • Ability to sort facet values
  • More robust scheduling policies for ingester
  • Fix dashboard save
  • Improve dashboards print
  • Fix mobile dashboard bottom widget cut off
  • Fix item detail widget padding on mobile
  • Open in place for cards widget
  • Hide space changer until permissions load
  • Breadcrumbs widget doesnt display query from tab (bookmark) widget
  • Fixing typeahead overflow
  • Improved default time bucketing for line charts and time aggregations

Added on March 5, 2019 (build 237 - patch 14)

  • Add feed plugin by default to globally shipped plugins

Added on March 11, 2019 (build 239 - patch 15)

  • Fix broken sdk on Squirro servers.
  • Fix ingester bug where sources with empty pipelet config will not be processed.
  • Support select on non created-at facets in timeline.
  • Isolate materialize navs and subnavs.

Added on March 12, 2019 (build 240 - patch 16)

  • Fix ingester bug where sources with empty pipelet config will not be processed.

Added on March 12, 2019 (build 242 - patch 17)

  • Fix an issue with dashboards throwing occasional error

Added on March 13, 2019 (build 243 - patch 18)

  • Fix an issue with smartfilter upload

Added on March 15, 2019 (build 244 - patch 19)

  • Item detail widget store controlled fails to show item if not in collection
  • Do not export data checkpoints when exporting ML Models

Added on March 20, 2019 (build 245 - patch 20)

  • Fix an issue config file encryption in a cluster setup.

Added on March 25, 2019 (build 247 - patch 21)

  • Fix an issue with deduplicating items caused by the source refactoring.

Added on March 26, 2019 (build 248 - patch 22)

  • Fix another issue with deduplicating items caused by the source refactoring.

Added on March 31, 2019 (build 249 - patch 23)

  • Update link for downloading fasttext language detection model
  • Fix project import/export on IE
  • Improve ES query for many duplicates

Added on March 31, 2019 (build 249 - patch 23)

  • Typeahead fixes
    • Facet name suggestions in typeahead results are now alphabetically sorted.
    • Properly quote typeahead suggestions to generate a valid Squirro query if the typeahead suggestions contain special characters (like parantheses)
  • Alphabetical sorting of facet values on dashboards.


Installation & Upgrade

Fresh Installation Instructions

Please follow the regular installation steps

Upgrade Instructions

Please ensure that your current version is 2.6.2. If you are on a version older than 2.6.2, please contact support.

Due to the source and provider framework refactoring the update instructions are more involved than usual. Please make sure you follow each step in the update instructions carefully.
During the upgrade some of the data in elasticsearch will be modified. This leads to a temporary increase of disk space use. Please ensure that you have enough free disks space available before you commence.

The default location of the index files is: /var/lib/elasticsearch

In this example the overall index size is 28 GB:

# du -sch /var/lib/elasticsearch/
28G	/var/lib/elasticsearch/
28G	total

For the migration to succeed, we recommend you ensure you have at least another 28 GB of free disk space available.

 1. Upgrade Storage Nodes and Cluster Nodes collocated on the same machine/VM

Make sure you execute all of the following steps:

  1. stop all squirro services except the cluster service:

    RHEL6 / CentOS6
    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit stop ${service##*/}; done;
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Stopping $service"; systemctl stop $service; done

  2. wait for and check that all squirro services are stopped (except cluster service):

    RHEL6 / CentOS6
    monit summary
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | sed -e "s|^.*/||"); do echo "Status $service"; systemctl status $service; done
  3. yum clean all

  4. Update JRE

    yum update jre

  5. Update squirro storage node users

    yum update squirro-storage-node-users

  6. Update squirro storage node

    yum update squirro-storage-node

  7. Migrate Elasticsearch data

    yum install squirro-data-migration-storagenode

    This may take a long time, as this will update all items in the index to make the items compatible with Version 2.6.3. Wait until its done. You can check the progress and potential errors by tailing /var/log/squirro/squirro-data-migration.log

  8. Update squirro cluster node users

    yum update squirro-cluster-node-users

  9. Install the new datasource service

    yum install squirro-python-squirro.service.datasource

    Resolve any common.ini.rpmnew issues.

  10.  Start the datasource service:

    RHEL6 / CentOS6
    monit start sqdatasourced
    RHEL7 / CentOS7
    systemctl start sqdatasourced.service

    and wait until it is running

  11. Run the data migrations

    yum install squirro-data-migration-clusternode

    check /var/log/squirro/squirro-data-migration.log for errors

  12. Update all the other squirro services and packages

    yum update squirro*
  13. resolve any rpmnew files in /etc/squirro/, e.g. configuration.ini.rpmnew and topic.ini.rpmnew

  14. Remove legacy packages:

    RHEL6 / CentOS6
    yum remove squirro-python-squirro.provider.youtube squirro-python-squirro.provider.feed squirro-python-squirro.provider.bing squirro-python-squirro.provider.diffbot squirro-python-squirro.provider.facebook squirro-python-squirro.provider.reddit squirro-python-squirro.provider.common squirro-python-squirro.provider.dataloader squirro-python-squirro.api.sourcer squirro-python-squirro.service.squirrel squirro-python-squirro.lib.bing squirro-python-squirro.lib.embedly squirro-python-squirro.lib.facebook squirro-python-squirro.provider.instagram squirro-python-squirro.service.fileimport squirro-python-squirro.lib.bulk squirro-python-squirro.api.bulk; 
service monit restart
    RHEL7 / CentOS7
    yum autoremove

  15. restart all services except cluster service:

    RHEL6 / CentOS6
    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit restart ${service##*/}; done;
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Restart $service"; systemctl restart $service; done
 2. Upgrade Storage and Cluster Nodes when they are on different servers (and there is only one storage node and one cluster node)

Stop all squirro services except the cluster service. On the cluster node run:

  1. stop all squirro services except the cluster service:

    RHEL6 / CentOS6
    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit stop ${service##*/}; done;
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Stopping $service"; systemctl stop $service; done

  2. wait for and check that all squirro services are stopped on all cluster nodes:

    RHEL6 / CentOS6
    monit summary
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | sed -e "s|^.*/||"); do echo “Status $service"; systemctl status $service; done

Upgrade the storage node by running:

  1. yum clean all
  2. yum update jre
  3. yum update squirro-storage-node-users
  4. yum update squirro-storage-node
  5. yum install squirro-data-migration-storagenode

    This may take a while, wait until its done. You can check the progress and potential errors by tailing /var/log/squirro/squirro-data-migration.log

Upgrade the cluster node by running:

  1. yum update squirro-cluster-node-users

  2. Install the new datasource service

    yum install squirro-python-squirro.service.datasource

    Resolve any common.ini.rpmnew issues.

  3. And start it:

    RHEL6 / CentOS6
    monit start sqdatasourced
    RHEL7 / CentOS7
    systemctl start sqdatasourced.service

    and wait until it is running

  4. yum install squirro-data-migration-clusternode

    check /var/log/squirro/squirro-data-migration.log for errors

  5. yum update squirro*
  6. resolve any rpmnew files in /etc/squirro/, e.g. configuration.ini.rpmnew and topic.ini.rpmnew

  7. Remove legacy packages:

    RHEL6 / CentOS6
    yum remove squirro-python-squirro.provider.youtube squirro-python-squirro.provider.feed squirro-python-squirro.provider.bing squirro-python-squirro.provider.diffbot squirro-python-squirro.provider.facebook squirro-python-squirro.provider.reddit squirro-python-squirro.provider.common squirro-python-squirro.provider.dataloader squirro-python-squirro.api.sourcer squirro-python-squirro.service.squirrel squirro-python-squirro.lib.bing squirro-python-squirro.lib.embedly squirro-python-squirro.lib.facebook squirro-python-squirro.provider.instagram squirro-python-squirro.service.fileimport squirro-python-squirro.lib.bulk squirro-python-squirro.api.bulk; 
service monit restart
    RHEL7 / CentOS7
    yum autoremove

  8. restart all services except cluster service:

    RHEL6 / CentOS6
    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit restart ${service##*/}; done;
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo “Restart $service"; systemctl restart $service; done
 3. Upgrade multi-node clusters (multiple Storage Nodes and/or multiple Cluster Nodes)

Stop all squirro services except the cluster service. On each cluster node run:

  1. stop all squirro services except the cluster service:

    RHEL6 / CentOS6
    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit stop ${service##*/}; done;
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Stopping $service"; systemctl stop $service; done

  2. wait for and check that all squirro services are stopped on all cluster nodes:

    RHEL6 / CentOS6
    monit summary
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | sed -e "s|^.*/||"); do echo "Status $service"; systemctl status $service; done

Upgrade each storage nodes by running:

  1. yum clean all
  2. yum update jre
  3. yum update squirro-storage-node-users
  4. yum update squirro-storage-node

  5. Install the data migration package and run the data migration

    yum install squirro-data-migration-storagenode

    This may take a while, wait until its done. You can check the progress and potential errors by tailing /var/log/squirro/squirro-data-migration.log

Upgrade the cluster nodes by running each step on every cluster node. Make sure you execute a step on each node before moving to the next step.

  1. yum update squirro-cluster-node-users

  2. Install the new datasource service

    yum install squirro-python-squirro.service.datasource

    Resolve any common.ini.rpmnew issues.

  3. Adjust the /etc/squirro/datasource.ini file, set

    db_endpoint_discovery = true
endpoind_discovery = true

  4. And start it:

    RHEL6 / CentOS6
    monit start sqdatasourced
    RHEL7 / CentOS7
    systemctl start sqdatasourced.service

    and wait until it is running

  5. yum install squirro-data-migration-clusternode

    check /var/log/squirro/squirro-data-migration.log for errors

  6. yum update squirro*
  7. resolve any rpmnew files in /etc/squirro/, e.g. configuration.ini.rpmnew and topic.ini.rpmnew

  8. Remove legacy packages:

    RHEL6 / CentOS6
    yum remove squirro-python-squirro.provider.youtube squirro-python-squirro.provider.feed squirro-python-squirro.provider.bing squirro-python-squirro.provider.diffbot squirro-python-squirro.provider.facebook squirro-python-squirro.provider.reddit squirro-python-squirro.provider.common squirro-python-squirro.provider.dataloader squirro-python-squirro.api.sourcer squirro-python-squirro.service.squirrel squirro-python-squirro.lib.bing squirro-python-squirro.lib.embedly squirro-python-squirro.lib.facebook squirro-python-squirro.provider.instagram squirro-python-squirro.service.fileimport squirro-python-squirro.lib.bulk squirro-python-squirro.api.bulk; 
service monit restart
    RHEL7 / CentOS7
    yum autoremove

  9. restart all services except cluster service:

    RHEL6 / CentOS6
    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit restart ${service##*/}; done;
    RHEL7 / CentOS7
    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Restart $service"; systemctl restart $service; done