We're excited to announce Squirro 2.6.3, released on January 22nd 2019.


  • If you are using custom widgets based on the horizontal result widget in one of your projects, do not update to 2.6.3 yet, but please contact support. to clarify required steps.

Contents -What's in the release?


New Features

Data Sources

Entities

Machine Learning

Widgets & Dashboard

New Widgets

Widget Improvements and Changes

Dashboard & Layer

Improvements

New libNLP steps and features

Return of "Import Project" Functionality

Refactoring of Services

Major Code Cleanup

Continued integration of project template functionality

Other Improvements

Bug Fixes & Improvements

  • SB-30 - Delimiter field disappears and created facet does not show up.
  • SB-49 - No error message(s) when using an incorrect password during login on a mobile device.
  • SB-67 - Single layer autosize to fit frame: Widgets do not resize on browser window resize
  • SB-69 - Dashboard edit: Adding a new widget shows the properties of the last selected widget
  • SB-79 - Warning message appears on navigating away from a dashboard, even when no changes are made
  • SB-107 - Result list widgets highlighting SF even when they shouldn't
  • SB-116 - Timeline widget selection state doesn't go away after removing the filter.
  • SB-132 - Thumbnail position is wrong
  • SB-135 - Breadcrumb in hidden layer is always collapsed
  • SB-137 - The default "Home" chip in the breadcrumb widget is not configurable.
  • SB-140 - Authentication isn't renewed after the 10 minute default validity interval with SAML
  • SB-146 - Typeahead facet display issue
  • SB-150 - 1000's Separator issue in Dashboard
  • SB-164 - Unable to restart walkthrough using the side bar right after finishing it
  • SB-165 - Slow loading of data on Table Widget onClick
  • SB-166 - Weird scrollbars on Table Widget when columns are small
  • SB-168 - Improvements in Recommendation widget explanation and fix of displaying issues.
  • SB-170 - Breadcrumb (Chips) not matching the query terms
  • SB-179 - "Matching pages" column shown without content if highlighting is disabled
  • SB-193 - Facet widget selections not cleared on Reset "Clear all"
  • SB-197 - Create project sidebar closes itself when I resize browser
  • SB-204 - Entity list window does not continue to bottom of item
  • SB-205 - Data labeling lane widget is overflowing
  • SB-206Replace overflow-y:scroll with overflow-y:auto so the scrollbars are not visible all the time
  • SB-220 - Deleting a layer says 'section' instead of layer in confirmation modal
  • SB-221 - Link widget: Custom frame input field should not be visible when the widget is controlled by the dashboard store
  • SQ-8280 - Result list widgets: Have config options to enable / disable metadata info and abstract.
  • SQ-9126 - 'Clear All' button should appear only when some filter is applied
  • SQ-9244 - Add more data loader mapping options
  • SQ-9313 - Trend widget config: When no trend is setup, show helper text inside the dropdown
  • SQ-9325 - Map widget config: When no geo-point facets are created, show helper text inside the dropdown
  • SQ-9329 - Barchart widget: Rename 'Facet-X Axis' label to 'Facet'
  • SQ-9566 - Item detail: Icons list jumps on click on tag, body icons
  • SQ-9668 - IE11: In widget browser, overflowing widget names should be displayed in two line
  • SQ-9728 - Have option to duplicate widget across layers for both desktop and mobile version of a dashboard
  • SQ-9735 - Wide modal view breaks on resize
  • SQ-9877 - Implement Walkthrough for CSV Data Loader
  • SQ-9966 - Have option to hide the keywords from the item detail
  • SQ-9988 - Add Entities Filter to Sidebar and Dashboard Sidebar
  • SQ-10007 - Append user_id to email of deleted user (so that a new user can be made with the same email)
  • SQ-10004 - Data labeling lane is showing empty pill when there are no tags
  • SQ-10020 - Implement Walkthrough for Excel Data Loader
  • SQ-10036 - Filter list on dashboard sidebar has wrong paddings
  • SQ-10038 - Filters keep reloading on expanding/collapse
  • SQ-10041 - Allow to define the maximum file size for uploads in configuration.ini
  • SQ-10043 - Dashboard widgets' containers have wrong size until they are fully loaded
  • SQ-10048 - Widgets in disable drilldown mode keep appending filters to history
  • SQ-10062 - Create Adhoc Inference Widget
  • SQ-10070 - Show number of items loaded per source
  • SQ-10072 - Add icon to item detail toolbar to generate direct link to items
  • SQ-10073 - Add link to explore items loaded by a given source
  • SQ-10077 - Project Template, manual refreshing of the browser is not acceptable
  • SQ-10082 - Keywords translation is gone
  • SQ-10083 - Tags should highlight before being removed (double backspace)
  • SQ-10084 - Heatmap values are overflowing its cell space
  • SQ-10085 - Item detail arrows move while the item loads
  • SQ-10086 - Item detail content is using full width, should keep the 700px limit
  • SQ-10087 - Paddings in side panel lists are not balanced
  • SQ-10088 - Hover color on disabled filter list makes no sense
  • SQ-10089 - Show upload progress (0-100%) while uploading templates
  • SQ-10092 - Remove tag by clicking in the small cross require two clicks
  • SQ-10093 - Enable keywords navigation for item detail endpoint
  • SQ-10094 - Item detail modal jumps when set to wide-modal
  • SQ-10096 - Weighted keywords only show weights in Annotation lanes
  • SQ-10097 - Side panel refreshes every time we open/close an item
  • SQ-10104 - Tags highlighting UX review
  • SQ-10135 - Explain smartfilter is currently being displayed on top of the list (regression)
  • SQ-10136 - Show progress when creating a project from a template
  • SQ-10143 - Bookmarks widget in dropdown mode has wrong padding
  • SQ-10150 - Fixing issue with Widget browser regression
  • SQ-10175 - Add Option to Card Widgets to hide Title & Metadata
  • SQ-10180 - Show active state of card in Annotation workbench
  • SQ-10184 - Scroll to Abstract / Keywords no longer works on explore screen
  • SQ-10198 - 'Copy to clipboard' action for token generation copied the old token
  • SQ-10202 - Layer with breadcrumb remained visible when query removed via searchbar
  • SQ-10212 - Rename-the-reset-prompt-to-not-include-the-delete-part

Deprecations

Changes of Python client


Class / Method Signature 2.6.2 (old)Signature 2.6.3 (new)Note
ItemUploader(..., processing_config=None, ... , object_id=None, ... , source_secret=None, ..., )ItemUploader(...)
  • object_id is deprecated and no longer needed
  • source_secret is deprecated and will be ignored
  • processing_config is deprecated and will be ignored
DocumentUploader(..., processing_config=None, ... )DocumentUploader(...)
  • processing_config is deprecated and will be ignored
delete_item(project_id, item_id, object_ids=None)delete_item(project_id, item_id)object_ids is no longer supported as input parameter
get_enrichment(project_id, enrichment_id, type=None)get_enrichment(project_id, enrichment_id)type is no longer supported as input parameter
update_enrichment(project_id, enrichment_id, type, name, config)update_enrichment(project_id, enrichment_id, name, config)type is no longer supported as input parameter
get_preview(project_id, provider, config)get_preview(project_id, config)provider is no longer supported as input parameter
get_entities_properties(project_id)get_entities_properties(project_id, query=None)New parameter query to filter entities
n.a.get_machinelearning_workflow_assets(project_id, ml_workflow_id,New method to return all binary assets like trained models associated with a ML workflow.
n.a.run_machinelearning_job(project_id, ml_workflow_id, ml_job_id)New method to schedule a machine learning job to run now
get_user_objects(...)n.a.deprecated
get_object(project_id, object_id)n.a.deprecated
new_object(...)n.a.deprecated
modify_object(...)n.a.deprecated
delete_object(...)n.a.deprecated
pause_object(...)n.a.deprecated
resume_object(...)n.a.deprecated
get_object_signals(...)n.a.deprecated
update_object_signals(...)n.a.deprecated
export_project(..., export_sources_dataloaders=True, ...)export_project(..., export_sources_dataloader_plugins=True, ...)parameter export_sources_dataloaders is now called export_sources_dataloader_plugins
n.a.import_project(...)New method to import parts or all configuration or metadata of a project
get_project_smartfilter_scores(project_idtagsobject_id=Nonefields=None)get_project_smartfilter_scores(project_id, tagsfields=None)Parameter object_id is deprecated
get_object_subscriptions(...)get_sources(project_idinclude_config=Noneinclude_run_stats=None)deprecated, use get_sources instead
get_subscription(...)get_source(project_id, source_idinclude_config=Noneinclude_run_stats=None)deprecated, use get_source instead
new_subscription(...)new_source(project_id, name, config, scheduling_optionspipeline_workflow_id=None, source_id=None)deprecated, use new_source instead
modify_subscription(...)modify_source(project_id, source_id, name=None, config=Nonescheduling_options=None, pipeline_workflow_id=None)deprecated, use modify_source instead
delete_subscription(...)delete_source(project_id, source_id)deprecated, use delete_source instead
pause_subscription(...)pause_source(project_id, source_id)deprecated, use pause_source instead
resume_subscription(...)resume_source(project_id, source_id)deprecated, use resume_source instead
n.a.run_source(project_id, source_id)New method to run a source now
n.a.reset_source(project_id, source_id, delete_source_data=None)New method to reset and run a source now


Deprecated Services

Deprecated Providers

All providers have been removed and will be replaced by dataloader plugins instead.

ProviderNote
bingTo be replaced with the Newsriver Plugin
dataloaderReplaced with the new datasource service
diffbotRemoved, no replacement planned
facebookRemoved
feedTo be replaced by a dataloader plugin 
instagramRemoved
redditTo be replaced with a dataloader plugin
youtubeRemoved

Added on January 31, 2019 (build 212 - patch 4)

Added on February 18, 2019 (build 232 - patch 5-11)

Added on February 21, 2019 (build 234 - patch 12)

Added on March 4, 2019 (build 236 - patch 13)

Added on March 5, 2019 (build 237 - patch 14)

Added on March 11, 2019 (build 239 - patch 15)

Added on March 12, 2019 (build 240 - patch 16)

Added on March 12, 2019 (build 242 - patch 17)

Added on March 13, 2019 (build 243 - patch 18)

Added on March 15, 2019 (build 244 - patch 19)

Added on March 20, 2019 (build 245 - patch 20)

Added on March 25, 2019 (build 247 - patch 21)

Added on March 26, 2019 (build 248 - patch 22)

Added on March 31, 2019 (build 249 - patch 23)

Added on March 31, 2019 (build 249 - patch 23)



Installation & Upgrade

Fresh Installation Instructions

Please follow the regular installation steps

Upgrade Instructions


Please ensure that your current version is 2.6.2. If you are on a version older than 2.6.2, please contact support.


Due to the source and provider framework refactoring the update instructions are more involved than usual. Please make sure you follow each step in the update instructions carefully.
During the upgrade some of the data in elasticsearch will be modified. This leads to a temporary increase of disk space use. Please ensure that you have enough free disks space available before you commence.

The default location of the index files is: /var/lib/elasticsearch

In this example the overall index size is 28 GB:

# du -sch /var/lib/elasticsearch/
28G	/var/lib/elasticsearch/
28G	total

For the migration to succeed, we recommend you ensure you have at least another 28 GB of free disk space available.


Make sure you execute all of the following steps:

  1. stop all squirro services except the cluster service:

    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit stop ${service##*/}; done;


    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Stopping $service"; systemctl stop $service; done


  2. wait for and check that all squirro services are stopped (except cluster service):

    monit summary


    for service in $(ls /lib/systemd/system/sq*d.service | sed -e "s|^.*/||"); do echo "Status $service"; systemctl status $service; done



  3. yum clean all


  4. Update JRE

    yum update jre


  5. Update squirro storage node users

    yum update squirro-storage-node-users


  6. Update squirro storage node

    yum update squirro-storage-node


  7. Migrate Elasticsearch data

    yum install squirro-data-migration-storagenode

    This may take a long time, as this will update all items in the index to make the items compatible with Version 2.6.3. Wait until its done. You can check the progress and potential errors by tailing /var/log/squirro/squirro-data-migration.log

  8. Update squirro cluster node users

    yum update squirro-cluster-node-users


  9. Install the new datasource service

    yum install squirro-python-squirro.service.datasource

    Resolve any common.ini.rpmnew issues.

  10.  Start the datasource service:

    monit start sqdatasourced


    systemctl start sqdatasourced.service

    and wait until it is running

  11. Run the data migrations

    yum install squirro-data-migration-clusternode

    check /var/log/squirro/squirro-data-migration.log for errors

  12. Update all the other squirro services and packages

    yum update squirro*


  13. resolve any rpmnew files in /etc/squirro/, e.g. configuration.ini.rpmnew and topic.ini.rpmnew
  14. Remove legacy packages:

    yum remove squirro-python-squirro.provider.youtube squirro-python-squirro.provider.feed squirro-python-squirro.provider.bing squirro-python-squirro.provider.diffbot squirro-python-squirro.provider.facebook squirro-python-squirro.provider.reddit squirro-python-squirro.provider.common squirro-python-squirro.provider.dataloader squirro-python-squirro.api.sourcer squirro-python-squirro.service.squirrel squirro-python-squirro.lib.bing squirro-python-squirro.lib.embedly squirro-python-squirro.lib.facebook squirro-python-squirro.provider.instagram squirro-python-squirro.service.fileimport squirro-python-squirro.lib.bulk squirro-python-squirro.api.bulk; 
    service monit restart


    yum autoremove


  15. restart all services except cluster service:

    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit restart ${service##*/}; done;


    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Restart $service"; systemctl restart $service; done



Stop all squirro services except the cluster service. On the cluster node run:

  1. stop all squirro services except the cluster service:

    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit stop ${service##*/}; done;


    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Stopping $service"; systemctl stop $service; done


  2. wait for and check that all squirro services are stopped on all cluster nodes:

    monit summary


    for service in $(ls /lib/systemd/system/sq*d.service | sed -e "s|^.*/||"); do echo “Status $service"; systemctl status $service; done


Upgrade the storage node by running:


  1. yum clean all



  2. yum update jre



  3. yum update squirro-storage-node-users



  4. yum update squirro-storage-node



  5. yum install squirro-data-migration-storagenode

    This may take a while, wait until its done. You can check the progress and potential errors by tailing /var/log/squirro/squirro-data-migration.log


Upgrade the cluster node by running:


  1. yum update squirro-cluster-node-users


  2. Install the new datasource service

    yum install squirro-python-squirro.service.datasource

    Resolve any common.ini.rpmnew issues.

  3. And start it:

    monit start sqdatasourced


    systemctl start sqdatasourced.service

    and wait until it is running


  4. yum install squirro-data-migration-clusternode

    check /var/log/squirro/squirro-data-migration.log for errors


  5. yum update squirro*


  6. resolve any rpmnew files in /etc/squirro/, e.g. configuration.ini.rpmnew and topic.ini.rpmnew
  7. Remove legacy packages:

    yum remove squirro-python-squirro.provider.youtube squirro-python-squirro.provider.feed squirro-python-squirro.provider.bing squirro-python-squirro.provider.diffbot squirro-python-squirro.provider.facebook squirro-python-squirro.provider.reddit squirro-python-squirro.provider.common squirro-python-squirro.provider.dataloader squirro-python-squirro.api.sourcer squirro-python-squirro.service.squirrel squirro-python-squirro.lib.bing squirro-python-squirro.lib.embedly squirro-python-squirro.lib.facebook squirro-python-squirro.provider.instagram squirro-python-squirro.service.fileimport squirro-python-squirro.lib.bulk squirro-python-squirro.api.bulk; 
    service monit restart


    yum autoremove


  8. restart all services except cluster service:

    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit restart ${service##*/}; done;


    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo “Restart $service"; systemctl restart $service; done




Stop all squirro services except the cluster service. On each cluster node run:

  1. stop all squirro services except the cluster service:

    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit stop ${service##*/}; done;


    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Stopping $service"; systemctl stop $service; done


  2. wait for and check that all squirro services are stopped on all cluster nodes:

    monit summary


    for service in $(ls /lib/systemd/system/sq*d.service | sed -e "s|^.*/||"); do echo "Status $service"; systemctl status $service; done


Upgrade each storage nodes by running:


  1. yum clean all



  2. yum update jre



  3. yum update squirro-storage-node-users



  4. yum update squirro-storage-node


  5. Install the data migration package and run the data migration

    yum install squirro-data-migration-storagenode

    This may take a while, wait until its done. You can check the progress and potential errors by tailing /var/log/squirro/squirro-data-migration.log

Upgrade the cluster nodes by running each step on every cluster node. Make sure you execute a step on each node before moving to the next step.


  1. yum update squirro-cluster-node-users


  2. Install the new datasource service

    yum install squirro-python-squirro.service.datasource

    Resolve any common.ini.rpmnew issues.

  3. Adjust the /etc/squirro/datasource.ini file, set

    db_endpoint_discovery = true
    endpoind_discovery = true


  4. And start it:

    monit start sqdatasourced


    systemctl start sqdatasourced.service

    and wait until it is running


  5. yum install squirro-data-migration-clusternode

    check /var/log/squirro/squirro-data-migration.log for errors


  6. yum update squirro*


  7. resolve any rpmnew files in /etc/squirro/, e.g. configuration.ini.rpmnew and topic.ini.rpmnew
  8. Remove legacy packages:

    yum remove squirro-python-squirro.provider.youtube squirro-python-squirro.provider.feed squirro-python-squirro.provider.bing squirro-python-squirro.provider.diffbot squirro-python-squirro.provider.facebook squirro-python-squirro.provider.reddit squirro-python-squirro.provider.common squirro-python-squirro.provider.dataloader squirro-python-squirro.api.sourcer squirro-python-squirro.service.squirrel squirro-python-squirro.lib.bing squirro-python-squirro.lib.embedly squirro-python-squirro.lib.facebook squirro-python-squirro.provider.instagram squirro-python-squirro.service.fileimport squirro-python-squirro.lib.bulk squirro-python-squirro.api.bulk; 
    service monit restart


    yum autoremove


  9. restart all services except cluster service:

    for service in $(ls /etc/init.d/sq* | grep -v sqclusterd); do monit restart ${service##*/}; done;


    for service in $(ls /lib/systemd/system/sq*d.service | grep -v sqclusterd | sed -e "s|^.*/||"); do echo "Restart $service"; systemctl restart $service; done