Trend Detection Tutorial

This tutorial goes step by step through setting up a Trend Detection on a Squirro project and visualizing it on a Squirro dashboard.

Table of Contents

What is Trend Detection?

The Trend Detection analysis can be used to detect unusual trends in the time series data. In the Squirro context, time series data is generated in the form of the number-of-items per time-unit in a particular project for a particular query. This time series data can be easily observed today with the histogram bins on the search page (see image below). Trend detection analysis aims to find unusually high peaks in this histogram/time-series automatically.

As an example scenario, consider a project `News` with a feed of all the news-items from a few of the major news publications. Now, a query like `Facebook AND Whatsapp` will filter the list of all the documents to a sub-list of documents/news-items containing both the words Facebook and Whatsapp. For the project `News` with query `Facebook AND Whatsapp`, we define a time-series as the number of items matching `Facebook AND Whatsapp` per time-unit, where time-unit can be hourly, weekly, daily, monthly or yearly.

The detection of unusual trends is done by learning from the historical/old data to auto-compute a reasonable threshold. So, in order for it to work properly it is important that we have enough historical data to learn from.

Setting up Trend Detection

A new trend-detection can now be set up using the Squirro UI. Please follow the screenshots below that will guide through the process of setting up new trend-detection one by one. Once set up, the detected anomalies can then be visualised on the dashboard using a Trend widget (covered later).

Moreover, a new trend-detection can be set up in two different modes i.e. on the item-count over time for items coming into a project & the changes in the values of a numerical facet over time.

Trend Detection on item counts

• On the "Search" tab of Squirro UI, select the "Create Trend" option from the "Save" button drop down.

• This will present the user with a "Create Trend" modal, which needs to be filled with the necessary information described below.
  
  
  where, 
  • Title: is the name of the name of the trend-detection you are setting up.
  • Query: is an optional Squirro Query which will be used to filter down the item-counts.
  • Data Aggregation Interval: is the length (in time units) of one time bucket.
  • Create Alert: is the email-address for sending alert emails whenever something unusual is detected
• Click on the "Save" button to complete setting up the Trend-Detection.

Trend Detection on numerical facets

• One can also use the "Create Trend" modal to create a new trend detection on a numerical facet.
• In the presence of a numerical facet, the "Create Trend" dialog will have an extra checkbox to set up the trend detection on a numerical facet rather than the item counts
  
• Using this "Numerical Aggregation" checkbox, one can select a particular numerical facet of interest.
• Once a numerical facet has been selected, one can choose the aggregation method to be used on the numerical facet before setting up the trend detection. Possible options for aggregation are: Sum, Average, and Minimum.
• The rest of the workflow for setting up the Trend Detection is similar to the workflow for setting up the Trend Detection on the item-counts.

Trend visualization on the dashboard

• Once set up, the detected anomalies can be visualized on the dashboard using a new "Trends widget" on the dashboard.
• One can select the "Trends" widget under the dashboard edit mode by adding a new widget of type "Trend" as show in the screenshot below.
  
  
• Once selected, one can choose between all the trend detections set up on the project as shown in the screenshot below, where one can select between two different trend-detections.
  
• Once selected, one can see the visualizations on the Trend widget as shown below