Trend Detection Tutorial

This tutorial goes step by step through setting up a Trend Detection on a Squirro project and visualizing it on a Squirro dashboard.

Table of Contents

What is Trend Detection?

The Trend Detection analysis can be used to detect unusual trends in the time series data. In the Squirro context, time series data is generated in the form of the number-of-items per time-unit in a particular project for a particular query. This time series data can be easily observed today with the histogram bins on the search page (see image below). Moreover, the numerical facets in Squirro can also be visualized on the dashboard as a time series using the "Line chart" widget. First example of time series data in Squirro context i.e. the item-counts is shown in the screenshot below. Trend detection analysis aims to find unusually high peaks in this histogram/time-series automatically.

As an example scenario, consider a project `News` with a feed of all the news-items from a few of the major news publications. Now, a query like `Facebook AND Whatsapp` will filter the list of all the documents to a sub-list of documents/news-items containing both the words Facebook and Whatsapp. For the project `News` with query `Facebook AND Whatsapp`, we define a time-series as the number of items matching `Facebook AND Whatsapp` per time-unit, where time-unit can be hourly, weekly, daily, monthly or yearly.

The detection of unusual trends is done by learning from the historical/old data to auto-compute a reasonable threshold. So, in order for it to work properly it is important that we have enough historical data to learn from.

Dataset Used

We are going to cover two different scenarios of setting up trend-detection and we are going to use two different datasets to demonstrate it.

In the first scenario, we will set up trend detection on the number of items arriving in the project over time. For this scenario, we are going to use the bug-tracking dataset from the bugtrackers. We have removed the summary of the bugs to reduce the size of the csv file for import. Please download this dataset from here if you want to follow along with the tutorial.

In the second scenario, we will set up trend detection on change in the values of numerical facet of a Squirro item over time. For this scenario, we are going to use an anonymized ITSM dataset. Please download the csv dataset from here if you want to follow along with the tutorial. Every row in the dataset contains three fields i.e. "date", "title" and "calls". Please note that this dataset does not contain any textual data because trend-analysis is done purely on numerical data. 

Scenario 1 - Trend Detection analysis on item counts

This section guides one through the process of setting up Trend Detection on the item counts in a Squirro project over time. Furthermore, these item counts can also be filtered with a Squirro Query as filtering step before setting up the Trend Detection analysis. Both of these use-cases are covered in the subsequent subsections below.

Importing data into Squirro Project

• Download the csv data for this section of the tutorial from here.
• Create a new Squirro project and upload the CSV data using the "CSV/Excel one-off import" option under the data tab
  
• While importing data, change the mapping of datetime row header to "Creation Date" while leaving all the other mappings to their default option.
  

Trend Detection without query

• On the "Search" tab of Squirro UI, select the "Create Trend" option from the "Save" button drop down.

• This will present the user with a "Create Trend" modal, which needs to be filled with the necessary information described below.
  
  
  where, 
  • Title: is the name of the name of the trend-detection you are setting up.
  • Query: is an optional Squirro Query which will be used to filter down the item-counts.
  • Data Aggregation Interval: is the length (in time units) of one time bucket.
  • Create Alert: is the email-address for sending alert emails whenever something unusual is detected
• Click on the "Save" button to complete setting up the Trend-Detection.
  
  

Visualization using Trend widget

One can visualize the Trend Detection analysis using the "Trends" widget on the dashboard. Please follow the steps below to create a new Trend Detection widget.

• While adding a new widget on the dashboard, you will see a new "Type" of widget namely "Trend".

• Once you select the "Trend" type of widget, one can see all the Trend Detections which have been set up on this particular project under the "Trend Detections" heading. On selecting a particular Trend Detection, the widget shows the result of trend detection analysis i.e. the time series data (in blue), the automatically computed thresholds (in grey) and finally the detected anomalies (in lettered label flags).

Trend Detection with query

While setting up the Trend Detection, we can choose to filter the project items with a Squirro query before setting up the Trend Detection. This is also achieved through the "Create Trend" modal.

• In order to achieve this through the "Create Trend" modal, type a Squirro query while setting up the Trend Detection under the "Query" heading. 

• Rest of the steps remain the same as above.

Visualization using Trend widget

Once, set up this trend detection can also be visualized by adding a new widget of type "Trend" and selecting the name of the newly created trend detection from the drop-down. The newly added trend widget will look something like this.

Scenario 2 - Trend Detection analysis on numerical facets

Importing data into Squirro Project

Since, we are going to set up the Trend Detection on a numerical facet, we will first have to configure a facet into the Squirro project before loading the data. Please follow the steps below to load data into a Squirro project with a numerical facet.

• Download the csv data for this section of the tutorial from here.

• Create a new Squirro project and add a new facet of type int with name "service_tickets" using the Squirro UI.
  

  Please make sure that the new facet is of type "int". Without proper configuration of the facet-type, you will not be able to set up the Trend Detection on this facet.

• Now import the CSV data into the Squirro project using the Squirro UI as described in the Scenario 1 set up. Make sure that the "datetime" field from the csv is mapped to "Creation Date" while leaving all the other mappings as default while using the CSV importer.

Trend Detection on numerical facet

• One can also use the "Create Trend" modal to create a new trend detection on a numerical facet.
• In the presence of a numerical facet, the "Create Trend" dialog will have an extra checkbox to set up the trend detection on a numerical facet rather than the item counts.
• In the current example, we set up the Trend Detection on a numerical facet named "service_tickets" which we have created while importing the data
  
• Using this "Numerical Aggregation" checkbox, one can select a particular numerical facet of interest. In this example, we select the facet "service_tickets"
• Once a numerical facet has been selected, one can choose the aggregation method to be used on the numerical facet before setting up the trend detection. Possible options for aggregation are: Sum, Average, and Minimum. We choose "Average" for this particular example.
• Moreover, in this example, we want to analyze data in a daily bucket rather than the default weekly bucket. So, we set up the "Data Aggregation Interval" to 1 day.
• The rest of the workflow for setting up the Trend Detection is similar to the workflow for setting up the Trend Detection on the item-counts.

Visualizations using Trend widget

• Once set up, the detected anomalies can be visualized on the dashboard using a new "Trends widget" on the dashboard.
• One can select the "Trends" widget under the dashboard edit mode by adding a new widget of type "Trend".
• After selecting the Trend widget, one can choose between all the trend detections set up on the project.
• Once selected, one should see the visualizations on the Trend widget as shown in the screenshot below.
  

Visualizations of future predictions using Trend Widget

For time series with recognizable repeating patterns, the future values of the time series can also be predicted and visualized. In order to turn on the predictions visualization, go to the edit mode of the "Trends" widget and check the "Enable Predictions" checkbox.

After checking the "Enable Predictions" check-box, the Trends widget will be updated with the future predictions of the time series data based on the repeating patterns recognized by Squirro in the historical data. Moreover, the number of future predictions being displayed can be adjusted using the "Predictions Range" slider.

Conclusion

The two examples in this tutorial led you through two main scenarios for using Trend-Detection

1. Setting up Trend Detection on the item counts over time in a Squirro project.
2. Setting up Trend Detection on the values of numerical facets over time in a Squirro project.

Consult the Trend Detection reference for reference or contact support if you have any questions about how to do something specific with the Trend Detection.