Page Comparison

The deduplication duplicate detection enrichment prevents duplication of items. Items that it has already seen are rejected and not indexed.

Table of Contents

Overview

Items sent to Squirro come from different sources. Those sources sometimes send their data multiple times or in the case of Twitter mention the same site. To avoid having duplicates in the index, Squirro tries to detect identical items (duplicates) and nearly identical items (near-duplicates). This page describes the duplicate detection, near-duplicate detection is a separate enrichment.

There are two parts when it comes to the duplicate handling, the first one is the detection and the second one is the action taken upon that.

Detection

Squirro first searches for an item with the same external_id (see Item Format). If no previous item with the same external_id exists, the deduplication enrichment then searches the index for a document with the same title and link.

Items from a bulk source (e.g. created by the ItemUploader or DocumentUploader) are considered private. To avoid data leakage, the deduplication will only look at items from this source for deduplication.

Policy

Once an item is considered to be a duplicate it can be associated, replaced or updated, depending on the policy. For details on the policies see the configuration belowThe deduplication step is used to detect duplicate items. Duplicates are discarded and not shown to the user in the interface.

Duplicate handling has two parts. First there is the actual detection of duplicates. Second is the action executed once a duplicate is detected. The executed actions are specified using a policy.

Image Added

Detection

The detection logic consults the index to find any items with the same properties (see Item Format) as the current item. The item properties for this lookup operation are configurable.

Unless a data source is considerate private (which is the case for sources created by the ItemUploader or DocumentUploader clients from the Python SDK), all sources of a project are included in the lookup. For private sources only the one source is used for the lookup. This behavior prevents data leakage between private and non-private sources.

Policy

If a duplicate item is detected a policy specifies which action is executed. The following table lists available policies and describes the executed action in detail.

Configuration

[
  ["external_id"],
  ["title", "link"]
]

Examples

To change the default policy, a processing_config needs to be passed to the DocumentUploader or ItemUploader:The following examples all use the Python SDK to show how the duplicate detection enrichment step can be used.

Item Uploader

The following example details how to update duplicate items.

from squirro_client import ItemUploader

# processing config to update duplicate items
processing_config = {
    'deduplication': {
        'enabled': True, 
        'policy': 'replace',update',
        'deduplication_fields': [
            ['external_id'],
            ['link'],
            ['title'],
        ],
    },
}

uploader = ItemUploader(…, processing_config=config)

In the example above the processing

...

If the API is used directly pass a processing_config in the config dictionary of the source.

Caveats

...

pipeline is instructed to update duplicate items based on the external_id, link and title properties individually.

New Data Source

The following example details how to update duplicate items for a new feed data source.

from squirro_client import SquirroClient
 
client = SquirroClient(None, None, cluster='https://next.squirro.net/')
client.authenticate(refresh_token='293d…a13b')
 
# processing config to update duplicate items
processing_config = {
    'deduplication': {
        'enabled': True,
        'policy': 'update',
        'deduplication_fields': [
            ['external_id'],
            ['link'],
            ['title'],
        ],
    },
}
 
# source configuration
config = {
    'url': 'http://newsfeed.zeit.de/index',
    'processing': processing_config
}
 
# create new source subscription
client.new_subscription(
    project_id='…', object_id='default', provider='feed', config=config)

In the example above the processing pipeline is instructed to update duplicate items based on the external_id, link and title properties individually.

Existing Data Source

The following example details how to update duplicate items for an existing source. Items which have already been processed are not updated.

from squirro_client import SquirroClient
 
client = SquirroClient(None, None, cluster='https://next.squirro.net/')
client.authenticate(refresh_token='293d…a13b')
 
# get existing source configuration (including processing configuration)
source = client.get_project_source(project_id='…', source_id='…')
config = source.get('config', {})
processing_config = config.get('processing_config', {})
 
# modify processing configuration
processing_config['deduplication'] = {
    'enabled': True,
    'policy': 'update',
    'deduplication_fields': [
        ['external_id'],
        ['link'],
        ['title'],
    ],
}
config['processing'] = processing_config
client.modify_project_source(project_id='…', source_id='…', config=config)

In the example above the processing pipeline is instructed to update duplicate items based on the external_id, link and title properties individually.