Page Comparison

There are two places this configuration can be specified:

• Per source / subscription: when creating a new subscription, the processing instructions can be passed in to fine-tune the behavior for that one source.
• Per project: a project also has a processing config, which applies to all items coming in for a project.
• Within Data Loader using the argument --source-config-file.

Source processing config

To set up a processing configuration, specify the processing field in a source's config. The value of that field is again a dictionary, with the enrichment names as keys.

Enrichments which can be specified include:

For example to set up a Twitter source with duplicate detection disabled, the following configuration would be used:

...

client = SquirroClient(None, None, cluster='https://nextdemo.squirro.net/')
client.authenticate(refresh_token='293d…a13b')
client.new_subscription(project_id, object_id='default', provider='twitter',
    processing_config={
        'query': 'Squirro',
        'processing': {
            'deduplication': {
                'enabled': False
            }
        }
    })

The enabled property is available for every built-in enrichment and can be set to true or false. Some of the enrichments have additional configuration options, that are described on the corresponding page.

To mark a source for processing by the new Squirro Pipeline 2.0 (available starting with Squirro Version 2.5.1):

client = SquirroClient(None, None, cluster='https://demo-25.squirro.net/')
client.authenticate(refresh_token='293d…a13b')
client.new_subscription(project_id, object_id='default', provider='bulk',
    config={
        'pipeline': 'ingester',
        'name': 'large_subscription_1',
        'ext_id': 'large_subscription_1_id'
    })

Project processing config

Please contact the Squirro team if you want to use project processing configs.