Dataloader Provider

The dataloader provider uses the developer uploaded dataloader plugins to fetch data from custom data sources and upload to a Squirro server.

Provider name	dataloader
Type	Polling provider

Configuration

In order to use your custom made dataloader plugins, you will first need to upload the dataloader plugin using the `squirro_asset` command line utility as described here.

Once the dataloader plugin has been uploaded successfully to the server, it is available to use from the Python SDK

Field	Description
dataloader_options	Dict of the options required by squirro_data_load.
dataloader_plugin_options	Dict of all the options required by the plugin being used. Can be retrieved by doing a list call on the `squirro_asset` utility for the corresponding plugin.

Configuration Example

Once the dataloader plugin has been uploaded, you can use the python SquirroClient to create a new subscription out of the dataloader plugin you just uploaded. One example of that is show below

Make sure to use the `plugin_name` key in the `dataloader_options` dict to specify which dataloader plugin should be used.


dataloader_plugin_options = {
    'hipchat_room_blacklist': 'squirro',
    'hipchat_conversation_length': 3,
    'hipchat_authtoken': '1N9lKixOJnEdPJ4MaTDFEcgRvaSd02xdZka3ENgG',
    'hipchat_conversation_age': 2,
    'hipchat_mode': 'message',
    'hipchat_url': 'https://www.hipchat.com'
}


dataloader_options = {
    'plugin_name': 'hipchat_thumbnail',
    'reset': True,
    'map_id': 'id',
    'map_created_at': 'date',
    'map_title': 'title',
    'source_name': 'HipChat',
    'map_body': 'body',
    'facets': {
        'emoticons': {
            'visible': True,
            'delimiter': '!',
            'name': 'Emoticons',
            'searchable': True,
            'group_name': 'HipChat'
        },
        'room': {
            'visible': True,
            'name': 'Room',
            'searchable': True,
            'group_name': 'HipChat'
        },
        'System': {
            'visible': True,
            'display_name': 'System'
        },
        'participants': {
            'visible': True,
            'delimiter': ';',
            'name': 'Participants',
            'searchable': True,
            'group_name': 'HipChat'
        },
        'names': {
            'visible': True,
            'delimiter': ';',
            'display_name': 'Names'
        },
        'Type': {
            'visible': True,
            'display_name': 'Type'
        },
        'grouping': {
            'visible': True,
            'name': 'Grouping',
            'group_name': 'HipChat'
        }
    }
}



client = SquirroClient(None, None, cluster='https://demo.squirro.net/')
client.authenticate(refresh_token='293d…a13b')
client.new_subscription(project_id, object_id='default', provider='dataloader',
	config={'dataloader_options': dataloader_options, 'dataloader_plugin_options': dataloader_plugin_options})

Documentation

Dataloader Provider

Table of Contents

Configuration

Configuration Example