Dataloader Provider
The dataloader provider uses the developer uploaded dataloader plugins to fetch data from custom data sources and upload to a Squirro server.
Provider name | dataloader |
---|---|
Type | Polling provider |
Table of Contents
Configuration
In order to use your custom made dataloader plugins, you will first need to upload the dataloader plugin using the `squirro_asset` command line utility as described here.
Once the dataloader plugin has been uploaded successfully to the server, it is available to use from the Python SDK
Field | Description |
---|---|
dataloader_options | Dict of the options required by squirro_data_load. |
dataloader_plugin_options | Dict of all the options required by the plugin being used. Can be retrieved by doing a list call on the `squirro_asset` utility for the corresponding plugin. |
Configuration Example
Once the dataloader plugin has been uploaded, you can use the python SquirroClient to create a new subscription out of the dataloader plugin you just uploaded. One example of that is show below
Make sure to use the `plugin_name` key in the `dataloader_options` dict to specify which dataloader plugin should be used.
dataloader_plugin_options = { 'hipchat_room_blacklist': 'squirro', 'hipchat_conversation_length': 3, 'hipchat_authtoken': '1N9lKixOJnEdPJ4MaTDFEcgRvaSd02xdZka3ENgG', 'hipchat_conversation_age': 2, 'hipchat_mode': 'message', 'hipchat_url': 'https://www.hipchat.com' } dataloader_options = { 'plugin_name': 'hipchat_thumbnail', 'reset': True, 'map_id': 'id', 'map_created_at': 'date', 'map_title': 'title', 'source_name': 'HipChat', 'map_body': 'body', 'facets': { 'emoticons': { 'visible': True, 'delimiter': '!', 'name': 'Emoticons', 'searchable': True, 'group_name': 'HipChat' }, 'room': { 'visible': True, 'name': 'Room', 'searchable': True, 'group_name': 'HipChat' }, 'System': { 'visible': True, 'display_name': 'System' }, 'participants': { 'visible': True, 'delimiter': ';', 'name': 'Participants', 'searchable': True, 'group_name': 'HipChat' }, 'names': { 'visible': True, 'delimiter': ';', 'display_name': 'Names' }, 'Type': { 'visible': True, 'display_name': 'Type' }, 'grouping': { 'visible': True, 'name': 'Grouping', 'group_name': 'HipChat' } } } client = SquirroClient(None, None, cluster='https://demo.squirro.net/') client.authenticate(refresh_token='293d…a13b') client.new_subscription(project_id, object_id='default', provider='dataloader', config={'dataloader_options': dataloader_options, 'dataloader_plugin_options': dataloader_plugin_options})