| Excerpt |
|---|
The dataloader provider uses the developer uploaded dataloader plugins to fetch data from custom data sources and upload to a Squirro server. |
| Provider name | dataloader |
|---|---|
| Type | Polling provider |
Table of Contents
| Table of Contents | ||||||
|---|---|---|---|---|---|---|
|
Configuration
In order to use your custom made dataloader plugins, you will first need to upload the dataloader plugin using the `squirro_asset` command line utility as described here.
Once the dataloader plugin has been uploaded successfully to the server, it is available to use from the Python SDK
| Field | Description |
|---|---|
| dataloader_options | Dict of the options required by squirro_data_load. |
| dataloader_plugin_options | Dict of all the options required by the plugin being used. Can be retrieved by doing a list call on the `squirro_asset` utility for the corresponding plugin. |
Configuration Example
Once the dataloader plugin has been uploaded, you can use the python SquirroClient to create a new subscription out of the dataloader plugin you just uploaded. One example of that is show below
| Info |
|---|
Make sure to use the `plugin_name` key in the `dataloader_options` dict to specify which dataloader plugin should be used. |
| Code Block |
|---|
dataloader_plugin_options = {
'hipchat_room_blacklist': 'squirro',
'hipchat_conversation_length': 3,
'hipchat_authtoken': '1N9lKixOJnEdPJ4MaTDFEcgRvaSd02xdZka3ENgG',
'hipchat_conversation_age': 2,
'hipchat_mode': 'message',
'hipchat_url': 'https://www.hipchat.com'
}
dataloader_options = {
'plugin_name': 'hipchat_thumbnail',
'reset': True,
'map_id': 'id',
'map_created_at': 'date',
'map_title': 'title',
'source_name': 'HipChat',
'map_body': 'body',
'facets': {
'emoticons': {
'visible': True,
'delimiter': '!',
'name': 'Emoticons',
'searchable': True,
'group_name': 'HipChat'
},
'room': {
'visible': True,
'name': 'Room',
'searchable': True,
'group_name': 'HipChat'
},
'System': {
'visible': True,
'display_name': 'System'
},
'participants': {
'visible': True,
'delimiter': ';',
'name': 'Participants',
'searchable': True,
'group_name': 'HipChat'
},
'names': {
'visible': True,
'delimiter': ';',
'display_name': 'Names'
},
'Type': {
'visible': True,
'display_name': 'Type'
},
'grouping': {
'visible': True,
'name': 'Grouping',
'group_name': 'HipChat'
}
}
}
client = SquirroClient(None, None, cluster='https://demo.squirro.net/')
client.authenticate(refresh_token='293d…a13b')
client.new_subscription(project_id, object_id='default', provider='dataloader',
config={'dataloader_options': dataloader_options, 'dataloader_plugin_options': dataloader_plugin_options}) |