Providers are the interface between an external data source and Squirro. They ensure that any new content from the source system is delivered to Squirro automatically. Providers are suited very well for connecting very large source data sets, of which only a partial selection needs to be imported into Squirro. For example, the built-in Twitter provider will only import tweets for users or keywords which a user has actively subscribed to from Squirro.
For regular imports of smaller data sets where no user interaction is required for data selection, the Loading Tools or the SDK tools are less complex options.
Table of Contents
Interaction with Providers
Providers are registered in Squirro. When adding a source, a provider and configuration need to be specified (see the Create Subscription resource). The corresponding provider is then contacted to register the new configuration.
From that moment in time, the provider will deliver any items matching the configuration to Squirro.
Provider Types
Most providers are implemented either by regularly polling the source system or by implementing some push technology such as webhooks. For Squirro's internal providers, the type has an impact on pricing.
Polling Providers
A polling provider regularly checks with the source system if new data is available. Whenever that is the case, the provider then pushes the data into Squirro.
Because polling providers need to check the source system continually, each check may incur charges in the source system. To reduce these costs, Squirro's internal polling providers usually implement a back-off for retrying. The poll interval is continually increased for sources where no new data is available.
Push Providers
Push providers are notified by their source system when new data is available. For some providers these updates are near real-time.
Pricing of the source system is generally done per notification or per item.
Configuration
Provider configurations are passed in as JSON dictionaries. The supported configuration keys depend on the individual provider and are documented in the respective section.
Built-in Providers
Custom Providers
The reference documentation for how to write a custom data provider is available on the Custom Provider (1) page.