Connectors are the interface between an external data source and Squirro. They ensure that any new content from the source system is delivered to Squirro automatically. Connectors are suited very well for connecting very large source data sets, of which only a partial selection needs to be imported into Squirro. For example, the built-in Twitter connector will only import tweets for users or keywords which a user has actively subscribed to from Squirro.
For regular imports of smaller data sets where no user interaction is required for data selection, the Loading Tools or the SDK tools are less complex options.
Table of Contents
Interaction with connectors
Connectors are registered in Squirro. When adding a source, a connector and configuration need to be specified (see the Create Subscription resource). The corresponding connector is then contacted to register the new configuration.
From that moment in time, the connector will deliver any items matching the configuration to Squirro.
Connector Types
Most connectors are implemented either by regularly polling the source system or by implementing some push technology such as webhooks.
Polling Connectors
A polling connectors regularly checks with the source system if new data is available. Whenever that is the case, the connectors then pushes the data into Squirro.
Because polling connectors need to check the source system continually, each check may incur charges in the source system. To reduce these costs, Squirro's internal polling connectors usually implement a back-off for retrying. The poll interval is continually increased for sources where no new data is available.
Push Connectors
Push connectors are notified by their source system when new data is available. For some connectors these updates are near real-time.
Pricing of the source system is generally done per notification or per item.
Configuration
Connector configurations are passed in as JSON dictionaries. The supported configuration keys depend on the individual connectors and are documented in the respective section.
Built-in Connectors
This page can now be found at Built-In Data Connectors on the Squirro Docs site.
Custom Connectors
The reference documentation for how to write a custom data connectors is available on the Custom Connectors page.