Versions Compared

Old Version 11

changes.mady.by.user Filippo Broggini

Saved on

compared with

New Version Current

changes.mady.by.user Nathan Gosse

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Excerpt

The data loader can be extended with data loader plugins. Those plugins are used to create custom data connectors, to sources that are not yet included in Squirro.

Squirro also distributes some connectors as data loader plugins for maximum flexibility. Some data loader plugins can also be installed from Squirro's repository. See Plugin Repository for more.

Usage

Data loader plugins can be used in Squirro's user interface, or on the command line with the Data Loader.

When using a data loader plugin in the user interface, it behaves exactly the same way as the built-in connectors. You can configure some options, select the mappings, and save the source.

On the command line, the plugin can be loaded by specifying the --source-script option. For example:

Code Block
languagebash
squirro_data_load ^
    -v ^
    --cluster %CLUSTER% ^
    --project-id %PROJECT_ID% ^
    --token %TOKEN% ^
    --source-script medline.py ^
    --map-id PMID ^
    --map-title TI
Expand
titleLinux
Code Block
languagebash
squirro_data_load \
    -v \
    --cluster $CLUSTER \
    --project-id $PROJECT_ID \
    --token $TOKEN \
    --source-script medline.py \
    --map-id PMID \
    --map-title TI

As can be seen, --source-script is used in place of the --source-type option. Apart from that, the behaviour is identical, and mapping, facet configuration, etc. all behave identically.

Writing plugins

Data loader plugins are implemented as Python classes. The topic is covered in Writing a custom data loader.This page can now be found atData Loader Plugins on the Squirro Docs site.