Squirro also distributes some connectors as data loader plugins for maximum flexibility. Please see the section /wiki/spaces/DOWN/pages/54624262 in the downloads section (license required).
Table of Contents
Introduction
Data loader plugins are created as a Python class (see the DataSource Class reference for how exactly).
That plugin is then loaded into the data loader using the --script-file
parameter. For example:
squirro_data_load ^ -v ^ --cluster %CLUSTER% ^ --project-id %PROJECT_ID% ^ --token %TOKEN% ^ --source-script medline.py ^ --map-id PMID ^ --map-title TI
Tutorial
The Data Loader Tutorial section Custom Data Source shows how to create a custom data loader plugin.
Class Reference
See the DataSource Class.
Custom Packages
Data loader plugins sometimes need custom Python packages. These can be provided in the pkg
folder, relative to where the data loader is invoked.
The recommendation is to include a requirements.txt
file in the folder where the data loader plugin is located. That file lists all the required packages, with one package dependency per line. For example:
dateutils requests
To download these packages into the pkg
folder, execute the following command (the pkg
folder must exist before calling this command):
pip install -r requirements.txt --download pkg