We have made several improvements in the dataloader UI to provide better control to the user. They are listed below:
Source Batch Size
The Squirro dataloaders now support the setting of source batch size. Source batch size refers to the number of items fetched from the source at once and is set to default at 1000.
...
Ingestion Batch size
One can also set an ingestion batch size by enabling “Use custom ingestion batch size“. Ingestion batch size refers to the number of items ingested in Squirro at once after they are fetched from the source.
...
Ingestion Limit
The maximum number of items to be ingested in Squirro can now be set by enabling “Use custom max no. of items to ingest“.
...
Kill a running dataloader job
If the dataloader job is running, it is possible to kill the job using “Kill“ option shown below:
...
Logs for individual dataloader job
The logs for each dataloader job can be fetched by clicking on the “Logs“ option.
...
Shown below are the logs of an example dataloader job:
...
To improve the log visibility for the datasources, users can now fetch the logs per source. These logs are placed in /var/lib/squirro/datasource/source_logs. Given a project_id and a source_id, the log file is represented as project_<project_id>_source_<source_id>.log
Language field mapping
While creating/editing a source, in addition to other Squirro item fields, “Language” can now be mapped with an incoming source field.
...
Entities transfer using Squirro Project:
While using the Squirro project dataloader, the entities can now be transferred from the source project. This can be set in the “Configure source” step while importing data from Squirro project as shown below.
...
The entities corresponding to squirro items will be imported as shown below:
...
This page can now be found at UI Data Loader on the Squirro Docs site.