...
Items are only displayed to the users once the full pipeline - with exception of Search Tagging - has run through. For details on the search tagging delay, see the Search Tagging and Alerting documentation.
The sqingesterd
service is responsible for executing the Pipeline workflows and their steps.
The configuration option processors
under the section ingester
of the /etc/squirro/ingester.ini
file controls the number of processors used by the sqingesterd
service to consume the batch files found under the /var/lib/squirro/inputstream
directory. Each processor works on a single batch file at a time. Under the hood, each processor is a separate Unix process. The default value of this option is 1 (i.e., a single processor is spawned by the service for ingesting data).
The configuration option workers
under the section processor
of the /etc/squirro/ingester.ini
file controls the number of threads spawned by each processor. This setting is being used for the execution of certain Pipeline steps which consume a single item from the batch at a time. Other Pipeline steps work on a batch level and therefore this option is irrelevant for them. The default value of this option is 3. (i.e., approximately 3 items of a batch are executed concurrently by a single processor).
Configuration
A project can have one or more pipelines. Each data source is associated with one such pipeline. The pipelines are configured using the Pipeline Editor in the Setup space.