The Data Loader uses config files for facet creation and for pipelets. The config file for facets has a HJson format (which is a more human friendly superset of Json) and the one used for pipelets has a Json format. In the following chapters you can find the description of the attributes used in these two files and some examples.
Table of Contents
| Table of Contents | ||||
|---|---|---|---|---|
|
Facets config file
Sample file
Below is an example of a fully valid Hjson file which is not a valid Json. Notice how some of the keys/values are missing quotes and some rows are missing commas a the the end.
| Code Block | ||
|---|---|---|
| ||
{
"InteractionSubject": {
"name": "Interaction Subject",
"display_name": Subject,
"group_name": Interactions,
"visible": true,
"searchable":true,
"typeahead":true
},
"InteractionType": {
"name": "Type of Interaction ",
"display_name": "Type",
"group_name": "Interactions",
visible: true,
searchable:true,
typeahead:false
},
"Date": {
"name": "Date of creation",
"data_type": "datetime",
"input_format_string": "%Y-%m-%dT%H:%M:%S"",
default_value:"1900/01/01"
"visible": true
},
"EntittledUsers": {
"name": "Users",
"auth": true
"auth_value_mandatory": true,
import: false
},
"InternalAttendees": {
"name": "Attendees",
"group_name": "Interactions"
"pivotal_group": "Members",
"delimiter": ";"
"visible": true
},
NoAtendees: {
name: "Number of Atendees",
"data_type": "int",
"visible": true
"searchable":false
},
} |
Structure
In the examples above, the curly brackets at the root of the file open a new dictionary. Within each dictionary there are a number of facet keys, which, in turn have their own dictionary of key / value pairs - their attributes.
Reference
The facets config file contains the following attributes.
...
auth
...
Boolean
...
If set to 'true' for a field, this field will be used to determine user access to the Squirro item. Default 'false'.
...
auth_value_mandatory
...
Boolean
...
If value of the filed in the data source is empty, the item will not be uploaded.
...
data_type
...
String
...
Data type of the facet. One of ('string', 'int', 'float', 'datetime'). The default is 'string'.
...
default_value
...
String
...
Default value of the facet if value is None.
...
delimiter
...
String
...
Specified for columns that have multiple values, i.e. pivotal and auth columns.
...
display_name
...
String
...
Name to show to the user in the front-end.
...
format_instr
...
String
...
Formatting instruction for the facet value display. This parameter defines how the Squirro UI will display the value. This is only relevant for int, float and dates. It’s also documented here.
...
group_name
...
String
...
Label of the group this facet is under.
...
import
...
Boolean
...
For specifying if a column will be loaded as a facet or not. Default 'true'.
...
input_format_string
...
String
...
Format of the date string coming from the source (used for csv and excel). When importing from database, the date values are of type datetime and don’t need a format. Only for datetime columns.
...
String
...
Name of the facet in queries and on incoming items.
...
output_format_string
...
String
...
Format of dates expected by Squirro. Keep the default if this param is not supplied. For the moment, the only accepted format is the Squirro date format '%Y-%m-%dT%H:%M:%S'. Only for datetime columns.
...
pivotal_group
...
String
...
If this attribute exists than the column is also a pivotal column. That means that the values of that column, delimited by 'delimiter', were joined from a detail table to generate only one line in the source. You can find more information here.
...
searchable
...
Boolean
...
If 'true' this facet values will be searchable. Default 'false'.
...
visible
...
Boolean
...
If `false` this facet will be hidden in the front-end. Default 'true'.
Pipelets config file
When creating a new pipelet it is mandatory to create a config file which references the pipelet itself.
Sample file
Below is a valid JSON file used for pipelet configuration.
| Code Block | ||
|---|---|---|
| ||
{
"DummyPipeletClass":{
"file_location":"pipelets\dummy_pipelet.py",
"stage":"before templating",
"config": {
"restricted_fields_columns": ["HeaderRestrictedFieldNames", "DetailRestrictedFieldNames"]
}
},
"DummyPipeletClass2":{
"file_location":"pipelets\dummy_pipelet.py",
"stage":"after templating",
"config": {
"restricted_fields_columns": ["HeaderRestrictedFieldNames", "DetailRestrictedFieldNames"]
}
}
} |
Structure
The format of the pipelets config file is Json. In the examples above, the curly brackets at the root of the file open a new dictionary. Within each dictionary there are a number of keys, each representing a pipelet class. The attributes of each class will be described below.
Reference
The pipelets config files contains the following attributes.
...
config
...
Dictionary
...
Contains the attributes of each piplet. This dictionary is passed in as a constructor argument to the pipelet.
...
file_location
...
String
...
The location of the Python file. This is mandatory. It can be the full path or the relative location from where the tool is executed
...
stage
...
String
Specifies when the pipelets consume method will be executed.
Possible values:
...
This page can now be found at Data Loader Command Line Interface Tool on the Squirro Docs site.