The squirro_asset tool can be used to manage all Squirro code artefacts called custom assets. Assets are Pipelets, Custom Widgets, and Custom Dashboard Loaders.
Table of Contents
squirro_assets tool
Starting with the 2.4.5 Squirro release, you can use the squirro_asset tools to manage all the custom "code artifacts" in Squirro server systems such as Dashboard Loaders, Dashboard Widgets, or Pipelets.
The first parameter is the type of asset: dashboard_loader, widget, or pipelet, and if at any time you are unsure of what parameters are allowed, you can request help by adding the --help parameter.
> squirro_asset --help
usage: squirro_asset [-h] [--verbose] [--log-file LOG_FILE] [--version]
{widget,pipelet,dashboard_loader,dataloader_plugin} ...
Squirro asset management tool
positional arguments:
{widget,pipelet,dashboard_loader}
asset type help
widget Manage custom widgets
pipelet Manage custom pipelets
dashboard_loader Manage custom dashboard_loader
dataloader_plugin Manage custom dataloader_plugin
General Options:
-h, --help Show this help message and exit.
--verbose, -v Show additional information.
--log-file LOG_FILE Log file on disk.
--version show program's version number and exit
Dataloader plugin
Note that the dataloader plugins using the deprecated method `add_args` can not be used with the self service Squirro dataloader UI. Please migrate the `add_args` method to `getArguments` method as listed on this page
You can upload new or modified dataloader plugins using the upload command, for example:
> squirro_asset dataloader_plugin upload --folder <dataloader-plugin-folder> --token <token> --cluster <cluster-url>
where <dataloader-plugin-folder> is a directory containing the following files:
- Dataloader plugin python file
- A `requirements.txt` listing any external python dependencies needed for the plugin
- A png file to be used as a logo for the dataloader plugin in the Squirro UI
- A `dataloader_plugin.json` file containing the proper metadata for the dataloader plugin (see format below) in hjson format
Format of `dataloader_plugin.json` file. Please note that the comments in the json file are only for display purposes and will not be treated as valid JSON when uploading the plugin. In the future, we intend to change the format to HJSON to ensure that the presence of comments do not break the upload of the dataloader plugin.
{
# title of the dataloader plugin to be displayed in the UI
"title": "Hipchat",
# descriptuon of the dataloader plugin to be displayed in the UI
"description": "For importing data into hipchat",
# Name of the plugin file
"plugin_file": "hipchat.py",
# Name of the requirements file containing the python dependencies
"requirements_file": "requirements.txt",
# Category under which to show this dataloader plugin, Possible options are:
# 'web', 'socialMedia', 'import', 'developers', 'enterprise', 'businessIntelligence', 'crm', 'itsm'
"category": "developers",
# Name of the PNG file to be used as a thumbnail while displaying the dataloader plugin in the UI
"thumbnail_file": "Hipchat_Atlassian_logo.png"
}
You can list all dataloader plugins using the list option
> squirro_asset dataloader_plugin list --token <token> --cluster <cluster-url>
[
{
"requirements_file": "requirements.txt",
"category": "developers",
"config_options": [
{
"required": true,
"name": "hipchat_url",
"help": "URL of HipChat public web interface."
},
{
"name": "hipchat_endpoint",
"help": "Endpoint of the HipChat API (URL)"
},
{
"required": true,
"name": "hipchat_authtoken",
"help": "Authentication token for HipChat API"
},
{
"default": 3,
"type": "int",
"name": "hipchat_conversation_length",
"help": "Number of days back into the past for which to fetch the message archive counting back from yesterday"
},
{
"default": 1,
"type": "int",
"name": "hipchat_conversation_age",
"help": "Number of days back into the past for which to fetch the message archive counting back from yesterday"
},
{
"name": "hipchat_room_blacklist",
"help": "Space-separated list of rooms to ignore"
},
{
"default": "message",
"name": "hipchat_mode",
"help": "Whether to individual messages (realtime) or groups conversations (older than 1 day)"
}
],
"hash": "d8c6ffe1-1886-11e7-a5a4-f45c89acc4cb",
"description": "For importing data into hipchat",
"title": "Hipchat",
"pipelets_files": [
"keywords_pipelet1.py",
"sentiment_pipelet2.py"
],
"thumbnail_file": "Hipchat_Atlassian_logo.png",
"name": "hipchat_thumbnail",
"body_template_file": "body.j2",
"thumbnail_b64_encoded": "+fn5/f3+AgICBgYGCgoKDg4OEhISFhYWGhoaHh4eIiIiJiYmKioqLi4uMjIyNjY2Ojo6Pj4+",
"plugin_file": "hipchat.py",
"resources": {
"txt": [
"requirements.txt"
],
"py": [
"keywords_pipelet.py",
"hipchat.py",
"load.py"
],
"pyc": [
"hipchat.pyc"
],
"png": [
"Hipchat_Atlassian_logo.png"
]
},
"title_template_file": "title.j2"
}
]
Dashboard loaders
You can upload new or modified dashboard loaders using the upload command, for example:
> squirro_asset dashboard_loader upload --folder <dashboard-loader-folder> --token <token> --cluster <cluster-url>
where <dashboard-folder> is a directory containing dashboard loader html, css and javascript files along with a file named dashboard_loader.json formatted in hjson, and you specify one of the options to connect to a Squirro server installation (using <token> and <cluster-url> in the above example).
You can list all dashboard loaders using the same parameters as for upload except --folder:
> squirro_asset dashboard_loader upload --token <token> --cluster <cluster-url>
[
{
"hash": "ec43609e-c5c6-11e6-ac8a-80e65013fba6",
"name": "name-1",
"title": "title-1",
"directory": "/var/lib/squirro/topic/assets/dashboard_loader/tenant",
"resources": {
"css": [
"mycss.css"
]
}
},
{
"directory": "/var/lib/squirro/topic/assets/dashboard_loader/tenant",
"hash": "bed8e082-c395-11e6-94ff-000c29dfbc49",
"name": "name-2",
"resources": {
"html": [
"main.html"
]
},
"title": "title-2"
}
]
Dashboard widgets
Starting with Squirro release 2.4.5 you can also use the squirro_asset tool in place of the squirro_widget tool to upload and list custom dashboard widgets using the same options as for dashboard loaders.
To upload:
> squirro_asset widget upload --folder <dashboard-widget-folder> --token <token> --cluster <cluster-url>
and to list widgets:
> squirro_asset widget list --token 0da7dc97bdacebf6123fb61f34482f247b8eb37f5a19cba0e296851b5094e77aa5edbc96041f34ad3aba43bd4708267f1cf09b41028137cf57e53f7ae9829998 --cluster https://testing.squirro.net
[
{
"hash": "9a6cee78-28e5-11e6-90b4-843835563242",
"name": "name-1",
"title": "title-1",
"directory": "/var/lib/squirro/topic/widgets/tenant",
"resources": {
"html": [
"cells.html",
"row.html",
"widget.html",
"header.html"
],
"css": [
"widget.css"
],
"js": [
"widget.js"
]
}
},
{
"directory": "/var/lib/squirro/topic/widgets/tenant",
"hash": "228da928-c076-11e6-b6c3-80e65013fba6",
"name": "name-2",
"resources": {
}
}
]
Pipelets
Also with release 2.4.5 you can use the squirro_asset tool in place of the pipelet tool mentioned under Pipelets to manage pipelets. Instead of
> squirro_pipelet <any options>
you can using squirro_asset followed by pipelet as asset type like so:
> squirro_asset <any options used for squirro_pipelet>
For convenience you can use the --help option to obtain detailed help on squirro_asset pipelet commands and options:
> squirro_asset pipelet --help
usage: squirro_asset pipelet [-h]
{consume,list,upload,source,rerun,validate} ...
positional arguments:
{consume,list,upload,source,rerun,validate}
sub-command help
validate Validates that the script doesn't violate any
restrictions
consume Runs one or more items through the consume method of
the pipelet
rerun Re-runs a pipelet on the server
upload Uploads the script to Squirro
source Creates a source that is configured with `name`
list List all available pipelets
optional arguments:
-h, --help show this help message and exit