Squirro items are represented in JSON format. This is true both for data loading and data consumption with the API. The following tables documents the properties a Squirro item can have.
Table of Contents
About Squirro Items
When executing a search, Squirro will show the user a list of matching items as the query result.
When planning and integrating a custom data source the following points should be considered:
- What is the smallest independent result entity that the user should be consuming? These should then be modelled as Squirro items.
- The formatting of the body content and title.
Some examples of individual items are:
- News story, web article, tweet, etc.
- Binary document (PDF, Office documents, etc.)
- Service ticket
- Chat message
- …
Items can also contain sub-items which are always shown in the context of the full item. Typically Squirro uses these sub-items for indexing the individual pages or chapters of PDF documents as separate sub-items.
Fields
Common Fields
The fields in this table are used in both the data loading and data consumption APIs.
Field | Data type | Description | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
id | Unique Identifier | While this field is called the same in data loading and consumption, it has different semantics. See the sections Item Format#Data Loading Fields and Item Format#Data Consumption Fields for details. | ||||||||
link | URL | Link to the item at its original location. | ||||||||
title | String | Item title. | ||||||||
body | HTML String | Item body. This field is in HTML format and special characters need to be escaped. | ||||||||
language | Language Codes | Content language of the item. If this is not specified, it is auto-detected based on the content. | ||||||||
created_at | Date and Time | Item creation date. Ideally, this is the creation date of the item in its source system. If this is not specified for data loading, the import process goes through the following steps:
| ||||||||
webshot_url | URL | Main item picture. This image is displayed in the result list to represent the story. For data loading, the If this is not set, it is automatically extracted from the web site specified with the | ||||||||
webshot_height | Integer | Height of the webshot in pixels. | ||||||||
webshot_width | Integer | Width of the webshot in pixels. | ||||||||
keywords | Dictionary, values represented as lists | Keywords attached to the item (see Facets for full documentation). They are the structured information of an item.
Keyword values can have different data types. Please reference the Data Types section of the Facets documentation for details and format specification. The default data type is string. To use other formats, configure it before loading any data into the system. See Facets API for information. Example item with keywords: { "title": "Our offices", "body": "We have offices in Munich, …", "keywords": { "country": ["Germany"], "city": ["Munich", "Berlin"] } } | ||||||||
entities | List of dictionaries | Entities attached to the item. See the documentation on the Catalyst Data Model for the data structure of individual entities. | ||||||||
comments | List of dictionaries. | A list of comments that are attached to the item. The user can toggle the display of these comments in the web interface on top of the item body. Three fields can be added to comments:
|
When importing data into Squirro at least one of the fields title
, body
or files
must be set. All other fields are optional.
Data Loading Fields
These fields can be specified in the data loading APIs. They will be transformed and output with different names in the data consumption APIs.
Field | Data type | Description |
---|---|---|
id | Unique Identifier | External item identifier. When a value is specified here at import, it is written into the Used by data providers to reference their source system. Squirro uses this identifier for deduplication. |
summary | Text String | Item summary text. If not specified, this is generated from the |
webshot_picture_hint | URL | Main item picture. If this URL exists and can be downloaded, the image is archived by Squirro. The resulting URL is written into the Note: processing of webshots is disabled by default for custom data imports (bulk provider). |
mime_type | String | The MIME type of the body. Set to text/html for HTML bodies. For all other types, a conversion to HTML will be attempted. If not specified, the MIME type is auto-detected. See the Content Conversion step for more details. |
files | List of dictionaries. | A list of files that are uploaded for the item. Note: this is modelled as a list, but only one file can currently be attached. The fields for individual files are:
Note, that at consumption this field also exists, but has a different layout. See below. |
Data Consumption Fields
Some fields are only available during data consumption because they are calculated on the fly or represent a user state. This table documents these fields.
Field | Data type | Description |
---|---|---|
id | Unique Identifier | Internal item identifier, generated by Squirro only. |
external_id | String | External item identifier. The external identifier is used for deduplication and can be used to link items to their source system. See the |
read | Boolean | True if the item has been read. |
starred | Boolean | True if the item has been starred. |
abstract | Text String | Item abstract. This is generated from the In case the item is returned as a matching result to a query search, the abstract is calculated around the most relevant matching keywords. |
score | Float | Relevant score of the item. This is only set when the result list is ordered by relevance. |
thumbler_url | Partial URL | Used internally by Squirro to display thumbnails of the webshot_url field. |
explanation | Dictionary | Returned for items when the Example: { … "explanation": { "matches": { "summary.stemmed": [ {"term": "eliminated", "score": 0.010479515}, {"term": "equipped", "score": 0.00846127} ], "body.stemmed": [ {"term": "eliminated", "score": 0.010341313000000001}, {"term": "equipped", "score": 0.008302803000000001} ], "language_code": [ {"term": "en", "score": 0.0009886466000000001} ] } } } |
related_items | List | Returned when the Example: { … "related_items": [ {"id": "UfA8Ah08TeSLSUo-RBzm7Q"}, {"id": "tjiS4mjaTgupKaIiZisYww"} ] } |
highlight_matches | Dictionary | A dictionary of matching query terms per field. { … "highlight_matches": { "body": ["asia"], "summary": ["asia"] }, … } |
matching_sub_items | List | List of sub-items (e.g. pages for a PDF) with matches for the current query. |
has_matching_sub_items | Boolean | True if the item consists of sub-items (e.g. pages for a PDF) that matches the current query. |
subscriptions | List | Returns a list of subscription details. { … "subscriptions": [{ "title": "Upload", "object_ids": ["-RmFf-hkQLq0GX8IOE_WvQ"], "link": null, "provider": "bulk", "source_id": "KJIMuj2DTM6UpSbdboqYIA", "id": "n_ualpEOTLqAW7e6sBPv1w" }], … } |
objects | List | Returns a list of object details. { … "objects": [{ "subscription_ids": ["n_ualpEOTLqAW7e6sBPv1w"], "project_id": "q6-ODvBGQby1vxFjckq3eg", "type": "default", "id": "-RmFf-hkQLq0GX8IOE_WvQ", "title": "default" }], … } |
files | List | A list of files that are uploaded for the item. Note: this is modelled as a list, but only one file can currently be attached. The fields for individual files are:
Note, that at ingestion this field also exists, but has a different layout. See above. |