Page Comparison

id

Unique Identifier

While this field is called the same in data loading and consumption, it has different semantics. See the sections

Item Format#Data Loading Fields and Item Format#Data Consumption Fields for details.

link

URL

Link to the item at its original location.

title

String

Item title.

body

HTML String

Item body. This field is in HTML format and special characters need to be escaped.

language

Language Codes

Content language of the item. If this is not specified, it is auto-detected based on the content.

created_at

Date and Time

Item creation date. Ideally, this is the creation date of the item in its source system.

If this is not specified for data loading, the import process goes through the following steps:

• If the files property is specified, Squirro tries to extract a creation date from the file metadata.

• As a fallback, the server's current date and time are used.

webshot_url

URL

Main item picture. This image is displayed in the result list to represent the story.

For data loading, the webshot_picture_hint field should be used because the picture will then automatically be archived.

If this is not set, it is automatically extracted from the web site specified with the link property, generally by using the first story picture.

webshot_height

Integer

Height of the webshot in pixels.

webshot_width

Integer

Width of the webshot in pixels.

keywords

Dictionary, values represented as lists

Keywords attached to the item (see Facets for full documentation). They are the structured information of an item.

• Item keywords are offered as filter options in the search screen.

• Keyword and their values are offered in the search field as typeahead options.

• Search Tagging, known entity extraction and /wiki/spaces/KB/pages/2949480 are used to add additional keywords to items.

Keyword values can have different data types. Please reference the Data Types section of the Facets documentation for details and format specification. The default data type is string. To use other formats, configure it before loading any data into the system. See Facets API for information.

Example item with keywords:

{
  "title": "Our offices",
  "body": "We have offices in Munich, …",
  "keywords": {
    "country": ["Germany"],
    "city": ["Munich", "Berlin"]
  }
}

entities

List of dictionaries

Entities attached to the item. See the documentation on the Catalyst Data Model for the data structure of individual entities.

id

String

External item identifier. When a value is specified here at import, it is written into the external_id data consumption field.

Used by data providers to reference their source system. Squirro uses this identifier for

 deduplication.

summary

Text String

Item summary text. If not specified, this is generated from the body field. Any HTML tags are removed.

webshot_picture_hint

URL

Main item picture. If this URL exists and can be downloaded, the image is archived by Squirro. The resulting URL is written into the webshot_url field. The picture width and height are calculated and written into webshot_width and webshot_height.

Note: processing of webshots is disabled by default for custom data imports (bulk provider).

mime_type

String

The MIME type of the body. Set to text/html for HTML bodies. For all other types, a conversion to HTML will be attempted. If not specified, the MIME type is auto-detected. See the Content Conversion step for more details.

files

List of dictionaries.

A list of files that are uploaded for the item. Note: this is modelled as a list, but only one file can currently be attached.

The fields for individual files are:

• content: Base64-encoded content of the file to upload. This or the url field are mandatory.

• url: URL where the file can be downloaded from. This or the content field are mandatory.

• name: File name without path. Mandatory when the content field is provided. If the url field is provided, the name is derived from the URL by default.

Note, that at consumption this field also exists, but has a different layout. See below.

id

Unique Identifier

Internal item identifier, generated by Squirro only.

external_id

String

External item identifier. The external identifier is used for deduplication and can be used to link items to their source system.

See the id field in the data loading fields for details.

read

Boolean

True if the item has been read.

starred

Boolean

True if the item has been starred.

abstract

Text String

Item abstract. This is generated from the summary field, or if that field doesn't exist, from the body.

In case the item is returned as a matching result to a query search, the abstract is calculated around the most relevant matching keywords.

score

Float

Relevant score of the item. This is only set when the result list is ordered by relevance.

thumbler_url

Partial URL

Used internally by Squirro to display thumbnails of the webshot_url field.

explanation

Dictionary

Returned for items when the explain_smartfilters option is used in the Get Item resource. The dictionary contains a list of fields, for each of which the matching Smart Filter tokens are listed.

Example:

{
  …
  "explanation": {
    "matches": {
      "summary.stemmed": [
        {"term": "eliminated", "score": 0.010479515},
        {"term": "equipped", "score": 0.00846127}
      ],
      "body.stemmed": [
        {"term": "eliminated", "score": 0.010341313000000001},
        {"term": "equipped", "score": 0.008302803000000001}
      ],
      "language_code": [
        {"term": "en", "score": 0.0009886466000000001}
      ]
    }
  }
}

related_items

List

Returned when the filter_related_items option is set in the List Items resource. A list of dictionaries which contain the field id of any related (duplicate) items.

Example:

{
  …
  "related_items": [
    {"id": "UfA8Ah08TeSLSUo-RBzm7Q"},
    {"id": "tjiS4mjaTgupKaIiZisYww"}
  ]
}

highlight_matches

Dictionary

A dictionary of matching query terms per field.

{
  …
  "highlight_matches": {
    "body": ["asia"],
    "summary": ["asia"]
  },
  …
}

matching_sub_items

List

List of sub-items (e.g. pages for a PDF) with matches for the current query.

has_matching_sub_items

Boolean

True if the item consists of sub-items (e.g. pages for a PDF) that matches the current query

.

files

List

A list of files that are uploaded for the item. Note: this is modelled as a list, but only one file can currently be attached.

The fields for individual files are:

• content_url: content:/// URL where the file is stored. This includes the storage bucket name and the path within that bucket. See /wiki/spaces/KB/pages/88187790 for information on storage buckets.

• mime_type: MIME type of the file.

• name: File name without path.

Note, that at ingestion this field also exists, but has a different layout. See above.