Known Entity Extraction (KEE) is configured using a configuration file in the root of the KEE project. That file is called config.json
and is written in the JSON format.
The format used by the config file is JSON, or rather a more human-friendly superset called Hjson. First, an example of a fully valid JSON file which is also valid Hjson:
{ "sources": { "demo": { "source_type": "csv", "source_file": "demo.csv", } }, "testing": { "fixtures_dir": "fixtures/", "snapshots_dir": "snapshots/" } } |
With Hjson it is possible to write this file in a more human-friendly way. The main improvements are comments and trailing commas (which are not allowed in standard JSON). Additionally most of the quotes can be left out if so desired.
{ sources: { demo: { // The data is in a CSV file in the current directory "source_type": "csv", "source_file": "demo.csv", }, }, /* Just repeating the default settings for testing here */ testing: { fixtures_dir: "fixtures/", snapshots_dir: "snapshots/", } } |
A full reference of Hjson is available on hjson.org.
In the examples above the curly brackets at the root of the file open a new dictionary. Within each dictionary there are a number of key / value pairs. For example, the keys of the top level dictionary shown above are sources
and testing
.
Each entry in the top-level dictionary indicates a different section of the KEE configuration. This reference describes the usage of each of the different sections below.
The kee
section configures a few basic parameters of the whole process. The full section is optional.
Key | Data Type | Description |
---|---|---|
pipelet | String | The name of the pipelet when uploading the KEE project to the Squirro server. This should be unique across your KEE projects to avoid any collision. |
database | String | File name, relative to the configuration file directory, where the lookup database is located. Default: "db/lookup.json" |
version | String | A version indicator for the current KEE configuration. This can be modified whenever the strategy or the data changes significantly, thus warranting a re-tagging of previous items. The kee rerun command makes use of this information to select older items for re-tagging. Default: not set |
version_keyword | String | Name of the keyword to use for version tagging on items. If the Default: "KEE Version" |
debug | Boolean | If Default: false |
example usage:
{ "kee": { "database": "db/product_list.json", "debug": false } } |
To add a KEE project to a Squirro project, some of the cluster information needs to be configured in the squirro
section.
See Connecting to Squirro for how to get this information.
This section is optional if you are not using the commands that connect to Squirro (get_fixture
or upload
).
Key | Data Type | Description |
---|---|---|
cluster | String | The endpoint where Squirro has been installed. |
token | String | The authentication token with which to log into the system. This token should be treated confidentially, so if the config file is shared, the token should not be included there. See the environment variables section for an alternative. |
project_id | String | Project identifier where the Known Entity Extraction is being used. |
example usage:
{ "sources": { "cluster": "http://www.mysquirrocluster.com", "token": "MY-ACCESS-TOKEN", "project_id": "MY-PROJECT-ID" } } |
The sources
section in the configuration contains a list of all the data sources for building the lookup database.
Each entry is in itself an object with the required configuration for the source. A partial example:
{ "sources": { "clients": { // The keys from the reference below come here }, "employees": { // The keys from the reference below come here }, … } } |
Each source takes the configuration. This contains first the actual data connection. For this, the Data Loader options, including Data Loader Plugins can be used. The second part is then the KEE behaviour for that particular source.
Key | Data Type | Description | |
---|---|---|---|
Data Source | |||
source_type | String | Which source type to connect to. Valid values:
| |
source_script | String | The Data Loader Plugin to load. This and the source_type are mutually exclusive. | |
... | Additional connection options are specified as key/value pairs as well. Use the same options as for the data loader, with the dashes replaced with underscores. For example if the data loader is invoked with
Check the Data Loader Reference for all the possible options. Plugin-specific options are documented in each plugin. | ||
KEE Configuration | |||
strategy | String | The name of the strategy to use for matching on this data source. This needs to reference to a strategy key that has been defined in the Strategies section of the config file. | |
field_id | String | The name of the column in the input data which is the unique identifier of a row. | |
generate_id | Boolean | Automatically generate unique identifiers for rows. Can be used if (and only if) field_id is not specified. | |
field_matching | List | Names of the columns of the input data that contain the object names used for the KEE matching. This is generally the primary name and often an alias column. | |
hierarchy | String | Specifies a hierarchy in the data. This hierarchy can be used in the tagging for example to tag an item with the matching company and all of the parent companies as well. The format of this configuration is See the hierarchy section for examples. | |
multivalue | List | A list of column names that can contain multiple values. This is commonly used for the alias column. The default separator for multiple values in the source data is the pipe ( Example:
|
The optional testing
section allows specification of where the tests files and snapshots are located. See KEE Testing for details on the testing process.
Again a partial example for this:
{ "testing": { "fixtures_dir": "fixtures/", "snapshots_dir": "snapshots/" } … // Other keys omitted } |
The valid configuration keys in this section are:
Key | Data Type | Description |
---|---|---|
fixtures_dir | String | Folder name, relative to the configuration file directory, where the test fixtures are located. Default: "fixtures" |
snapshots_dir | String | Folder name, relative to the configuration file directory, where the snapshots are stored. Default: "snapshots" |
The section extraction
configures how the Squirro items are processed during the KEE process.
Key | Data Type | Description |
---|---|---|
item_fields | List | Which item fields to use for Known Entity Extraction. See Item Format for possible values. Default: |
A strategy in KEE defines how the mapping is executed and which keywords should be added to Squirro items upon a successful match.
The matching is modified depending on a number of factors, including:
The strategy is referenced in the input data and based on that name looked up in the strategies
configuration. The following incomplete example references a strategy called "companies" that is correspondingly defined in the strategies
section.
{ // Most keys omitted for brevity "sources": { "clients": { "source_type": "csv", "source_file": "…", "strategy": "companies" }, }, "strategies": { "companies": { "tokenizer": "default", // … }, }, } |
The following table describes all the configuration keys with which a strategy can be tailored to specific requirements.
Key | Data Type | Description | |
---|---|---|---|
Matching | |||
tokenizer | String | For processing the text input, the text is split into individual tokens. The Supported tokenizers:
| |
filters | List | Together with the Available filters are:
Default: by default only the | |
min_score | Float | How good a score is required for a token to match. 1.0 is a perfect match, 0.0 is no match at all. Use KEE Testing to find the right balance for each use case. Turning on verbose logging or tracing (see the Default: 0.9 | |
spellfix | Boolean | Allow small spelling mistakes. This allows at most one letter swap, so e.g. "Apple" and "Appel" will both match each-other. Default: false | |
blacklist | List | A list of entity names to ignore. If any of the | |
suffix_list | String | The suffix list that is used to remove common suffixes in the entity names. See the section suffix list below for details. | |
geo_strategy | String | How to deal with geographic names in entity names. Possible values:
| |
Keywords | |||
keywords | List | The keywords section defines which keywords are added to a Squirro item based on any matching entity. This is a list of keywords that can be added, where each individual entry contains the input file column to write and the keyword name into which to store it. The target value can make use of simple template substitution to add keyword names based on the data of the matching row. The syntax is a field name surrounded by curly brackets. Example:
| |
parent_keywords | List | The same setting as This recursively processes all parent entities (if any) and sets keywords on the item based on these rules. For this to work there must be a | |
clean_keywords | List | A list of keywords that should be removed from the items before applying the KEE tagging. This is useful when re-running KEE tagging to ensure that old keywords are removed. Example:
|
The suffix list is used by the strategy to ignore common suffixes. Examples for such suffixes:
These suffixes may often be omitted when writing about said entities and thus the can be ignored for matching.
To create a custom suffix list, add it to the suffix_list
section and define the various patterns as key/value pairs. The keys are currently ignored by the KEE extraction and can be used to group the tokens into countries or other logical groupings.
An example:
{ // Most keys omitted for clarity "strategies": { "orgs": { "suffix_list": "companies", } }, "suffix_list": { "companies": { "GLOBAL": ["Inc", "Limited"] "DEU": ["AG", "GmbH"] "ZAF": ["(Pty) Ltd", "LIMITED"] "INDUSTRY": ["Bank"] } }, } |
For improved KEE matching, Squirro can make use of a language model. That allows the matching to handle common words in entity names correctly. Two simple examples will show the possibilities:
The required frequency model is created using the ngram definition. Please contact support to get access to Squirro's pre-compiled language models.
An example configuration for ngram is as follows:
{ // Most keys omitted for clarity "strategies": { "orgs": { "ngram": "companies", } }, "ngram": { "companies": { "source": "ngram/", "whitelist": ["Apple"], } }, } |
The following is a reference of all of the keys in an individual ngram section.
Key | Data Type | Description | |
---|---|---|---|
source | String | Folder name, relative to the configuration file directory, where the ngram database is located. Please contact support to get access to Squirro's pre-compiled language models. | |
default_language | String | Sets the default language for language model lookups. When the ngram folder does not contain a model for the language of the Squirro item that is being processed, then the default language is read. Default: en | |
whitelist | List | A list of company names for which ngram correction is not done. This can be done in some corner cases where a lax match is desired, even though a company name is penalized from the language model. Example:
| |
common | List | A list of prefixes that should be treated as common language terms. This can be used to overwrite the language model to be more strict about certain words. This is sometimes necessary to overwrite imprecise matching for prefix words from other languages. For example "Svensk" is the Swedish word for "Swedish". So any company that starts with "Svensk" may just be saying "Swedish Acme Corp." and thus this shouldn't yet match just based on "Svensk" in any text. The following example snippet takes care of this problem:
|
The settings in the squirro
section can also be set through environment variables. That is especially helpful to avoid writing the token into the config file.
This reference can not go into details on how to set environment variables. Please consult the documentation of your system, such as Bash or Windows PowerShell, for documentation on environment variables.
The environment variables that are respected are:
SQ_CLUSTER
SQ_TOKEN
SQ_PROJECT_ID
The following sections give a few examples for how to achieve common use cases.
Hierarchies are created using the hierarchy
setting on a source. Tagging of hierarchies is achieved using the parent_keywords
setting in the strategy.
The input data here is a CSV file with the following contents (top 3 lines only):
Id,Name,Aliases,ParentId 1,Apple Inc. 2,Google Inc.,Googl|Goog,3 3,Alphabet Inc.,abc.xyz |
The KEE configuration file that makes full use of this data can look as follows:
{ "sources": { "demo": { "source_type": "csv", "source_file": "hierarchy.csv", "strategy": "demo", "multivalue": "Aliases", "field_id": "Id" "field_matching": ["Name", "Aliases"] // The data is hierarchical, with the children declaring their // parent (ParentId field points to a valid Id from another row). "hierarchy": "ParentId->Id", } }, "strategies": { "demo": { // Score at which the hit is a good one "min_score": 0.6, // Depending on Type we assign different keywords "keywords": "Name -> Name", "parent_keywords": "Name -> Parent Name", }, }, } |