KEE Config Reference

Known Entity Extraction (KEE) is configured using a configuration file in the root of the KEE project. That file is called config.json and is written in the JSON format.

Table of Contents

Format

JSON

The format used by the config file is JSON, or rather a more human-friendly superset called Hjson. First, an example of a fully valid JSON file which is also valid Hjson:

{
    "sources": {
        "demo": {
            "source_type": "csv",
            "source_file": "demo.csv",
        }
    },

    "testing": {
        "fixtures_dir": "fixtures/",
        "snapshots_dir": "snapshots/"
    }
}

HJson

With Hjson it is possible to write this file in a more human-friendly way. The main improvements are comments and trailing commas (which are not allowed in standard JSON). Additionally most of the quotes can be left out if so desired.

{
    sources: {
        demo: {
            // The data is in a CSV file in the current directory
            "source_type": "csv",
            "source_file": "demo.csv",
        },
    },

    /* Just repeating the default settings for testing here */
    testing: {
        fixtures_dir: "fixtures/",
        snapshots_dir: "snapshots/",
    }
}

A full reference of Hjson is available on hjson.org.

Structure

In the examples above the curly brackets at the root of the file open a new dictionary. Within each dictionary there are a number of key / value pairs. For example, the keys of the top level dictionary shown above are sources and testing.

Each entry in the top-level dictionary indicates a different section of the KEE configuration. This reference describes the usage of each of the different sections below.

• kee
• squirro
• sources
• testing
• extraction
• strategies
• suffix_list
• ngram

Reference

KEE

The kee section configures a few basic parameters of the whole process. The full section is optional.

example usage:

{
    "kee": {
        "database": "db/product_list.json",
        "debug": false
    }
}

Squirro

To add a KEE project to a Squirro project, some of the cluster information needs to be configured in the squirro section.

See Connecting to Squirro for how to get this information.

This section is optional if you are not using the commands that connect to Squirro (get_fixture or upload).

example usage:

{
    "sources": {
        "cluster": "http://www.mysquirrocluster.com",
        "token": "MY-ACCESS-TOKEN",
        "project_id": "MY-PROJECT-ID" 
    }
}

Sources

The sources section in the configuration contains a list of all the data sources for building the lookup database.

Each entry is in itself an object with the required configuration for the source. A partial example:

{
    "sources": {
        "clients": {
            // The keys  from the reference below come here
        },
        "employees": {
            // The keys  from the reference below come here
        },
        … 
    }
}

Each source takes the configuration. This contains first the actual data connection. For this, the Data Loader options, including Data Loader Plugins can be used. The second part is then the KEE behaviour for that particular source.

{
    "source_type": "csv",
    "source_file": "test.csv",
}

{
    "sources": {
        "companies": {
            // Other keys omitted for clarity
            …
            "multivalue": [
                "Aliases",  // Aliases are multiple values, separate with pipe |
                "Sectors:,",  // Sectors are multiple values, separate with comma ,
            ]
        }
    }
}

Testing

The optional testing section allows specification of where the tests files and snapshots are located. See KEE Testing for details on the testing process.

Again a partial example for this:

{
    "testing": {
        "fixtures_dir": "fixtures/",
        "snapshots_dir": "snapshots/"
    }

    … // Other keys omitted
}

The valid configuration keys in this section are:

Extraction

The section extraction configures how the Squirro items are processed during the KEE process.

Strategies

A strategy in KEE defines how the mapping is executed and which keywords should be added to Squirro items upon a successful match.

The matching is modified depending on a number of factors, including:

• How precise should the matching be? In certain cases false positives (matches that shouldn't have been done) are a smaller problem than false negatives (matches that were missed). In other cases it's the opposite.
• The format of the incoming data also matters. If the input data is all uppercase (as often happens in legacy data) the KEE matching has less precision to work with. If the input data is known to be of high quality, then signals like camel case can be taken into account.
• The data domain. For example company names have many suffixes that are often not spelled out in common language (e.g. Inc., Limited, (Pty) Ltd, etc.).

The strategy is referenced in the input data and based on that name looked up in the strategies configuration. The following incomplete example references a strategy called "companies" that is correspondingly defined in the strategies section.

{
    // Most keys omitted for brevity

    "sources": {
        "clients": {
            "source_type": "csv",
            "source_file": "…",
            "strategy": "companies"
        },
    },

    "strategies": {
        "companies": {
            "tokenizer": "default",
            // …
        },
    },
}

The following table describes all the configuration keys with which a strategy can be tailored to specific requirements.

…
    "keywords": [
        "Name",  // Takes the "Name" column and writes it into a "Name" keyword
        "Id -> Company ID",  // Writes the "Id" column into a "Company ID" keyword
        // Adds the "Id" a second time. If the "Type" of the match is e.g. "SME" then
        // this adds the keyword "SME ID".
        "Id -> {Type} ID",
    ]
…

…
    "clean_keywords": ["Name", "Company ID"],

    "keywords": [
        "Name", "Id -> Company ID",
    ]
…

Suffix list

The suffix list is used by the strategy to ignore common suffixes. Examples for such suffixes:

• Companies: Names for enterprises legally end with Inc., Pty, AG, a.s., …
• Stock ticker symbols: These may often include also the stock exchange, as as NASDAQ, NYSE, etc

These suffixes may often be omitted when writing about said entities and thus the can be ignored for matching.

To create a custom suffix list, add it to the suffix_list section and define the various patterns as key/value pairs. The keys are currently ignored by the KEE extraction and can be used to group the tokens into countries or other logical groupings.

An example:

{
    // Most keys omitted for clarity

    "strategies": {
        "orgs": {
            "suffix_list": "companies",
        }
    },

    "suffix_list": {
        "companies": {
            "GLOBAL": ["Inc", "Limited"]
            "DEU": ["AG", "GmbH"]
            "ZAF": ["(Pty) Ltd", "LIMITED"]
            "INDUSTRY": ["Bank"]
        }
    },
}

Language handling (ngram)

For improved KEE matching, Squirro can make use of a language model. That allows the matching to handle common words in entity names correctly. Two simple examples will show the possibilities:

• The Capital Group: this is a company that contains only quite common words. So if a text talks about "the capital" it shouldn't match this company yet. But when the name is fully spelled out, then the entity should match.
• Carrefour Group: The word "Carrefour" is sufficiently rare in common language usage that just writing "Carrefour" in a text should be enough to match this entity.

The required frequency model is created using the ngram definition. Please contact support to get access to Squirro's pre-compiled language models.

An example configuration for ngram is as follows:

{
    // Most keys omitted for clarity

    "strategies": {
        "orgs": {
            "ngram": "companies",
        }
    },

    "ngram": {
        "companies": {
            "source": "ngram/",
            "whitelist": ["Apple"],
        }
    },
}

The following is a reference of all of the keys in an individual ngram section.

// …
"ngram": {
    "companies": {
        "source": "ngram/",
        "whitelist": ["Apple"],
    }
},

// …
"ngram": {
    "companies": {
        "source": "ngram/",
        "common": ["Svensk"],
    }
},

Environment Variables

The settings in the squirro section can also be set through environment variables. That is especially helpful to avoid writing the token into the config file.

This reference can not go into details on how to set environment variables. Please consult the documentation of your system, such as Bash or Windows PowerShell, for documentation on environment variables.

The environment variables that are respected are:

• SQ_CLUSTER
• SQ_TOKEN
• SQ_PROJECT_ID

Examples

The following sections give a few examples for how to achieve common use cases.

Hierarchy

Hierarchies are created using the hierarchy setting on a source. Tagging of hierarchies is achieved using the parent_keywords setting in the strategy.

The input data here is a CSV file with the following contents (top 3 lines only):

Id,Name,Aliases,ParentId
1,Apple Inc.
2,Google Inc.,Googl|Goog,3
3,Alphabet Inc.,abc.xyz

The KEE configuration file that makes full use of this data can look as follows:

{
    "sources": {
        "demo": {
            "source_type": "csv",
            "source_file": "hierarchy.csv",

            "strategy": "demo",
            "multivalue": "Aliases",
            "field_id": "Id"
            "field_matching": ["Name", "Aliases"]

            // The data is hierarchical, with the children declaring their
            // parent (ParentId field points to a valid Id from another row).
            "hierarchy": "ParentId->Id",
        }
    },

    "strategies": {
        "demo": {
            // Score at which the hit is a good one
            "min_score": 0.6,

            // Depending on Type we assign different keywords
            "keywords": "Name -> Name",
            "parent_keywords": "Name -> Parent Name",
        },
    },
}