Extracting fields from free-text

Abstract

The Field Extraction Library (FEL) provides functions for named-entity extraction within free text. FEL models the content structure of the specified named-entities rather than relying on brittle, context-specific separator logic. Users specify the names of the fields they wish to extract, which determine the number of states for an underlying Hidden Markov Model. The observable emission set is pre-determined by FEL's tokenizer. Once the model topology is set, users provide training examples of the form: x = raw text, y {fieldl: val1, field2:val2, ... } FEL learns the parameters of the underlying Hidden Markov Model by maximum likelihood model-estimation on the training examples. FEL is designed to operate on small, sparse training data. As a result, users can provide few (less than 10) training examples to bootstrap the model. FEL offers 3 iterative mechanisms for scaling data quality as users provide guidance through additional feedback: (1) accept more training examples, (2) create landmark states, and (3) bridge related states with state bridges. FEL detects ambiguities both in its internal model and in the extraction results to prompt users for more feedback. Once the model yields acceptable result quality, users can extract fields into a table for easy querying and exporting.