EXCITEMENT open platform: Architecture and Interfaces

The key words must, must not, required, shall, shall not, should, should not, recommended, may, and optional are to be interpreted as described in [RFC 2119]. Note that for reasons of style, these words are not capitalized in this document.

An option which we initially considered was the CoNLL shared task tabular format (http://ufal.mff.cuni.cz/conll2009-st/task-description.html). We rejected this possibility because it could not meet several of our requirements. (1), it does not specify a pipeline, just an interchange format. (2), it is extensible but at the same time fairly brittle and unwieldy for more complex annotations (cf. the handling of semantic roles in CoNLL 2009 which requires a flexible number of columns). (3), it only specifies an on-disk file format but no in-memory data structure. (4), no support for metadata.

UIMA (Unstructured Information Management Applications) is a framework that started as a common platform for IBM NLP components. It evolved into a well-developed unstructured information processing platform, which is now supported by Apache Foundation as an open source framework. It has been used by many well known NLP projects and has healthy communities in both academic and commercial domain.

UIMA provides a unified way of accessing linguistic analysis components and their analysis results. All analysis modules are standardized into UIMA components. UIMA components shares a predefined way of accessing input, output, configuration data, etc. Within UIMA, it is easy to setting up a composition of analysis components to provide new pipelines, or adopt a newly developed analysis module to already existing pipelines.

On the top level, the unification of UIMA components is achieved on two levels.

The following figure shows the Apache UIMA Java implementation as the LAP of EXCITEMENT. It provides a unified way of calling linguistic analysis components for user code.

Figure 3. UIMA as the linguistic analysis pipeline of EXCITEMENT

UIMA comes with various components and tools. However, in this specification, we will only review the components that an EDA developer should provide to the EXCITEMENT LAP as UIMA components. They are also the components that EXCITEMENT top level users can access from the linguistic analysis pipeline. Figure 3, “UIMA as the linguistic analysis pipeline of EXCITEMENT ” shows them. We first provide generic definition of AE, AAE and collection processing. Then we will state what EDA implementers should provide to realize an EXCITEMENT LAP.

AE. Analysis modules of UIMA are called as Analysis Engines (AEs). For example, individual analysis components like POS tagger, parser, NER annotators are AEs. All AEs share a set of contracted methods, and they will be accessed by the UIMA framework. All analysis results of an AE are returned to the caller as added layers on a CAS.

AAE. UIMA also provides analysis pipelines that are called Aggregated Analysis Engines (AAEs). An AAE is defined by composing multiple AEs. For example, a pipeline that outputs parsing results can be composed by subsequent calls of a sentence splitter AE, a POS tagger AE, and a parser AE. UIMA provides a standard way to compose analysis pipelines without changing anything on the individual analyzers.

Collection Processing (and UIMA Runtime). A linguistic analysis pipeline often needs to work on a collection of documents and produce a set of enriched documents. For example, a user might want to optimize parameters of a TE system with repeated experiments on the same dataset. In this case, repeating the same linguistic preprocessing would be redundant. AAEs do not provide any capability for accessing files or processing collections. It is the responsibility of the AAEs caller to recognize such cases and avoid redundant calls. To guarantee that this happens in a uniform manner, UIMA provides the concepts of UIMA runtime and collection reader.

A collection reader is a UIMA component that is specialized in data reading and CAS generation. Compared to an AE, it shares a different set of contractual methods. A collection reader reads an input (document) and generates a CAS that can be processed by AEs.

A UIMA runtime is a runner mechanism that enables running of AEs. If a user calls an AE directly (via the framework), the component is actually running in the caller's space (same process, thread). In this case, running AEs on multiple files would still be the caller's responsibility. A UIMA runtime is designed to take over that responsibility. A runtime is capable of iterating over a set of documents in its own space. Apache UIMA and the UIMA community provide various runtimes, from simple single thread runtime to asynchronous distributed runtimes. Examples for UIMA runtimes are UIMA CPE/CPM (collection processing engine / collection processing manager), UIMA AS (Asynchronous scale-out), UIMAfit simple pipeline, and the UIMAfit hadoop runtime.

The LAPs of EXCITEMENT exist primarily for processing textual entailment data. As a secondary goal, the LAPs need to provide linguistic analysis capabilities for the user level code and the transduction layer. With these goals in mind, it is possible to formulate more focused requirements for EXCITEMENT LAP components. An EXCITEMENT LAP should provide the following capabilities:

At the time of writing of this specification (V1.0), the EXCITEMENT consortium has made the following decision regarding the adoption of UIMA within the linguistic analysis pipeline:

More formally, the following requirements fall out of these decisions:

This specification provides no further information on UIMA. Please consult external UIMA documentation such as [UIMA-doc] to learn more about the UIMA framework and on how to build UIMA components.

Artifact is a UIMA concept which describes the raw input. For example, a document, a web page, or a RTE test dataset, etc, are all artifacts. For the moment, let's assume that we have a web page as an artifact, and the goal is to provide various analysis on the text of the web page.

The generic analysis tools (like POS tagger, parser, etc) do not know about HTML tags of the webpage. And it is not desirable to add such specific tag support to generic tools. In UIMA, generic tools can be used without modifications for such cases, by adopting Views. For the previous example, an HTML detagger module can generate a plain text version of the webpage. This can be regarded as a new perspective (view) on the original data (artifact). In UIMA this new view can be introduced into a CAS. Once the view is introduced in the CAS, subsequent analysis like POS tagging and parsing can process the plain text view of the web page without knowing that it is from a webpage.

Subject OF Analysis (SOFA) is another UIMA concept. It is referring the data subject that is associated with a given view. In the website example, it is the plain text version of the web page. One view has one SOFA, and one CAS can have multiple views. Views (and Sofas) can have names that can uniquely identify them among a given CAS. Annotations of CAS generally have a span: beginning position and the end positing of the annotated text. Such spans are expressed as offset of each SOFA. Thus, annotators (individual analysis engines) are annotating SOFAs, instead of artifacts directly.

A pictorial example of CAS, views and annotations are given after introduction of the types for textual entailments, in Figure 4, “Example of an entailment problem in CAS”.

The UIMA framework itself defines only a handful of default types. They are including primitive types (like string, numerical values), and basic annotation types.

Primitive types only represent simple values, but user-defined types can express arbitrary data by composing primitive types, and other types. For example, a token type might have a string (primitive type) for lemma, another string for POS, and two numbers for starting position and end position of the token. Within UIMA CAS, this list of values are called as feature structures. Each feature in the feature structure has a name and a type, thus only a specific type can be pointed by the feature.

UIMA type system is based on single-parent inheritance system. Each type must have a one super-type, and it gets all features from the super-type.

The followings are non-exhaustive list of the default types that are used in the adopted and proposed type system of the specification.

Fully detailed information on CAS, and types of default CAS can be found in the CAS reference section of the UIMA reference document [UIMA-CAS].

This section outlines adopted and proposed UIMA types for generic NLP analysis results. The full list of the types are given in the Appendix A, Type Definition: types for general linguistic analysis.

In this section, the string DKPRO is used as a short-form of de.tudarmstadt.ukp.dkpro.core.api.

This section outlines the types proposed for textual entailments and related sub-tasks. Full listing of the mentioned types are listed in Appendix B, Type Definition: types related to TE tasks .

3.3.4.1. Representation of Text and Hypothesis

The basic approach of the platform is to use a single CAS to represent a single entailment problem. The CAS holds text, hypothesis and associated linguistic analysis annotations.

Let's first consider Figure 4, “Example of an entailment problem in CAS” which shows an example of entailment problem represented in a. It includes UIMA CAS elements like artifact, view, and SOFA (cf. Section 3.3.1, “Relevant UIMA concepts: Artifact, View, SOFA”). The original raw input is shown in the figure as the artifact. It is a simple XML annotation with a T-H pair. The artifact contains two plain-text subjects of analysis (SOFA). Analysis tools like taggers will only see this SOFA data, instead of the artifact XML. All annotations (all types that inherits uima.tcas.Annotation) are annotated on top of the SOFA. For example, the first token annotation of the text ("this") is annotated on TextView SOFA position 0 to position 3. This SOFA positional span underlies the most common way of accessing fetching UIMA text annotations, namely to retrieve, e.g., "all annotations of type Token from the TextView SOFA span 0 - 14".

Each view has exactly one SOFA. In this example, each view is enriched by token annotations and POS annotations. Both token and POS are annotation types, and have begin and end fields, which shows the span of the annotation. (POS types also have begin and end span. They are omitted here for clarity.)

Figure 4. Example of an entailment problem in CAS

The figure also shows a set of entailment specific annotation types: they are entailment.Pair, entailment.Text, entailment.Hypothesis, and entailment.EntailmentMetadata. The example has a single entailment.Pair, which points entailment.Text and entailment.Hypothesis. The Pair has an ID, and it also has a goldAnswer. The CAS has no additional pair, and the entailment problem described in this example is a single T-H pair. There is one entailment.EntailmentMetadata in this CAS. It shows that the language of the two views is "EN" (English as defined in [ISO 639-1]), and the task is "QA". It also has a string for channel value as "email".

The following subsections define the annotations in detail.

3.3.4.1.1. Type entailment.EntailmentMetadata

EXCITEMENT.entailment.EntailmentMetadata provides metadata for each entailment problem. It is an extension of uima.cas.TOP. An instance of this type should be indexed directly on the CAS (not on TextView or HypothesisView).

• language (uima.cas.String): this string represents the language of the CAS and its entailment problem. It holds two character language identifier, as defined in [ISO 639-1]).

• task (uima.cas.String): this string holds task description which can be observed in the RTE challenge data.

• channel (uima.cas.String): this metadata field can holds a string that shows the channel where this problem was originated. For example, "customer e-mail", "online forum", or "automatic transcription", "manual transcription", etc.

• origin (uima.cas.String): this metadata field can hold a string that shows the origin of this text and hypothesis. A company name, or a product name.

• TextDocumentID (uima.cas.String): This field can hold a string that identifies the document of the TextView. This feature must have a value, if TextCollectionID is not null.

• TextCollectionID (uima.cas.String): This field can hold a string that identifies the collection name where the document of the TextView belongs to.

• HypothesisDocumentID (uima.cas.String): This field can hold a string that identifies the document of the HypothesisView. This feature must have a value, if HypothesisCollectionID is not null.

• HypothesisCollectionID (uima.cas.String): This field can hold a string that identifies the collection name where the document of the HypothesisView belongs to.

The values of these fields are unrestricted Strings. Pipeline users are responsible for documenting the semantics of the values if they want to ensure interoperability.

3.3.4.1.2. Type entailment.Pair

Text-Hypothesis pairs are represented by the type EXCITEMENT.entailment.Pair. The type is a subtype of uima.cas.TOP. An instance of Pair should be indexed directly on the CAS (not on TextView or HypothesisView). If the CAS represents multiple text and/or hypothesis, the CAS will holds multiple Pair instances.

The type has following features:

• pairID (uima.cas.String): ID of this pair. The main purpose of this value is to distinguish a certain pair among multiple pairs.

• text (EXCITEMENT.entailment.Text): this features points the text part of this pair.

• hypothesis (EXCITEMENT.entailment.Hypothesis): this features points the hypothesis part of this pair.

• goldAnswer (EXCITEMENT.entailment.Decision): this features records the gold standard answer for this pair. If the pair (and CAS) represents a training data, this value will be filled in with the gold standard answer. If it is a null value, the pair represents a entailment problem that is yet to be answered.

3.3.4.1.3. entailment.Text, entailment.Hypothesis and entailment.Decision

EXCITEMENT.entailment.Text is an annotation (extending uima.tcas.Annotation) that annotates a text item within the TextView. It can occur multiple times (for multi-text problems). It does not have any features. A Text instance is always referred to by a Pair instance.

Note that Text annotations and TextView are two different concepts. A TextView is a container that contains a SOFA plus all of its annotations, including entailment.Text Thus, the fact that a sentence forms part of a TextView does not automatically mean that it is compared to a hypothesis. This only happens if it is annotated with entailment.Text and linked with entailment.Pair.

The situation is similar for EXCITEMENT.entailment.Hypothesis. It is also an annotation (uima.tcas.Annotation) that annotates a hypothesis item within the HypothesisView. It can occur multiple times (in multiple-hypothesis cases). It does not have any features either. A Hypothesis instance is always referred to by a Pair instance. Hypothesis annotations and HypothesisView are different entities, and sentences are only compared to a specific text annotation when they are annotated with entailment.Hypothesis, like for texts.

The EDA process interface receives CAS data as a JCas object (the UIMA-created Java type hierarchy corresponding to the UIMA CAS types). CASes that feed into the EDA process interface must have at least a single entailment.Pair. Naturally, each view (TextView and HypothesisView) must have at least one entailment.Text and entailment.Hypothesis. Multiple text and multiple hypothesis cases will be represented by multiple number of entailment.Pair. Note that the relationship between pairs on one side and texts and hypotheses is not a one-to-one relationship. Several pairs can point the same hypothesis, or to the same text. For example, if one hypothesis is paired with several potential texts, there will be multiple pairs. All pairs will point to the same hypothesis, but to different texts.

EXCITEMENT.entailment.Decision represents the entailment decision. It is a string subtype, which is a string (uima.tcas.String) that is only permitted with predefined values. entailment.Decision type can only haves one of "ENTAILMENT", "NONENTAILMENT", "PARAPHRASE", "CONTRADICTION", and "UNKNOWN" (The type can be further expanded in the future). This UIMA type has a corresponding Java enum type within the entailment core (see Section 4.2.1.6, “enum DecisionLabel”).

3.3.4.2. Generalization to multiple pairs

Figure 5, “Example of an entailment problem with multiple pairs” shows another example of a CAS with entailment types. In this figure, the CAS has multiple pairs. The TextView holds a document with six sentences. All six sentences are annotated by sentence splitter, and additional analyzers (for clarity, only sentence annotations are shown). Among the analyzed sentences, the entailment.Text annotations are marked on the first, third and sixth sentences. The HypothesisView holds only a single sentence, which is the single Hypothesis annotation. The CAS holds three pairs, and it is a multiple text entailment decision problem. In the figure, linguistic analysis annotations other than sentences are omitted for clarity. The sentences that are not annotated with a Text annotation can be regarded as context for the annotated texts. For example, co-reference resolutions will take benefits from the sentences. Also, components like distance calculation components, or text rewriting components can use the tokens in the context sentences to get better results. (Note that generic components like tokenizer, parser, and co-reference resolver, do not aware of entailment specific annotations, and process all data in the given view. "Generic" verses "TE specific" LAP component is described in the next subsection Section 3.3.4.3, “Two groups of LAP components”.)

Figure 5. Example of an entailment problem with multiple pairs

Note that EDAs will only decide entailment of pairs that are explicitly marked with Pair annotations. Existence of text or hypothesis annotation does not automatically lead to processing in an EDA. For example, if the pair-2 is missing from the figure, the Text annotated on sentence-3 will not be compared to the hypothesis sentence. Also note that nothing stops us from annotating a text item that overlaps with other text items. For example, it would be a legal annotation if we add a new text item that covers from sentence-2 to sentence-4 (which will include the text of pair-1).

3.3.4.4. Other cases: T-H from multiple documents, T-H on a same document

Ordinary annotators (taggers, parsers, etc) do not (and should not) know about the views of TextView and HypothesisView. They only see one view at a time, its SOFA as a single document. Thus the two views of T and H naturally represent two separate documents, one for text and one for hypothesis.

One problem of this setup is that we have to process other cases like:

• Text (or Hypothesis) from Multiple documents.

• Text and Hypothesis on the same document.

The following subsections describe how we will represent such cases with the proposed types and CAS.

3.3.4.4.1. Multiple document problem: decomposition to multiple CASes

Some problems involve multiple documents. Examples are the main task of RTE6 and RTE7. They have TE problem that consists of multiple texts (T) and a single hypothesis. The texts are scattered over a set of documents (collection). Additionally, there are global features associated with the document collection (e.g., topic).

We can break down this case with multiple CASes with the proposed CAS representation. The following figure shows one artificial example. In the figure, it has a single hypothesis and multiple texts. The texts are located in three different documents. It is possible to represent this problem with three CASes.

Figure 6. Multiple document TE problem decomposed into multiple CASes.

Each CAS of the decomposed one will look like Figure 5, “Example of an entailment problem with multiple pairs”. Note that the multi document example is decomposed into several CASes whose number corresponds to the number of documents. Each view holds a single document in this setup, and normal annotators can process each view just as a normal document.

EXCITEMENT.entailment.EntailmentMetadata holds two additional features for multiple document cases. It has documentID and collectionID for both text and hypothesis. Document ID identifies the document assigned in the view, and collection ID can show the collection where the document is belong to.

Note that this decomposition assumes that breaking down the multiple document case into multiple single document problem does not change the final output. See Section 3.3.4.4.3, “More complex problems and their use-cases ” for more complex cases of multiple text/hypothesis.

3.3.4.4.2. T and H on same document : Two views hold the same document

Since we have two separate views for text and hypothesis, it can be problematic when we have the text and the hypothesis from the same document.

In such a case, we will have two views with the same document (two views have the same document with same analysis result). All other TE related annotations stays normal, like metadata and entailment problem annotation. Only document specific annotations (like tagging, parsing) are duplicated into two views.

Note that this "duplicated analysis result in two views" does not mean the analysis of the document has to be done twice. We can setup a special pipeline for single document T-H case, where it first runs analysis and copies the result into two views.

3.3.4.4.3. More complex problems and their use-cases

Note that the above two mappings (multiple documents THs and single TH) simplify EDA interfaces. For EDAs, all inputs look alike with two views with entailment.Pair type annotations. However, we can imagine more complex use cases where no trivial mapping is possible. Within the WP3 discussions, the following tasks have been discussed for possible future support.

• Text expansion: Automatically generating a (large) set of additional texts that can entail the given input text.

• Dialog system: Ranking hypothesis for the given text. The goal is to rank, instead of decision.

• Hypothesis with variables: Hypothesis that includes variables, where the variables can be replaced by parts (like words) that appears in the text.

For such cases, we will probably additional interfaces on EDA side, also with new problem descriptions. However, this version of specification only deals with existing, well-known RTE problem sets. The WP3 members have agreed that additional use-cases will only be discussed in a future version of the specification.

It is possible that users want to add additional information to entailment problems. For example, task-oriented TE modules might want to include additional information such as ranks among texts, relevance of each hypothesis to some topics, etc.

The canonical way to represent such information is to extend the entailment.Pair and related types. The type and related types (like MetaData, Text and Hypothesis, etc) are presented to serve the basic need of the EXCITEMENT platform, and additional data can be embedded into CAS structure by extending the basic types.

Naturally, the extension should be performed in a consistent manner. This means that the implementer can only define a new type that is extending the existing types, and may not change already existing types. Also, it is recommended that attributes and methods inherited from the existing types should be used in the same way so that components and EDAs which are unaware of the extensions can still can operate on the data.

The type EXCITEMENT.entailment.Decision is also open for future extensions. When we define a new relation between text and hypothesis, this type should be first extended to cover the new relation. (This extension should always done with the extension of internal DecisionLabel enumeration, see Section 4.2.1.6, “enum DecisionLabel”.

Unlike EXCITEMENT types, generic types (of Section 3.3.3, “Types for generic NLP analysis results”) should in general not be extended. Exceptional cases may arise, though, for example when an EDA utilizes some additional information of a specific linguistic analysis tool. In such a case, one may choose to extend generic types to define special types (for example, defining myNN by extending NN, etc). The EDA can then confirm the data is processed by a specific tool (like existence of myNN) and can use the additional data that is only available in the extended type. Other modules that do not recognize the extended types are still able to use the super-types in the output (in the example, myNN will map onto NN).

LAP (Linguistic Analysis Pipelines) is an important part of the platform that provides all linguistic analysis capabilities for the EXCITEMENT platform and its users. It provides not only normal NLP annotation capabilities like tagging and parsing, but also the capability to generate well-formed CASes for EXCITEMENT entailment decision algorithms (EDAs).

The purpose of this set of LAP interfaces is twofold. First, it defines the common minimal capability that each pipeline should provide, in terms of interface definition. Each EDA implementer must provide a compatible pipeline for the EDA. For the developer who will provide a new pipeline, these interfaces can serve as an implementation guideline.

The second purpose of the interface is to make the underlying pipeline mechanism transparent for the end user. As stated in the previous sections (Section 3.2.3, “Adoption of UIMA in EXCITEMENT”), the EXCITEMENT consortium already agreed on using UIMA CAS. However, we have not yet come to a decision about the adoption of UIMA runtime mechanism (like AE, AAE or CPE, etc). In the first iteration of the platform, it is expected that some of the pipelines are composed by a set of UIMA AEs, while other pipelines will only "translate" their pipeline output to the UIMA CAS.

In this situation, it is important to define a coherent set of interfaces that is shared by all pipelines, regardless to the underlying mechanism. The users should be able to call all different pipelines with the same argument and signatures. This is more important for users who just want to use the platform only as "off-the-shelf TE engine". For example, knowledge of CAS or UIMA types should not exposed to such users. Those users should be able to use LAP pipelines for their EDA calls without knowing what CAS has within it.

The following sections define the three set of methods that will be shared by all pipelines.

interface LAPAccess extends Components

This interface defines the minimal set of methods that a working LAP pipeline should provide for EXCITEMENT platform. It defines three types of methods. One for generic annotation, one for single pair online-generation, and one for collection processing of single-pair input files.

To determine the set of reusable components and their interfaces, we started from the three systems that we primarily build on and followed a bottom-up generalization approach to identify shared functions (components), interfaces and types. The results are discussed in Section 4.1.3, “Identifying common components and interfaces ”. Further functionality of these components (e.g. details of their training regimens) are not be specified but in the scope of the component's developer

By following the bottom-up approach, we have identified a set of major common interfaces. The following figure shows the common components within the entailment core.

Figure 7. Common components within EXCITEMENT entailment core.

The top-level interfaces of the entailment core consist of the EDA interface and the mode helper interface. EDAs are the access points for entailment decision engines. They receive annotated input in the form of CAS, and return a decision object. The EDA interface also includes support for multiple Text and/or multiple Hypothesis modes. The EDA interfaces are described in Section 4.2, “EDA Interface ”. The mode helper is a wrapper tool that supports multiple text and multiple hypothesis mode for EDAs that can only process single T-H pairs. Its interface is described in Section 4.3, “Auxiliary Layer for EDAs”.

We have identified three main classes of components that EDAs might wish to consult. The first type is distance calculation components. EDAs can use the semantic distance between T-H pairs as their primary decision factor, or can use them as features in more general decision algorithms. The interface of distance calculation components specifies that they accept a CAS as the input, and return an object the distance between the two views of the CAS. The interface is described in Section 4.6, “Interface of distance calculation components”.

The second and third component classes deal with different kinds of linguistic knowledge, namely lexical knowledge and syntactic knowledge. Lexical knowledge components describe relationships between two words. Various lexical resources are generalized into EXCITEMENT common lexical knowledge interface. The interface is described in Section 4.7, “Interface of lexical knowledge components ”. Syntactic knowledge components contain describes entailment rules between two (typically lexicalized) syntactic structures, like partial parse trees. For entailments, the knowledge can be generalized into rewriting rules at partial parse trees, where left hand sides are entailing the right hand sides. The common interface for syntactic knowledge is described in Section 4.8, “Interface of syntactic knowledge components ”.

The EXCITEMENT platform uses UIMA CAS type system for the linguistic analysis pipeline (Section 3). The UIMA types are expressive enough to describe the relevant data structures, and can be used both in-memory and serialized. It is thus a natural question whether CAS types can be re-used for the communication between EDAs and components.

Our general guideline within the Entailment Core is that the interfaces are supposed to use UIMA CAS only when a complete text (Text or Hypothesis, potentially with context) is exchanged as a parameter. The reason is that even though CAS types (JCas objects) are expressive enough to describe any data, they are basically flat structures --- tables with references. This limits their capability to be used efficiently. A second reason is that JCas objects only works within CAS context. Even simple JCas objects like tokens need a external CAS to be initialized and used. This makes it impractical to use JCas objects and their types in interfaces that do not refer to complete texts (which are modeled in CASes). To repeat, we use JCas and related types only when a reference to the whole text is exchanged. Furthermore, we only exchange references to the top-level JCas object and never to any embedded types like views or annotation layers.

A second aspect that must be handled with care is the extension of JCas objects. JCas provides the flexibility to extend objects while keeping the resulting objects consistent with the original JCas types. However, this can cause duplicated definitions when mismatches are introduced. Moreover, CAS types will be used and shared among all EXCITEMENT developers which makes extensions introduced by one site problematic. For these reasons, our policy is that JCas objects should not be extended.

This interface defines the basic capability of EDAs that all EDAs must support. It has four methods. process is main access method of textual entailment decision. It uses JCas as input type, and TEDecision as output type.

public interface EDAMultiT<T extends TEDecision>

public interface EDAMultiH<T extends TEDecision>

public interface EDAMultiTH<T extends TEDecision>

Each of the interface defines a method, namely processMultiT, processMultiH, and processMultiTH. EDAs may choose to support them or not, since supporting multiple T/H interfaces are optional. Multiple Texts and Hypotheses are marked in the CAS by multiple EXCITEMENT.entailment.Pair annotations (compare Figure 5, “Example of an entailment problem with multiple pairs”).

public abstract class InitializationHelper<T extends TEDecision>

The main access point to the entailment core is the EDA interfaces. Users of the platform will process the data with LAP, and will use EDA interfaces to get the entailment results. To use an EDA, an instance of the EDA object must be first instantiated, and then it must be initialized with a proper configuration. Then, the EDA will initialize all needed components accordingly (see also Section Section 5.1.6, “Component selection”.

Thus, to start an EDA's initialization sequence, one must first instantiate an instance, and then feed it a configuration object. This also assumes that the configuration file is already read, and the main EDA that is defined in the configuration is already determined. InitializationHelper is an auxiliary layer that takes care of this instantiation and initialization. Without the helper, all of those process would have to be done by user code, which would require different initialization sequences for different EDAs.

InitializationHelper solves this problem by providing a common initialization capability. The helper code accepts a configuration file, and then it reads the configuration file to determine the main EDA class. Then, the helper instantiate and initialize the EDA, and returns a ready-to-use EDA instance.

public abstract class MultipleTHModeHelper<T extends TEDecision>

The "mode helper" is a wrapper implementation that uses a single-mode EDA (EDABasic) to support multiple T-H cases (that of EDAMultiT, EDAMultiH, and EDAMultiTH). It receives an already initialized EDA and iteratively calls the EDA to produce results for multiple T/H cases. It is a baseline implementation that can be used with any EDA.

It is defined as an abstract Java object. It is expected that the platform will share one implementation of mode helper for all EDAs.

public interface SinglePairProcessHelper<T extends TEDecision>

This interface is a convenience interface for "end users" of an entailment engine. The implementation of this interface uses a specific implementation of EDA (EDABasic), and provides a method that allows the user to run the complete engine including LAP and Entailment Core with a single call. The access method receives two String objects as input (the text and the hypothesis, respectively), and returns the result (an object that extends TEDecision) for them.

Since this is a convenience interface, it is limited with regard to functionality and efficiency. It does not support context sentences, or multiple text/hypothesis situations. Also, calling the LAP for just one sentence pair is presumably quite inefficient.

Note that, unlike MultipleTHModeHelper, the platform cannot share a single ProcessHelper implementation for all EDAs. Each EDA needs to run a different LAP, thus each EDA is expected to have its own ProcessHelper.

public String getComponentName()

This method provides the (human-readable) name of the component. It is used to identify the relevant section in the common configuration for the current component. See Sections Section 5.1.2, “Overview of the common configuration ” and Section 4.10.3, “Component name and instance name”.

public String getInstanceName()

This method provides the (human-readable) name of the instance. It is used to identify the relevant subsection in the common configuration for the current component. See Sections Section 5.1.2, “Overview of the common configuration ” and Section 4.10.3, “Component name and instance name”. Note that this method can return null value, if and only if all instances of the component shares the same configuration.

Previously, the method void initialize(CommonConfig c) was included in the interface Component. This means that all components are supposed to be initialized by this method, with a configuration object. Since specification verion 1.1.2, the method is removed from the interface. This changes the expected behavior of a component for its initialization.

All component implementations should satisfy the following conditions.

Note that the overridden constructor that knows how to deal a the configuration object can have forms other than the single argument of CommonConfig. For example, components that permits multiple instances might need additional arguments. It might look like GermaNet(CommonConfig c, String instanceName), where the instanceName will identify proper configuration subsection for that instance, etc.

Also note that EDA still keeps its initialize() method, in the EDABasic interface. This will force all EDA implementations to have this method. This reflects that EDA will always have a dependency to the configurations, and the method will be used by top-level initialization helper and other user-layer codes.

The specification does not define how and where component and instance names is stored internally within the component implementations. It is expected that the implementation efforts will implement a common practice of keeping instance names.

Like all other components, the interface extends interface Component. The interface adds a single method calculateScores(), which gets a JCas as the input argument, and returns a Vector<Double>.

The interface is a subinterface of ScoringComponent. The new methods it adds is calculation(), which gets a JCas and returns an object of DistanceValue.

This type holds the distance calculation result. It has some member variables and public access functions for the variable. The type is used to exchange data between the component and the EDA. Its variables are set during initialization.

public final class LexicalRule<I extends RuleInfo>

This type represents a generalization of lexical relationships between two words, which are provided by resources like WordNet, VerbOcean and distributional resources. It has two parts: a left hand side (LHS) and a right hand side (RHS). The basic arrangement of lexical resource is that the left hand side has a relationship to the right hand side. Normally, this relationship is entailment (LHS entailment RHS; return values of Section 4.7.2, “Interface LexicalResource”). However, they can also represent resource specific or canonical relations (see Section 4.7.3, “interface LexicalResourceWithRelation ” for querying such relations). A lexical rule also provides additional pieces of information include confidence, relation name, and resource name. Finally, lexical rules are parametrized by a type I (short for information) which allows the type to hold additional resource-specific properties.

Before going into details of the class variables, let us first see a few examples. The following figure shows a pictorial example of the data type. It shows two examples of lexical relations. Each relation is represented with a few data features.

Figure 8. Examples of lexical rule instance

On the left, it shows an instance of LexicalRule that represents the relationship between "pipe" and "tube". The value originalRelation holds the relation (here the "hypernym") from left ("pipe") to the right ("tube"), as recorded in the knowledge resource (here, WordNet). In this example, the left hand side entails the right hand side. The confidence is not provided by the implementation, and it is given as DEFAULT. Additional information is provided in the example with info variable. The type variable I is instantiated by a type "WordNetInfo" which holds information about synsets.

On the right, another example shows that "Magritte" entails "painter". In this case, the knowledge is captured from Wikipedia, and the resource dependent information provides various Wikipedia-based links and scores. Here, the implementation also provides a confidence value.

All lexical knowledge of EXCITEMENT resource will be delivered as an instance of this object.

public interface LexicalResource<I extends RuleInfo>

A lexical resource is a collection of lexical rules of a certain type (like WordNet, or VerbOcean) which can be queried. The interface provides three query methods. Queries are specified by lemma and POS pairs, and the results are returned as a list of LexicalRule objects. LexicalResource is a subinterface of Component and adds the following methods:

The implementation of getRules methods must return an empty list (not null) if no applicable rules are found.

Note that all lexical rules returned by this interface represent entailments (LHS->RHS entailments). In this sense, the methods only support querying of "entailments". If your lexical resource supports additional relational queries, you can provide them with the additional interface Section 4.7.3, “interface LexicalResourceWithRelation ”.

public interface LexicalResourceWithRelation<I extends RuleInfo, R extends RelationSpecifier> extends LexicalResource<I>

The interface LexicalResource (Section 4.7.2, “Interface LexicalResource”) defines the common method of querying and getting lexical entailment rules of the given words. This abstraction is useful in the sense that it abstracts various underlying lexical resources with the entailment relation. This enables the caller of LexicalResource implementations to query various knowledge resources in the same fashion.

However, this comes with a price: the LexicalResource interface cannot provide querying capability of relations others than entailment. For example, user of that interface cannot query NonEntailment words of the given term (terms that you are sure that is not entailment), nor resource-specific relations (like synonym or hypernym of WordNet, or Stronger-than or happens-before of VerbOcean, etc). These resource-specific relations are reported back as part of the query result (RuleInfo within LexicalRule), but cannot be directly asked for.

LexicalResourceWithRelation is defined to recover this capability. It permits lexical resource implementers to define additional query methods with relations. It enables not only the canonical relation queries but also queries with resource-specific relations.The interface extends LexicalResource, and adds three methods. Essentially, each of the method has one additional argument that represents the relation to be queried.

public interface LexicalResourceWithRelation<I extends RuleInfo, R extends RelationSpecifier> extends LexicalResource<I>

Each of the method gets one additional parameter R relation that specifies the relation to be fetched. For example, if relation is "NonEntailment", the resource will return rules that specifies "NonEntailment" (where LHS->RHS is confidently cannot be entailment). If the relation holds "Synonym", it means the query should return Lexical rules where LHS to RHS is synonymy.

Note that the relation is represented by a generic parameter R. Each R is an extension of RelationSpecifier, where the R can be tailored for each resource. We adopt a simple hierarchy for this R. The interface RelationSpecifier and related objects are described in the following subsections.

The following figure shows the simple hierarchy of RelationSpecifier. The interface RelationSpecifier stays in the top, and two interfaces (CanonicalRelationSpecifier and OwnRelationSpecifier<E>) extends this top interface.

                   RelationSpecifier 
                    /             \
                   /               \
CanonicalRelationSpecifier     OwnRelationSpecifier<E> 

The intention of adopting the hierarchy on relation specifier is to explicitly mark the class of relations that are supported by each LexicalResourceWithRelation. For example, all lexical resources that support canonical relation bears the same signature of R (as <R extends CanonicalRelation>). This is the same for resources that support resource-specific relations (R as <R extends OwnRelationSpecifier<E>>).

Let's check each of the interfaces.

public class GermaNetRelation implements CanonicalRelationSpecifier, 
                         OwnRelationSpecifier<WordNetRelation> 
{

   public GermaNetRelation(TERuleRelation canonicalRelation, 
                           WordNetRelation ownRelation)
   {
      this.canonicalRel = canonicalRelation; 
      this.ownRelation = ownRelation; 
   }

   public GermaNetRelation(TERuleRelation canonicalRelation) {
      this(canonicalRelation, null);
   }

   public GermaNetRelation(WordNetRelation ownRelation) {
      this(null, ownRelation); 
   }

   @Override // from canonical relation specifier 
   public TERuleRelation getCanonicalRelation() {
      return canonicalRel;  
   }

   @Override // from own relation specifier 
   public WordNetRelation getOwnRelation() {
      return ownRelation; 
   }

   private final TERuleRelation canonicalRel; 
   private final WordNetRelation ownRelation; 
}

public class GermaNetImplementation implements 
   LexicalResourceWithRelation<GermaNetInfo, GermaNetRelation> { ...

GermaNetImplementation germaNet = new GermaNetImplementation(); 

String lemma = "beautiful";  
PartOfSpeech pos = new GermanPOS("adjective"); 

// query for synonym of "Beautiful"
GermaNetRelation rel1 = new GermaNetRelation(WordNetRelation.SYNONYM); 
germaNet.getRulesForLeft(lemma, pos, rel1); 

// query for NONENTAILMENT terms to "Beautiful" 
GermaNetRelation rel2 = new GermaNetRelation(TERuleRelation.NONENTAILMENT);
germaNet.getRulesForLeft(lemma, pos, rel2); 

String lemma = "apple";  
PartOfSpeech pos = new GermanPOS("noun"); 

// query for ENTAILMENT and hypernym of "apple"
GermaNetRelation rel3 = new GermaNetRelation(TERuleRelation.ENTAILMENT,
                                             WordNetRelation.HYPERNYM);
germaNet.getRulesForLeft(lemma, pos, rel3); 

public class SyntacticRule

The class represents an instance of a syntactic rule.

4.8.1.3. Basic Node: Common representation for dependency parse tree

To share a common representation for syntactic knowledge, a shared syntactic representation is required. The class BasicNode provides the common syntactic representation of EXCITEMENT. It represents a dependency parse tree node, linked to other parse tree nodes. By acquiring the top node, it can also represent a parse tree, or a partial parse tree.

The term "basic" hear means that it contains only the basic information - information that is common to almost all languages. It does not contain information that appear only on some languages like grammatical role, morphological form, etc. It is expected that specific language nodes will be implemented by extending the BasicNode.

Figure 11, “Contents of a basic node” gives a graphical overview of the basic node.

Figure 11. Contents of a basic node

A node basically holds two sets of member variables. One is about tree itself (parent-children relation), and the other is information on edges and nodes.

• children: this member variable holds an array of child nodes. They are nodes of the same type. Note that the node does not explicitly keep its parent.

• antecedent: this member variable holds a reference to "duplicated node" in the dependency tree. For example, in a sentence like "He managed to go", it is possible to generate a duplicated node "He". The antecedent field of the duplicated one will point the original node.

• info: this member variable holds various information of its edge (incoming edge from the parent), and the node content.

The member info consists of two parts: edgeInfo holds information on edge, and nodeInfo has node content. edgeInfo holds the following information.

• relation: this is an instance of data object that represent the dependency relation. The instance holds two variable. One is itsStringRepresentation that holds the relation output of the dependency parser. And the other is type which holds the an enum value that represents dependency relationship. The enum DependencyRelationType closely reflects the adopted UIMA dependency types (the subtypes of type.dependency.Dependency).

The nodeInfo holds the following information:

• word: this string holds the word (token) itself.

• lemma: the lemma of this token

• namedEntity: it holds a enum value that represents a NER class. If this node is not a named entity, this value is null. The enum type NamedEntity closely reflects the adopted UIMA NER types (subtypes of ner.type.NamedEntity).

• syntacticInfo: this variable holds information about POS tag of this token. It holds two variables: the string output of the POS tagger (posTagString) and the normalized POS tag (cannonicalPosTag) that is equivalent to POS types of UIMA CAS.

Note that (not shown in the figure), nodeInfo also provides two methods isVariable() and getVariableID(). isVariable() method returns true if and only if the node represents a variable of a syntactic rule. If the node is a variable, getVariableID() returns ID of the variable as an integer. (The ID is only for human readers and rule visualizations -- actual rule matches and applications are done by bidirectional mappings provided in the rule.)

The BasicNode is implemented with two generic parameters. One is the info type (that holds nodeinfo and edgeinfo), and the other is node type itself (children and antecedent). Generic parameters are omitted in the text paragraph for readability, but they are described in the actual class code.

A syntactic resource is a collection of syntactic rules. For syntactic rule collections, it is not practical to provide simple access interfaces like getLeftForPOS for lexical knowledge. Due to the exponential number of subtrees in any text and hypothesis, naive querying based on the input is infeasible. Instead, findMatches() method is defined to outline common behavior of platform syntactic rules.

Modern computer hardware is very well prepared for concurrent and parallel processing. Even modest PCs now have multiple cores with multithreading capability. Given users' requirements of quick or even real-time processing, it is important that the EXCITEMENT platform supports concurrent processing.

The EXCITEMENT architecture (as seen in Figure 7, “Common components within EXCITEMENT entailment core.”) can incorporate concurrent processing at two different levels. We will call them concurrent component execution, and concurrent EDA execution. They have different advantages and disadvantages.

Concurrent component execution. Imagine an EDA that requires the results of three sub-components for its entailment decision (a typical situation for TIE or EDITS). Without concurrent processing, the EDA must call three sub-components sequentially and has to postpone its decision to when all three results are available. Concurrent component execution can reduce the total running time by reducing the time needed to produce a single TE decision.

Concurrent EDA execution. Imagine an EDA which must process a set of textual entailment problems. If the time needed to process the set is n, we can reduce the required time to n/m by running m threads for the EDA. This level of concurrency will reduce the time to produce a set of TE decisions by concurrent processing of the selected EDA.

We have decided on the following contract for concurrent processing.

The goal of this interface is to provide an access method for EDAs (and/or EDA wrappers) that can process a set of entailment decision problems concurrently.

List<TEDecision> processDataSetConcurrently (List<JCas> casList)

The interface receives a set of TE decision problems as a list of JCAS. Each CAS object holds an entailment problem (as defined in Section 3.3.4, “Additional Types for Textual Entailment”). The result is returned as a list of TEDecision objects. The interface is not an asynchronous interface: it will block until it process all the given problem list, and return the results only after all processing is done.

The expected behavior of implementation of this interface is to process the given dataset concurrently with a number of threads, using general Java concurrent capabilities like Executors and thread pools. The number of threads and other configurations should be defined in the common configuration. Proper initialization (for example, initializing multiple instances for non thread-safe components) should be ensured by the implementation so that the user does not need to care about sub-components and how they are being run.

This interface can be directly implemented by an EDA (i.e., a class that already implements EDABasic and/or other interfaces). It can be also implemented by a concurrent running wrapper (a "runner") for the EDA, if the implementation of this interface does not lend itself naturally to the internal structure of the EDA.

The WP3 members have agreed that we will approach this issue with incremental steps. We can imagine a set of future improvements including:

Additional concurrent processing supports will be realized as the platform matures.

Recall that a central goal of the EXCITEMENT platform is to provide a testing ground for various pluggable components that decide textual entailment. This is done by providing a set of common interfaces, and common data representations. Another requirement is to provide multilingual support.

From this perspective, compatibility check becomes a necessity, since components do not know in what context they are being called. Each component should provide a minimum level of compatibility check, on its configurations as well as on its inputs. This subsection describes recommended policy of metadata checking for the EDA, the core components, and between the EDA and the core components.

Compatibility checks can be classified into two groups. One is checks at startup time (configuration setting and initialization time). The other is checks at data processing time. The general guideline is that EDAs should do both types of checks, and components should at least do the startup time check.

An EDA must perform a compatibility check at its initialization time. This is a two-step process. First, it should check the provided configuration and check that the provided configuration is compatible with the EDA. For example, the language to be processed is supported, and parameters of the EDA is valid, etc. Second, the EDA must initialize its direct components (components that will be directly called by the EDA). Each component, in turn, will do their own checks, using the information from the common configuration file (or equivalent arguments). Likewise, if a component uses a sub-component, it will also initialize the sub-component.

Thus, two types of metadata check failure can happen at the EDA initialization time. First, the EDA itself can report an exception, like inability to process a certain language, inability to provide a mode (like multi-text entailment), missing parameters, etc. The sub-components of the EDA can report exceptions as well. In this case, the EDA must hand through the exception to the user code. Thus, the successful completion of the EDA initialization must be interpretable to mean that all sub components are successfully initialized and are ready to process input.

An EDA must also perform a compatibility check at the data processing time, when its process method is called. The input to EDA process() is a CAS (as JCas object). EDA must check the validity of the input. At least two things should be checked, namely its language (meta data at document annotation type of CAS), and the existence of the analysis layers required by the components (like POS tagging, or dependency parsing, etc).

Components do not need to do additional checks at the process time. Once a component is properly configured and initialized with a configuration, it is the EDA's responsibility to use the component properly within its capabilities. However, a component may choose to provide meta data checking at processing time.

Some components may need the ability to reconfigure. For example, a user of a distance calculation component may wants to compute 10 distance values, by using 10 different "weighting schemes" using the same underlying distance calculation component. One way of doing this would be to generate ten instances of the same component. However, this might not be desirable (e.g. if the component does not permit more than a single instance, or if the initialization of the component takes a lot of time).

The interface Reconfigurable provides a way to modify the component configuration at runtime. The term "reconfiguration" here means changing some of the configurable parameters of a component, without re-initializing all resources, or changing other settings that may remain constant.

Any entailment core component that supports reconfiguration should implement this capability though this interface. This also includes EDAs. If EDA needs to support online reconfiguration, it should support the capability with this interface. Note that when an EDA provides this interface, the EDA should also update the states of its sub-components according to the configuration method argument. It is the responsibility of EDA implementers to ensure that configuration changes are handed through to sub-components, if applicable. This may mean simply reconfiguring some sub-components (which support this) and re-initializing others (which do not).

The central configuration scheme assumes that each component knows its own name, and it can retrieve corresponding configuration data from common configuration by querying the configuration with its name.

Component name is a String value that is designed to be read and written by users. This name must be able to uniquely identify a component or EDA within the EXCITEMENT framework. This name will be used in the configuration files to denote a section that describes the configuration for the component. All entailment core components and EDAs must have this component name string. Conversely, for each component that is initialized there must be a corresponding subsection in the configuration file. The components share a common way of access the names (String getComponentName). To see the use case of names within configuration interface, see Section 5.1, “Common Configuration ”.

Instance name is a String value that is needed for components that permits multiple instances with different configurations. This name will be used in the configuration files to denote a subsection that describes the configuration for the instance. Conversely, for each instance that is initialized there must be a corresponding subsection in the configuration file. The components share a common way of access the names (String getInstanceName). Note that this string can be null if and only if the component does not permit multiple instances with different configurations.

The specification do not describe how the name are stored internally in the component implementations. It is expected that the platform implementation efforts will come up with a common best practice to keep the names in component implementations.

The lexical resource interface (LexicalResource) is imported from BIUTEE to EOP, and it is well-tested by both BIUTEE and other EOP EDAs. It is expanded to include "resource-specific" relations (LexicalResourceWithOwnRelation) if a resource can support such relations. The interfaces are stable with many implementations.

However, the interfaces are not providing any means of passing the context from EDA to the resource. Thus, the existing interfaces are not adequate to represent "context-sensitive" lexical resource. So we need to define one extension that permits context-passing, and all related matters.

Let's briefly outline current assumptions and requirements before outlining the actual interface.

The followings are requirements, rather than assumptions.

The following interface outlines an interface that satisfies all of the above assumptions and requirements.

___01234567890123456789012345678901  -- CAS SOFA position
T: John walked his dog in the park.
H: John took his pet for a walk.

___01234567890123456789012345678901  -- CAS SOFA position
T: John walked his dog in the park.
H: [ don't care ]

___01234567890123456789012345678901  -- CAS SOFA position.
T: [ don't care ]
H: John took his pet for a walk.

As described in the section 3 (LAP and type system), EOP relies on CAS as the main data structure to keep various levels of annotations. From LAP level, CAS holds generic language processing results like POS, lemma, syntactic dependencies, NER, and so on.

The expressive power of CAS is not limited to generic language processing. They can also represent entailment-specific, or task-specific data types. For example, alignment annotation between Text and Hypothesis can be represented in CAS. This annotation, unlike generic LAP annotation, knows about the task and task-specific data, like Entailment pair, text side and hypothesis side, and so on.

We define this kind of component as a separate type of Entailment Core component, and call it the Annotator component. An annotator component gets one CAS, analyzes the given CAS and add additional information in forms annotation. The added annotation gives extra information for EDAs and/or other components, where the information helps EDA's decision on entailment.

What is the difference between a AnnotatorComponent and a LAP component? It seems that both adds annotations in CAS. The following list outlines the major differences.

We also define sub-types of Annotator component, to identify annotator components by their types and roles. For example, the AlignmentComponent is the subtype of AnnotatorComponent.

The interface defines only one method.

Alignment component is a sub-type of annotator component. The component interface does not add any new method. This (empty) interface is defined as an independent interface mainly to serve the purpose of identifying alignment components among alignment components.

All components that implement AlignmentComponent must annotate according to the specific CAS types that are designed for alignment annotations within EOP, which are described in the following section.

Spec 1.1 defined AlignedText CAS type. The type had been designed to express surface level of alignments (token, and phrase levels). The simple type was good for surface types, but it is not appropriate for aligning more complex ones, such as nodes, trees, graphs, or any future data that will be added for EOP CASes. In this section, we define a new, more general annotation types. The new types are designed for alignment representation on any annotation.

The design goal behind the choice to add new annotation type, is to keep the maximum flexibility while maintaining simple types that are generic enough. We define two types for this purpose.

We make no assumptions regarding what annotations are aligned by Link and Target types. One Target can be linked by an arbitrary number of Link, also a Target can group an arbitrary number of Annotations. Note that uima.tcas.Annotation is the super type of almost all CAS annotation data. Since a Target can group Annotation, it can group any type of annotations in CAS.

You can find the formal definitions of the two CAS types in the following sections. Their definitions are pretty straightforward, but please take a look on "notes on usage" of each type for its begin-end behavior, before using the types.

A RTE system generally requires the process of training: actual content of training can vary, but a working RTE engine requires various parameters, or knowledge to make a good entailment decision. Such parameters are normally acquired via a training data, and then they are stored in a model.

In the earlier versions of the specification (pre-2.0), the training capability of RTE engine was only defined on the top level, on EDA. EDA has startTraining, and initialize. The two methods works with configurable values (in the form of common configurations), and starts training with the given configuration and generate a new model(startTraining), or load a pre-existing model (initialize). This choice of top-level-only training is reasonable for systems where there is only one entry point for training. For example, TIE, EDITs and BIUTEE worked well with this setup. An instance of the EDAs only uses one model per EDA (per EDA configuration).

However, not all training behavior of RTE systems can be mapped with this way. An example is alignment-based approach. Alignment is an essential step for alignment-based RTE systems. This monolingual alignment process can be expressed as a component (as explained in annotation component) that can be used by multiple EDAs, or as of aligner module itself. This means the trained parameters of the component is independent to a specific EDA, and it should be saved as its own model.

Our approach on "trainable component" is again minimal, similar to our definition of the training method on EDA. We define a common starting point for training. Each component that can produce a model and training capability should implement the Trainable interface. This interface provides a common entry point for starting the training process. Each training process (and their required data) can vary greatly. Those varying options will be provided in the configuration.

Note that we do not define common model storage methods (e.g. what to save, where to save, what kind of format the model file has, etc). This is provided by each component, and each component that provides this interface must document the required configurable values in its JavaDoc documents. Thus, this interface only provides a common starting point for training. Although this is a minimal contract, the interface enables several common behavior.

The interface only provides one method.

void startTraining(CommonConfig configuration)

Calling this method makes the component to start the training process, as defined in the configuration values. Once the training process is finish, the method will return without any return value, but the training result will be written in a designated position in the configuration.

Note that once startTraining() is finished, the component instance does not need to comeback to "prediction (or work)" mode. For example, let's assume we have an alignment component. Normal (prediction) mode call would be annotate() method as defined in the alignment component. However, Once startTraining() method is called and the model is saved via the method, the instance do not need to be able to process further annotate() calls. Component writers can consider startTraining() as a one-way mode change command for the component to training mode. This assumption is added to make actual implementation of trainable components simpler.

Also note that we do not provide initialize() method in this interface, in contrast to EDABasic interface.

public interface SDABasic<T extends SemanticSimilarityDecision>

This interface supports basic capabilities for STS task. It defines 4 methods, similar to EDABasic interface.

In order to make an easy combination of EDAs and components possible, the configurations of all entailment core components should be stored and accessed in a uniform manner. A framework for the EXCITEMENT common configuration must meet the following requirements:

This specification assumes that the platform shares a single implementation of configuration access. We start with a set of design decisions that follow from the requirements above.

Figure 13, “Overview of Common Configuration” shows a conceptual overview of common configuration object. On the left hand side, the common configuration object is shown as a big table with hierarchy. It has two depths, sections and subsections. Each section is corresponding to a "region" for a specific entailment component. Actual configurations values are represented as lists of "(Name,Value)" pairs. A set of name-value pairs are called as a name-value table in this section.

Figure 13. Overview of Common Configuration

On the right hand side of the figure, the access of configuration values are described on a conceptual level. A component can access an instance of the common configuration object with getSection() or getSubSection() methods for components and their instances, respectively. A successful call returns a name-value table. Each value associated with a name can be accessed by a get method.

Each section and subsection is represented as a name-value table. Once retrieved as a table, there is no differences between subsection and section. Note that one cannot access subsections of a section from a name-value table. A name-value table only holds the list of name-value pairs. The table is a typed key-value list. Thus, a value (which was written in an XML file as a string) can be accessed with getString(name), getInteger(name), or getDouble(name). The XML strings will be converted automatically into the requested type.

As an in-memory object, all name-value pairs are simply seen as a pair of a name (string) and an associated value. However, in XML file level, they can actually come from variables defined in the XML file (like, <property name="something">&variable</property>). Also, the configuration file should easily encode current system environment values in the XML configuration file (For example, it should permit expressions that uses $PATH variable, like <property name="mypath">{$PATH}/etc</property>). Many configuration management libraries already support such features. (Note. The above pseudo-XML code is given only as an example. The specification does not assume anything about how the name-value pairs are represented in XML.)

For all entailment core components, the configuration is delivered as a whole (single big block contains all pairs) via initialization(). It is the responsibility of the initialization (thus of its implementers) to retrieve and use the relevant sections (by using getSection or getSubSection).

Instance configuration areas (SubSections) are only needed for the components that support multiple instances with different configurations. If all instances of a component have the same configuration, the component do not use instance configuration and uses only the component configuration. For example, knowledge components generally have no need to have different configuration even if they have multiple instances.

However, some components must have multiple instances with different configurations. Let's assume there is a distance calculation component that is shipped with two parameters. One is a path to a static resource (say, a dictionary), and the other is a numeric parameter alpha. Assume that the EDA needs to run two instances: one with alpha as 1.0, the other with alpha as 2.0. By having instance-wise configurations, the EDA can run two instances of the distance calculation component, each with different configuration. In this example, parameter path will stay in the component section, and parameter alpha will be written in the instance subsection.

Note that there is no automatic inheritance of section-level information at the subsection level. That is, component instances will typically have to query both the section level for component-wide configuration and the subsection level for their own instance configuration. (In the above example, parameter path is not visible to the name-value table of the instance subsection.)

This section first outlines the adopted XML configuration format with an example. Then, it iterates a few issues around configuration names and GLOBAL section (PlatformConfiguration).

Let's first see an example file that follows the proposed file format. It has a few sections (components) within it. Each name-values are represented with a property element. The element has one attribute (name), and actual value is written as element value within the property element.

 
<?xml version="1.0" encoding="utf-8"?> 
<!DOCTYPE configuration [
<!ENTITY myVar "Some common #PCDATA that can be reused... "> 
]> 

<configuration>
<section name="PlatformConfiguration">
<property name="activatedEDA">core.MyEDA</property> 
<property name="language">EN</property> 
</section>

<section name="PhoneticDistanceComponent">
<property name="KlingonDictionaryPath">$RESOURCE/VD/KDict.cvs</property>
<property name="beta">0.1</property> 

<subsection name="instance1">
<property name="consonantScore">1.0</property> 
<property name="vowelScore">0.6</property>
<property name="alpha">0.17663311</property> 
</subsection> 

<subsection name="instance2">
<property name="consonantScore">0.6</property> 
<property name="vowelScore">1.0</property>
<property name="alpha">0.17663311</property> 
</subsection> 

</section> 

<section name="core.MyEDA"> 
<property name="myLongKey">&myVar;</property> 
<property name="someOption">PhoneticDistanceComponent, 
                       EditDistanceComponent</property> 
</section> 

</configuration> 

Note that,

If any violation is observed (including those and possible others), CommonConfig must raise an exception while loading the file.

A small note on configuration value names and PlatformConfiguration (Global) region:

This section defines is a minimalist approach to configuration management. We leave possible extensions (like default values or possible ranges) to future versions.

The responsibility for component selection -- i.e., the decision which components to activate for a given run of the engine -- could be assigned to different layers: to the user level code, to the initialization helper, or to the top-level EDA. We make it a responsibility of the EDA, in order to allow EDAs to "hide" some complexity to its users.

Consequently, the selection which components to use forms part of the configuration of the EDA and is specified in the EDA section and/or subsection. We expect EDAs to specify how the set of components to be activated is specified as a configuration option. (Note that currently the configuration does not lists). It is the responsibility of the EDA, not of the user level code, to initialize these components.

This section describes the raw input data. By "raw input data" we mean text-hypothesis pairs without linguistic analysis layers. The RTE dataset formats (RTE1-5) are the most widely used file format for this purpose.

The internal EXCITEMENT entailment problem representation is the CAS representation described in Section 3. The raw input data formats only serve as as basis from which the CAS representations is created. Each linguistic analysis pipeline (LAP) must provide readers for the RTE dataset formats that produce CAS objects from corresponding files. All RTE formats to be supported are listed in Appendix C, Supported Raw Input Formats.

NB. RTE1-5 only support the encoding of "classical" (single H, single T) entailment problems. That is, LAPs are only required to deal with classical entailment problems. Support for multi H-T or T-multi H problems is optional. Each LAP should specify clearly in its documentation which additional problem types it supports and what file format it expects or these problems (such as RTE-6 or later).

The following section describes the RTE data formats, and adds a simple modification.

The RTE challenge data formats are the most well known and widely used data format among textual entailment community. RTE-1 to RTE-5 formats are focused on single T-H pairs.

One problem of RTE format is the lack of language marking. Language designation is important in the EXCITEMENT platform. Thus we adopted a tiny modification to RTE format. The following is the DTD of RTE-5 main task data representation.

<!ELEMENT entailment-corpus (pair+)>
<!ELEMENT pair (t,h)>
<!ATTLIST pair
         id CDATA #REQUIRED
         entailment (ENTAILMENT|CONTRADICTION|UNKNOWN) #REQUIRED
         task (IR|IE|QA) #REQUIRED >
<!ELEMENT t (#PCDATA)>
<!ELEMENT h (#PCDATA)>

It defines an XML format that holds top element entailment-corpus. The element can have one or more pair elements, which have t and h as string data. A pair must have a set of attributes. Here they are id (identifier of the pair, as string), entailment (the entailment decision), and task (as one of information retrieval, information extraction, or question answering).

We modify the DTD as follows:

<!ELEMENT entailment-corpus (pair+)>
<!ATTLIST entailment-corpus
          lang CDATA #IMPLIED
          channel CDATA #IMPLIED>
<!ELEMENT pair (t,h)>
<!ATTLIST pair
          id CDATA #REQUIRED
          entailment (ENTAILMENT|NONENTAILMENT|CONTRADICTION|
                          PARAPHRASE|UNKNOWN) #REQUIRED
          task (IR|IE|QA|SUM) #IMPLIED >
<!ELEMENT t (#PCDATA)>
<!ELEMENT h (#PCDATA)>

Now the top element entailment-corpus has an optional attribute lang and channel. This attribute holds the language identification string according to ISO 639-1 [ISO 639-1]. For example, "EN" for English, "DE" for German, and "IT" for Italian. Since it is an optional (designated as #IMPLIED) attribute, existing RTE-5 XML data will be processed as a valid XML for this DTD. In that case, no language checking will take place. This is the same for channel value. The value records the input channel (source of the input like e-mail, transcribed phone call, etc), and it is an optional value. Also, the possible values of entailment relationship are extended to match those of Section 4.2.1.6, “enum DecisionLabel”.

Support for this xtended RTE-5 format is mandatory. EDA implementers must support processing of the format. Other RTE formats are optional. EDAs (and their LAPs) may support them.

At the current stage (non-decomposed pipelines), each LAP will have to implement its own readers. (It is permissible to have one reader that can deal with all RTE1-5 formats, given the large similarities between the formats.) In the future, we foresee that RTE1-5 readers will be provided as a "collection reader" component of UIMA component, and it is expected that such components will be implemented only once and shared among all EDAs and EDA implementers.

Write modular modules.

EXCITEMENT platform will adopt some Java code annotations, starting with the followings.

Actual definitions of the Java annotations and their usage will be provided by the implementation team (WP4). Also, the list of the annotations will be expanded along with the implementation effort.