EXCITEMENT open platform: Architecture and Interfaces

The key words must, must not, required, shall, shall not, should, should not, recommended, may, and optional are to be interpreted as described in [RFC 2119]. Note that for reasons of style, these words are not capitalized in this document.

An option which we initially considered was the CoNLL shared task tabular format (http://ufal.mff.cuni.cz/conll2009-st/task-description.html). We rejected this possibility because it could not meet several of our requirements. (1), it does not specify a pipeline, just an interchange format. (2), it is extensible but at the same time fairly brittle and unwieldy for more complex annotations (cf. the handling of semantic roles in CoNLL 2009 which requires a flexible number of columns). (3), it only specifies an on-disk file format but no in-memory data structure. (4), no support for metadata.

UIMA (Unstructured Information Management Applications) is a framework that started as a common platform for IBM NLP components. It evolved into a well-developed unstructured information processing platform, which is now supported by Apache Foundation as an open source framework. It has been used by many well known NLP projects and has healthy communities in both academic and commercial domain.

UIMA provides a unified way of accessing linguistic analysis components and their analysis results. All analysis modules are standardized into UIMA components. UIMA components shares a predefined way of accessing input, output, configuration data, etc. Within UIMA, it is easy to setting up a composition of analysis components to provide new pipelines, or adopt a newly developed analysis module to already existing pipelines.

On the top level, the unification of UIMA components is achieved on two levels.

The following figure shows the Apache UIMA Java implementation as the LAP of EXCITEMENT. It provides a unified way of calling linguistic analysis components for user code.

Figure 3. UIMA as the linguistic analysis pipeline of EXCITEMENT

UIMA comes with various components and tools. However, in this specification, we will only review the components that an EDA developer should provide to the EXCITEMENT LAP as UIMA components. They are also the components that EXCITEMENT top level users can access from the linguistic analysis pipeline. Figure 3, “UIMA as the linguistic analysis pipeline of EXCITEMENT ” shows them. We first provide generic definition of AE, AAE and collection processing. Then we will state what EDA implementers should provide to realize an EXCITEMENT LAP.

AE. Analysis modules of UIMA are called as Analysis Engines (AEs). For example, individual analysis components like POS tagger, parser, NER annotators are AEs. All AEs share a set of contracted methods, and they will be accessed by the UIMA framework. All analysis results of an AE are returned to the caller as added layers on a CAS.

AAE. UIMA also provides analysis pipelines that are called Aggregated Analysis Engines (AAEs). An AAE is defined by composing multiple AEs. For example, a pipeline that outputs parsing results can be composed by subsequent calls of a sentence splitter AE, a POS tagger AE, and a parser AE. UIMA provides a standard way to compose analysis pipelines without changing anything on the individual analyzers.

Collection Processing (and UIMA Runtime). A linguistic analysis pipeline often needs to work on a collection of documents and produce a set of enriched documents. For example, a user might want to optimize parameters of a TE system with repeated experiments on the same dataset. In this case, repeating the same linguistic preprocessing would be redundant. AAEs do not provide any capability for accessing files or processing collections. It is the responsibility of the AAEs caller to recognize such cases and avoid redundant calls. To guarantee that this happens in a uniform manner, UIMA provides the concepts of UIMA runtime and collection reader.

A collection reader is a UIMA component that is specialized in data reading and CAS generation. Compared to an AE, it shares a different set of contractual methods. A collection reader reads an input (document) and generates a CAS that can be processed by AEs.

A UIMA runtime is a runner mechanism that enables running of AEs. If a user calls an AE directly (via the framework), the component is actually running in the caller's space (same process, thread). In this case, running AEs on multiple files would still be the caller's responsibility. A UIMA runtime is designed to take over that responsibility. A runtime is capable of iterating over a set of documents in its own space. Apache UIMA and the UIMA community provide various runtimes, from simple single thread runtime to asynchronous distributed runtimes. Examples for UIMA runtimes are UIMA CPE/CPM (collection processing engine / collection processing manager), UIMA AS (Asynchronous scale-out), UIMAfit simple pipeline, and the UIMAfit hadoop runtime.

The LAPs of EXCITEMENT exist primarily for processing textual entailment data. As a secondary goal, the LAPs need to provide linguistic analysis capabilities for the user level code and the transduction layer. With these goals in mind, it is possible to formulate more focused requirements for EXCITEMENT LAP components. An EXCITEMENT LAP should provide the following capabilities:

At the time of writing of this specification (V1.0), the EXCITEMENT consortium has made the following decision regarding the adoption of UIMA within the linguistic analysis pipeline:

More formally, the following requirements fall out of these decisions:

This specification provides no further information on UIMA. Please consult external UIMA documentation such as [UIMA-doc] to learn more about the UIMA framework and on how to build UIMA components.

Artifact is a UIMA concept which describes the raw input. For example, a document, a web page, or a RTE test dataset, etc, are all artifacts. For the moment, let's assume that we have a web page as an artifact, and the goal is to provide various analysis on the text of the web page.

The generic analysis tools (like POS tagger, parser, etc) do not know about HTML tags of the webpage. And it is not desirable to add such specific tag support to generic tools. In UIMA, generic tools can be used without modifications for such cases, by adopting Views. For the previous example, an HTML detagger module can generate a plain text version of the webpage. This can be regarded as a new perspective (view) on the original data (artifact). In UIMA this new view can be introduced into a CAS. Once the view is introduced in the CAS, subsequent analysis like POS tagging and parsing can process the plain text view of the web page without knowing that it is from a webpage.

Subject OF Analysis (SOFA) is another UIMA concept. It is referring the data subject that is associated with a given view. In the website example, it is the plain text version of the web page. One view has one SOFA, and one CAS can have multiple views. Views (and Sofas) can have names that can uniquely identify them among a given CAS. Annotations of CAS generally have a span: beginning position and the end positing of the annotated text. Such spans are expressed as offset of each SOFA. Thus, annotators (individual analysis engines) are annotating SOFAs, instead of artifacts directly.

A pictorial example of CAS, views and annotations are given after introduction of the types for textual entailments, in Figure 4, “Example of an entailment problem in CAS”.

The UIMA framework itself defines only a handful of default types. They are including primitive types (like string, numerical values), and basic annotation types.

Primitive types only represent simple values, but user-defined types can express arbitrary data by composing primitive types, and other types. For example, a token type might have a string (primitive type) for lemma, another string for POS, and two numbers for starting position and end position of the token. Within UIMA CAS, this list of values are called as feature structures. Each feature in the feature structure has a name and a type, thus only a specific type can be pointed by the feature.

UIMA type system is based on single-parent inheritance system. Each type must have a one super-type, and it gets all features from the super-type.

The followings are non-exhaustive list of the default types that are used in the adopted and proposed type system of the specification.

Fully detailed information on CAS, and types of default CAS can be found in the CAS reference section of the UIMA reference document [UIMA-CAS].

This section outlines adopted and proposed UIMA types for generic NLP analysis results. The full list of the types are given in the Appendix A, Type Definition: types for general linguistic analysis.

In this section, the string DKPRO is used as a short-form of de.tudarmstadt.ukp.dkpro.core.api.

This section outlines the types proposed for textual entailments and related sub-tasks. Full listing of the mentioned types are listed in Appendix B, Type Definition: types related to TE tasks .

3.3.4.1. Representation of Text and Hypothesis

The basic approach of the platform is to use a single CAS to represent a single entailment problem. The CAS holds text, hypothesis and associated linguistic analysis annotations.

Let's first consider Figure 4, “Example of an entailment problem in CAS” which shows an example of entailment problem represented in a. It includes UIMA CAS elements like artifact, view, and SOFA (cf. Section 3.3.1, “Relevant UIMA concepts: Artifact, View, SOFA”). The original raw input is shown in the figure as the artifact. It is a simple XML annotation with a T-H pair. The artifact contains two plain-text subjects of analysis (SOFA). Analysis tools like taggers will only see this SOFA data, instead of the artifact XML. All annotations (all types that inherits uima.tcas.Annotation) are annotated on top of the SOFA. For example, the first token annotation of the text ("this") is annotated on TextView SOFA position 0 to position 3. This SOFA positional span underlies the most common way of accessing fetching UIMA text annotations, namely to retrieve, e.g., "all annotations of type Token from the TextView SOFA span 0 - 14".

Each view has exactly one SOFA. In this example, each view is enriched by token annotations and POS annotations. Both token and POS are annotation types, and have begin and end fields, which shows the span of the annotation. (POS types also have begin and end span. They are omitted here for clarity.)

Figure 4. Example of an entailment problem in CAS

The figure also shows a set of entailment specific annotation types: they are entailment.Pair, entailment.Text, entailment.Hypothesis, and entailment.EntailmentMetadata. The example has a single entailment.Pair, which points entailment.Text and entailment.Hypothesis. The Pair has an ID, and it also has a goldAnswer. The CAS has no additional pair, and the entailment problem described in this example is a single T-H pair. There is one entailment.EntailmentMetadata in this CAS. It shows that the language of the two views is "EN" (English as defined in [ISO 639-1]), and the task is "QA". It also has a string for channel value as "email".

The following subsections define the annotations in detail.

3.3.4.1.1. Type entailment.EntailmentMetadata

EXCITEMENT.entailment.EntailmentMetadata provides metadata for each entailment problem. It is an extension of uima.cas.TOP. An instance of this type should be indexed directly on the CAS (not on TextView or HypothesisView).

• language (uima.cas.String): this string represents the language of the CAS and its entailment problem. It holds two character language identifier, as defined in [ISO 639-1]).

• task (uima.cas.String): this string holds task description which can be observed in the RTE challenge data.

• channel (uima.cas.String): this metadata field can holds a string that shows the channel where this problem was originated. For example, "customer e-mail", "online forum", or "automatic transcription", "manual transcription", etc.

• origin (uima.cas.String): this metadata field can hold a string that shows the origin of this text and hypothesis. A company name, or a product name.

• TextDocumentID (uima.cas.String): This field can hold a string that identifies the document of the TextView. This feature must have a value, if TextCollectionID is not null.

• TextCollectionID (uima.cas.String): This field can hold a string that identifies the collection name where the document of the TextView belongs to.

• HypothesisDocumentID (uima.cas.String): This field can hold a string that identifies the document of the HypothesisView. This feature must have a value, if HypothesisCollectionID is not null.

• HypothesisCollectionID (uima.cas.String): This field can hold a string that identifies the collection name where the document of the HypothesisView belongs to.

The values of these fields are unrestricted Strings. Pipeline users are responsible for documenting the semantics of the values if they want to ensure interoperability.

3.3.4.1.2. Type entailment.Pair

Text-Hypothesis pairs are represented by the type EXCITEMENT.entailment.Pair. The type is a subtype of uima.cas.TOP. An instance of Pair should be indexed directly on the CAS (not on TextView or HypothesisView). If the CAS represents multiple text and/or hypothesis, the CAS will holds multiple Pair instances.

The type has following features:

• pairID (uima.cas.String): ID of this pair. The main purpose of this value is to distinguish a certain pair among multiple pairs.

• text (EXCITEMENT.entailment.Text): this features points the text part of this pair.

• hypothesis (EXCITEMENT.entailment.Hypothesis): this features points the hypothesis part of this pair.

• goldAnswer (EXCITEMENT.entailment.Decision): this features records the gold standard answer for this pair. If the pair (and CAS) represents a training data, this value will be filled in with the gold standard answer. If it is a null value, the pair represents a entailment problem that is yet to be answered.

3.3.4.1.3. entailment.Text, entailment.Hypothesis and entailment.Decision

EXCITEMENT.entailment.Text is an annotation (extending uima.tcas.Annotation) that annotates a text item within the TextView. It can occur multiple times (for multi-text problems). It does not have any features. A Text instance is always referred to by a Pair instance.

Note that Text annotations and TextView are two different concepts. A TextView is a container that contains a SOFA plus all of its annotations, including entailment.Text Thus, the fact that a sentence forms part of a TextView does not automatically mean that it is compared to a hypothesis. This only happens if it is annotated with entailment.Text and linked with entailment.Pair.

The situation is similar for EXCITEMENT.entailment.Hypothesis. It is also an annotation (uima.tcas.Annotation) that annotates a hypothesis item within the HypothesisView. It can occur multiple times (in multiple-hypothesis cases). It does not have any features either. A Hypothesis instance is always referred to by a Pair instance. Hypothesis annotations and HypothesisView are different entities, and sentences are only compared to a specific text annotation when they are annotated with entailment.Hypothesis, like for texts.

The EDA process interface receives CAS data as a JCas object (the UIMA-created Java type hierarchy corresponding to the UIMA CAS types). CASes that feed into the EDA process interface must have at least a single entailment.Pair. Naturally, each view (TextView and HypothesisView) must have at least one entailment.Text and entailment.Hypothesis. Multiple text and multiple hypothesis cases will be represented by multiple number of entailment.Pair. Note that the relationship between pairs on one side and texts and hypotheses is not a one-to-one relationship. Several pairs can point the same hypothesis, or to the same text. For example, if one hypothesis is paired with several potential texts, there will be multiple pairs. All pairs will point to the same hypothesis, but to different texts.

EXCITEMENT.entailment.Decision represents the entailment decision. It is a string subtype, which is a string (uima.tcas.String) that is only permitted with predefined values. entailment.Decision type can only haves one of "ENTAILMENT", "NONENTAILMENT", "PARAPHRASE", "CONTRADICTION", and "UNKNOWN" (The type can be further expanded in the future). This UIMA type has a corresponding Java enum type within the entailment core (see Section 4.2.1.6, “enum DecisionLabel”).

3.3.4.2. Generalization to multiple pairs

Figure 5, “Example of an entailment problem with multiple pairs” shows another example of a CAS with entailment types. In this figure, the CAS has multiple pairs. The TextView holds a document with six sentences. All six sentences are annotated by sentence splitter, and additional analyzers (for clarity, only sentence annotations are shown). Among the analyzed sentences, the entailment.Text annotations are marked on the first, third and sixth sentences. The HypothesisView holds only a single sentence, which is the single Hypothesis annotation. The CAS holds three pairs, and it is a multiple text entailment decision problem. In the figure, linguistic analysis annotations other than sentences are omitted for clarity. The sentences that are not annotated with a Text annotation can be regarded as context for the annotated texts. For example, co-reference resolutions will take benefits from the sentences. Also, components like distance calculation components, or text rewriting components can use the tokens in the context sentences to get better results. (Note that generic components like tokenizer, parser, and co-reference resolver, do not aware of entailment specific annotations, and process all data in the given view. "Generic" verses "TE specific" LAP component is described in the next subsection Section 3.3.4.3, “Two groups of LAP components”.)

Figure 5. Example of an entailment problem with multiple pairs

Note that EDAs will only decide entailment of pairs that are explicitly marked with Pair annotations. Existence of text or hypothesis annotation does not automatically lead to processing in an EDA. For example, if the pair-2 is missing from the figure, the Text annotated on sentence-3 will not be compared to the hypothesis sentence. Also note that nothing stops us from annotating a text item that overlaps with other text items. For example, it would be a legal annotation if we add a new text item that covers from sentence-2 to sentence-4 (which will include the text of pair-1).

3.3.4.4. Other cases: T-H from multiple documents, T-H on a same document

Ordinary annotators (taggers, parsers, etc) do not (and should not) know about the views of TextView and HypothesisView. They only see one view at a time, its SOFA as a single document. Thus the two views of T and H naturally represent two separate documents, one for text and one for hypothesis.

One problem of this setup is that we have to process other cases like:

• Text (or Hypothesis) from Multiple documents.

• Text and Hypothesis on the same document.

The following subsections describe how we will represent such cases with the proposed types and CAS.

3.3.4.4.1. Multiple document problem: decomposition to multiple CASes

Some problems involve multiple documents. Examples are the main task of RTE6 and RTE7. They have TE problem that consists of multiple texts (T) and a single hypothesis. The texts are scattered over a set of documents (collection). Additionally, there are global features associated with the document collection (e.g., topic).

We can break down this case with multiple CASes with the proposed CAS representation. The following figure shows one artificial example. In the figure, it has a single hypothesis and multiple texts. The texts are located in three different documents. It is possible to represent this problem with three CASes.

Figure 6. Multiple document TE problem decomposed into multiple CASes.

Each CAS of the decomposed one will look like Figure 5, “Example of an entailment problem with multiple pairs”. Note that the multi document example is decomposed into several CASes whose number corresponds to the number of documents. Each view holds a single document in this setup, and normal annotators can process each view just as a normal document.

EXCITEMENT.entailment.EntailmentMetadata holds two additional features for multiple document cases. It has documentID and collectionID for both text and hypothesis. Document ID identifies the document assigned in the view, and collection ID can show the collection where the document is belong to.

Note that this decomposition assumes that breaking down the multiple document case into multiple single document problem does not change the final output. See Section 3.3.4.4.3, “More complex problems and their use-cases ” for more complex cases of multiple text/hypothesis.

3.3.4.4.2. T and H on same document : Two views hold the same document

Since we have two separate views for text and hypothesis, it can be problematic when we have the text and the hypothesis from the same document.

In such a case, we will have two views with the same document (two views have the same document with same analysis result). All other TE related annotations stays normal, like metadata and entailment problem annotation. Only document specific annotations (like tagging, parsing) are duplicated into two views.

Note that this "duplicated analysis result in two views" does not mean the analysis of the document has to be done twice. We can setup a special pipeline for single document T-H case, where it first runs analysis and copies the result into two views.

3.3.4.4.3. More complex problems and their use-cases

Note that the above two mappings (multiple documents THs and single TH) simplify EDA interfaces. For EDAs, all inputs look alike with two views with entailment.Pair type annotations. However, we can imagine more complex use cases where no trivial mapping is possible. Within the WP3 discussions, the following tasks have been discussed for possible future support.

• Text expansion: Automatically generating a (large) set of additional texts that can entail the given input text.

• Dialog system: Ranking hypothesis for the given text. The goal is to rank, instead of decision.

• Hypothesis with variables: Hypothesis that includes variables, where the variables can be replaced by parts (like words) that appears in the text.

For such cases, we will probably additional interfaces on EDA side, also with new problem descriptions. However, this version of specification only deals with existing, well-known RTE problem sets. The WP3 members have agreed that additional use-cases will only be discussed in a future version of the specification.

It is possible that users want to add additional information to entailment problems. For example, task-oriented TE modules might want to include additional information such as ranks among texts, relevance of each hypothesis to some topics, etc.

The canonical way to represent such information is to extend the entailment.Pair and related types. The type and related types (like MetaData, Text and Hypothesis, etc) are presented to serve the basic need of the EXCITEMENT platform, and additional data can be embedded into CAS structure by extending the basic types.

Naturally, the extension should be performed in a consistent manner. This means that the implementer can only define a new type that is extending the existing types, and may not change already existing types. Also, it is recommended that attributes and methods inherited from the existing types should be used in the same way so that components and EDAs which are unaware of the extensions can still can operate on the data.

The type EXCITEMENT.entailment.Decision is also open for future extensions. When we define a new relation between text and hypothesis, this type should be first extended to cover the new relation. (This extension should always done with the extension of internal DecisionLabel enumeration, see Section 4.2.1.6, “enum DecisionLabel”.

Unlike EXCITEMENT types, generic types (of Section 3.3.3, “Types for generic NLP analysis results”) should in general not be extended. Exceptional cases may arise, though, for example when an EDA utilizes some additional information of a specific linguistic analysis tool. In such a case, one may choose to extend generic types to define special types (for example, defining myNN by extending NN, etc). The EDA can then confirm the data is processed by a specific tool (like existence of myNN) and can use the additional data that is only available in the extended type. Other modules that do not recognize the extended types are still able to use the super-types in the output (in the example, myNN will map onto NN).

It would be ideal if each linguistic analysis module (like POS taggers, parsers or coreference resolvers) were provided as a generic individual UIMA analysis engine (AE). With individual AEs, user code would be able to call these modules as UIMA components, as described in Section 3.2.2, “The EXCITEMENT linguistic analysis pipeline”.

By generic, we mean they are supposed to process normal text and not T/H pairs. The modules are not supposed to process or even be aware of annotations related to entailment problems. This ensures their reusability. The necessary specific processing for entailment data (H/T markup) should be implemented within a complete pipeline, cf. the following subsection.

As described in Section 3.3.5, “Extending existing types”, all analysis engines must annotate the given data with the common type system. In the rare case where the provided common type system is not sufficient, the implementer must not modify any of the existing types and should only expand existing types with an own type that inherits existing types.

It is recommended that the implementer should reuse existing AEs as much as possible. For example, DKPro already provides many of the well known NLP tools including major parsers and taggers for various languages. ClearTK or OpenNLP also ships various NLP tools as UIMA components. ClearTK or OpenNLP uses different type systems, thus one needs to add additional glue code to map the types into adopted DKPro types. However, such mappings are generally trivial and should not pose large problems.

Note that there is more than one flavor in providing UIMA components. This specification only defines types and components in terms of the bare UIMA framework. Each component is supposed to have its component description, and should be runnable by the UIMA framework. However, the implementers are free to utilize additional tools. For example, DKPro and ClearTK both uses UIMAFit as tool layer, which provides the functionalities of the UIMA framework with additional tool layers. The implementer may utilize such tools, as long as the end results are compatible with what are defined by this specification.

In some cases, one may choose NOT to provide individual analysis components as UIMA components. See Section 3.4.4, “An alternative: Independent pipelines with CAS output” for such a case.

Individual analysis engines (AEs) are generic. They process a single input (single CAS) and add some annotation layers back to CAS. As mentioned above, however, EDAs require more the sum of single AE annotations, namely T-H views, and T-H pair annotations. Thus, it is desirable to provide complete "pipelines" that is properly annotated with types needed for textual entailment.

A pipeline for EXCITEMENT EDA must process the followings:

EDA implementers must provide at least a single pipeline that produces suitable input for their EDA, or declare that an already existing pipeline as the suitable pipeline. If the EDA supports additional input mode like multiple texts/hypotheses, the implementer should provide a corresponding pipeline that produce multiple text/hypothesis.

Pre-processing of a set of document (collection) is often needed for NLP applications. In textual entailment context, a pre-processed set of training/testing data can be beneficial for various cases.

EDA implementers should provide the collection pre-processing capability.

Note that XMI (XML Media Interchange format) serialization is the standard serialization supported by UIMA framework. It stores each CAS as an XML file, and the stored files can be easily deserialized by UIMA framework utilities. You can find additional information about CAS serialization in [UIMA-ser]

It is recommended that individual analysis components are supplied as UIMA analysis engine (AE). This will make sure those analysis modules can be used by top level users in various situations, as suited for the need of application programmers.

However, there can be situations where EDAs depend on specialized analysis modules which form a pipeline that cannot easily be divided into individual UIMA analysis engines. For example, assume that there is a parser and a coreference resolution system that are designed to work together to produce a specific output for a given EDA. Then, it might not be desirable to break them into individual analysis engines. In such a case, one may choose to provide only the pipeline as a single analysis engine which translates the final output of the pipeline into the common representation format.

Even in these cases, the developers should provide the pipeline as an UIMA component. This will ensure that the pipeline can be called in an identical manner to normal pipelines.

To determine the set of reusable components and their interfaces, we started from the three systems that we primarily build on and followed a bottom-up generalization approach to identify shared functions (components), interfaces and types. The results are discussed in Section 4.1.3, “Identifying common components and interfaces ”. Further functionality of these components (e.g. details of their training regimens) are not be specified but in the scope of the component's developer

By following the bottom-up approach, we have identified a set of major common interfaces. The following figure shows the common components within the entailment core.

Figure 7. Common components within EXCITEMENT entailment core.

The top-level interfaces of the entailment core consist of the EDA interface and the mode helper interface. EDAs are the access points for entailment decision engines. They receive annotated input in the form of CAS, and return a decision object. The EDA interface also includes support for multiple Text and/or multiple Hypothesis modes. The EDA interfaces are described in Section 4.2, “EDA Interface ”. The mode helper is a wrapper tool that supports multiple text and multiple hypothesis mode for EDAs that can only process single T-H pairs. Its interface is described in Section 4.3, “Auxiliary Layer for EDAs”.

We have identified three main classes of components that EDAs might wish to consult. The first type is distance calculation components. EDAs can use the semantic distance between T-H pairs as their primary decision factor, or can use them as features in more general decision algorithms. The interface of distance calculation components specifies that they accept a CAS as the input, and return an object the distance between the two views of the CAS. The interface is described in Section 4.6, “Interface of distance calculation components”.

The second and third component classes deal with different kinds of linguistic knowledge, namely lexical knowledge and syntactic knowledge. Lexical knowledge components describe relationships between two words. Various lexical resources are generalized into EXCITEMENT common lexical knowledge interface. The interface is described in Section 4.7, “Interface of lexical knowledge components ”. Syntactic knowledge components contain describes entailment rules between two (typically lexicalized) syntactic structures, like partial parse trees. For entailments, the knowledge can be generalized into rewriting rules at partial parse trees, where left hand sides are entailing the right hand sides. The common interface for syntactic knowledge is described in Section 4.8, “Interface of syntactic knowledge components ”.

The EXCITEMENT platform uses UIMA CAS type system for the linguistic analysis pipeline (Section 3). The UIMA types are expressive enough to describe the relevant data structures, and can be used both in-memory and serialized. It is thus a natural question whether CAS types can be re-used for the communication between EDAs and components.

Our general guideline within the Entailment Core is that the interfaces are supposed to use UIMA CAS only when a complete text (Text or Hypothesis, potentially with context) is exchanged as a parameter. The reason is that even though CAS types (JCas objects) are expressive enough to describe any data, they are basically flat structures --- tables with references. This limits their capability to be used efficiently. A second reason is that JCas objects only works within CAS context. Even simple JCas objects like tokens need a external CAS to be initialized and used. This makes it impractical to use JCas objects and their types in interfaces that do not refer to complete texts (which are modeled in CASes). To repeat, we use JCas and related types only when a reference to the whole text is exchanged. Furthermore, we only exchange references to the top-level JCas object and never to any embedded types like views or annotation layers.

A second aspect that must be handled with care is the extension of JCas objects. JCas provides the flexibility to extend objects while keeping the resulting objects consistent with the original JCas types. However, this can cause duplicated definitions when mismatches are introduced. Moreover, CAS types will be used and shared among all EXCITEMENT developers which makes extensions introduced by one site problematic. For these reasons, our policy is that JCas objects should not be extended.

This interface defines the basic capability of EDAs that all EDAs must support. It has four methods. process is main access method of textual entailment decision. It uses JCas as input type, and TEDecision as output type. The interface as Java code is listed in Section C.1, “interface EDABasic and related objects”.

public interface EDAMultiT<T extends TEDecision>

public interface EDAMultiH<T extends TEDecision>

public interface EDAMultiTH<T extends TEDecision>

Each of the interface defines a method, namely processMultiT, processMultiH, and processMultiTH. EDAs may choose to support them or not, since supporting multiple T/H interfaces are optional. Multiple Texts and Hypotheses are marked in the CAS by multiple EXCITEMENT.entailment.Pair annotations (compare Figure 5, “Example of an entailment problem with multiple pairs”).

public abstract class InitializationHelper<T extends TEDecision>

The main access point to the entailment core is the EDA interfaces. Users of the platform will process the data with LAP, and will use EDA interfaces to get the entailment results. To use an EDA, an instance of the EDA object must be first instantiated, and then it must be initialized with a proper configuration. Then, the EDA will initialize all needed components accordingly (see also Section Section 5.1.6, “Component selection”.

Thus, to start an EDA's initialization sequence, one must first instantiate an instance, and then feed it a configuration object. This also assumes that the configuration file is already read, and the main EDA that is defined in the configuration is already determined. InitializationHelper is an auxiliary layer that takes care of this instantiation and initialization. Without the helper, all of those process would have to be done by user code, which would require different initialization sequences for different EDAs.

InitializationHelper solves this problem by providing a common initialization capability. The helper code accepts a configuration file, and then it reads the configuration file to determine the main EDA class. Then, the helper instantiate and initialize the EDA, and returns a ready-to-use EDA instance.

public abstract class MultipleTHModeHelper<T extends TEDecision>

The "mode helper" is a wrapper implementation that uses a single-mode EDA (EDABasic) to support multiple T-H cases (that of EDAMultiT, EDAMultiH, and EDAMultiTH). It receives an already initialized EDA and iteratively calls the EDA to produce results for multiple T/H cases. It is a baseline implementation that can be used with any EDA.

It is defined as an abstract Java object. The interface code is listed in Section C.4, “class MultipleTHModeHelper” It is expected that the platform will share one implementation of mode helper for all EDAs.

public interface SinglePairProcessHelper<T extends TEDecision>

This interface is a convenience interface for "end users" of an entailment engine. The implementation of this interface uses a specific implementation of EDA (EDABasic), and provides a method that allows the user to run the complete engine including LAP and Entailment Core with a single call. The access method receives two String objects as input (the text and the hypothesis, respectively), and returns the result (an object that extends TEDecision) for them.

Since this is a convenience interface, it is limited with regard to functionality and efficiency. It does not support context sentences, or multiple text/hypothesis situations. Also, calling the LAP for just one sentence pair is presumably quite inefficient.

Note that, unlike MultipleTHModeHelper, the platform cannot share a single ProcessHelper implementation for all EDAs. Each EDA needs to run a different LAP, thus each EDA is expected to have its own ProcessHelper.

public String getComponentName()

This method provides the (human-readable) name of the component. It is used to identify the relevant section in the common configuration for the current component. See Sections Section 5.1.2, “Overview of the common configuration ” and Section 4.10.3, “Component name and instance name”.

public String getInstanceName()

This method provides the (human-readable) name of the instance. It is used to identify the relevant subsection in the common configuration for the current component. See Sections Section 5.1.2, “Overview of the common configuration ” and Section 4.10.3, “Component name and instance name”. Note that this method can return null value, if and only if all instances of the component shares the same configuration.

Previously, the method void initialize(CommonConfig c) was included in the interface Component. This means that all components are supposed to be initialized by this method, with a configuration object. Since specification verion 1.1.2, the method is removed from the interface. This changes the expected behavior of a component for its initialization.

All component implementations should satisfy the following conditions.

Note that the overridden constructor that knows how to deal a the configuration object can have forms other than the single argument of CommonConfig. For example, components that permits multiple instances might need additional arguments. It might look like GermaNet(CommonConfig c, String instanceName), where the instanceName will identify proper configuration subsection for that instance, etc.

Also note that EDA still keeps its initialize() method, in the EDABasic interface. This will force all EDA implementations to have this method. This reflects that EDA will always have a dependency to the configurations, and the method will be used by top-level initialization helper and other user-layer codes.

The specification does not define how and where component and instance names is stored internally within the component implementations. It is expected that the implementation efforts will implement a common practice of keeping instance names.

Like all other components, the interface extends interface Component. The interface adds a single method calculateScores(), which gets a JCas as the input argument, and returns a Vector<Double>.

The interface is a subinterface of ScoringComponent. The new methods it adds is calculation(), which gets a JCas and returns an object of DistanceValue. The interface definition in Java code can be found in Section C.6, “interface DistanceCalculation and related objects”.

This type holds the distance calculation result. It has some member variables and public access functions for the variable. The type is used to exchange data between the component and the EDA. Its variables are set during initialization. The actual java code listing is included in Section C.6, “interface DistanceCalculation and related objects”.

public final class LexicalRule<I extends RuleInfo>

This type represents a generalization of lexical relationships between two words, which are provided by resources like WordNet, VerbOcean and distributional resources. It has two parts: a left hand side (LHS) and a right hand side (RHS). The basic arrangement of lexical resource is that the left hand side has a relationship to the right hand side. Normally, this relationship is entailment (LHS entailment RHS; return values of Section 4.7.2, “Interface LexicalResource”). However, they can also represent resource specific or canonical relations (see Section 4.7.3, “interface LexicalResourceWithRelation ” for querying such relations). A lexical rule also provides additional pieces of information include confidence, relation name, and resource name. Finally, lexical rules are parametrized by a type I (short for information) which allows the type to hold additional resource-specific properties.

Before going into details of the class variables, let us first see a few examples. The following figure shows a pictorial example of the data type. It shows two examples of lexical relations. Each relation is represented with a few data features.

Figure 8. Examples of lexical rule instance

On the left, it shows an instance of LexicalRule that represents the relationship between "pipe" and "tube". The value originalRelation holds the relation (here the "hypernym") from left ("pipe") to the right ("tube"), as recorded in the knowledge resource (here, WordNet). In this example, the left hand side entails the right hand side. The confidence is not provided by the implementation, and it is given as DEFAULT. Additional information is provided in the example with info variable. The type variable I is instantiated by a type "WordNetInfo" which holds information about synsets.

On the right, another example shows that "Magritte" entails "painter". In this case, the knowledge is captured from Wikipedia, and the resource dependent information provides various Wikipedia-based links and scores. Here, the implementation also provides a confidence value.

All lexical knowledge of EXCITEMENT resource will be delivered as an instance of this object. Java source code of the class LexicalRule is provided in Section C.7, “class LexicalResource and related objects”.

public interface LexicalResource<I extends RuleInfo>

A lexical resource is a collection of lexical rules of a certain type (like WordNet, or VerbOcean) which can be queried. The interface provides three query methods. Queries are specified by lemma and POS pairs, and the results are returned as a list of LexicalRule objects. The Java source code of the interface is listed in the Section C.7, “class LexicalResource and related objects”. LexicalResource is a subinterface of Component and adds the following methods:

The implementation of getRules methods must return an empty list (not null) if no applicable rules are found.

Note that all lexical rules returned by this interface represent entailments (LHS->RHS entailments). In this sense, the methods only support querying of "entailments". If your lexical resource supports additional relational queries, you can provide them with the additional interface Section 4.7.3, “interface LexicalResourceWithRelation ”.

public interface LexicalResourceWithRelation<I extends RuleInfo, R extends RelationSpecifier> extends LexicalResource<I>

The interface LexicalResource (Section 4.7.2, “Interface LexicalResource”) defines the common method of querying and getting lexical entailment rules of the given words. This abstraction is useful in the sense that it abstracts various underlying lexical resources with the entailment relation. This enables the caller of LexicalResource implementations to query various knowledge resources in the same fashion.

However, this comes with a price: the LexicalResource interface cannot provide querying capability of relations others than entailment. For example, user of that interface cannot query NonEntailment words of the given term (terms that you are sure that is not entailment), nor resource-specific relations (like synonym or hypernym of WordNet, or Stronger-than or happens-before of VerbOcean, etc). These resource-specific relations are reported back as part of the query result (RuleInfo within LexicalRule), but cannot be directly asked for.

LexicalResourceWithRelation is defined to recover this capability. It permits lexical resource implementers to define additional query methods with relations. It enables not only the canonical relation queries but also queries with resource-specific relations.The interface extends LexicalResource, and adds three methods. Essentially, each of the method has one additional argument that represents the relation to be queried.

public interface LexicalResourceWithRelation<I extends RuleInfo, R extends RelationSpecifier> extends LexicalResource<I>

Each of the method gets one additional parameter R relation that specifies the relation to be fetched. For example, if relation is "NonEntailment", the resource will return rules that specifies "NonEntailment" (where LHS->RHS is confidently cannot be entailment). If the relation holds "Synonym", it means the query should return Lexical rules where LHS to RHS is synonymy.

Note that the relation is represented by a generic parameter R. Each R is an extension of RelationSpecifier, where the R can be tailored for each resource. We adopt a simple hierarchy for this R. The interface RelationSpecifier and related objects are described in the following subsections.

The following figure shows the simple hierarchy of RelationSpecifier. The interface RelationSpecifier stays in the top, and two interfaces (CanonicalRelationSpecifier and OwnRelationSpecifier<E>) extends this top interface.

                   RelationSpecifier 
                    /             \
                   /               \
CanonicalRelationSpecifier     OwnRelationSpecifier<E> 

The intention of adopting the hierarchy on relation specifier is to explicitly mark the class of relations that are supported by each LexicalResourceWithRelation. For example, all lexical resources that support canonical relation bears the same signature of R (as <R extends CanonicalRelation>). This is the same for resources that support resource-specific relations (R as <R extends OwnRelationSpecifier<E>>).

Let's check each of the interfaces.

public class GermaNetRelation implements CanonicalRelationSpecifier, 
                         OwnRelationSpecifier<WordNetRelation> 
{

   public GermaNetRelation(TERuleRelation canonicalRelation, 
                           WordNetRelation ownRelation)
   {
      this.canonicalRel = canonicalRelation; 
      this.ownRelation = ownRelation; 
   }

   public GermaNetRelation(TERuleRelation canonicalRelation) {
      this(canonicalRelation, null);
   }

   public GermaNetRelation(WordNetRelation ownRelation) {
      this(null, ownRelation); 
   }

   @Override // from canonical relation specifier 
   public TERuleRelation getCanonicalRelation() {
      return canonicalRel;  
   }

   @Override // from own relation specifier 
   public WordNetRelation getOwnRelation() {
      return ownRelation; 
   }

   private final TERuleRelation canonicalRel; 
   private final WordNetRelation ownRelation; 
}

public class GermaNetImplementation implements 
   LexicalResourceWithRelation<GermaNetInfo, GermaNetRelation> { ...

GermaNetImplementation germaNet = new GermaNetImplementation(); 

String lemma = "beautiful";  
PartOfSpeech pos = new GermanPOS("adjective"); 

// query for synonym of "Beautiful"
GermaNetRelation rel1 = new GermaNetRelation(WordNetRelation.SYNONYM); 
germaNet.getRulesForLeft(lemma, pos, rel1); 

// query for NONENTAILMENT terms to "Beautiful" 
GermaNetRelation rel2 = new GermaNetRelation(TERuleRelation.NONENTAILMENT);
germaNet.getRulesForLeft(lemma, pos, rel2); 

String lemma = "apple";  
PartOfSpeech pos = new GermanPOS("noun"); 

// query for ENTAILMENT and hypernym of "apple"
GermaNetRelation rel3 = new GermaNetRelation(TERuleRelation.ENTAILMENT,
                                             WordNetRelation.HYPERNYM);
germaNet.getRulesForLeft(lemma, pos, rel3); 

public class SyntacticRule

The class represents an instance of a syntactic rule.

4.8.1.3. Basic Node: Common representation for dependency parse tree

To share a common representation for syntactic knowledge, a shared syntactic representation is required. The class BasicNode provides the common syntactic representation of EXCITEMENT. It represents a dependency parse tree node, linked to other parse tree nodes. By acquiring the top node, it can also represent a parse tree, or a partial parse tree.

The term "basic" hear means that it contains only the basic information - information that is common to almost all languages. It does not contain information that appear only on some languages like grammatical role, morphological form, etc. It is expected that specific language nodes will be implemented by extending the BasicNode.

Figure 11, “Contents of a basic node” gives a graphical overview of the basic node.

Figure 11. Contents of a basic node

A node basically holds two sets of member variables. One is about tree itself (parent-children relation), and the other is information on edges and nodes.

• children: this member variable holds an array of child nodes. They are nodes of the same type. Note that the node does not explicitly keep its parent.

• antecedent: this member variable holds a reference to "duplicated node" in the dependency tree. For example, in a sentence like "He managed to go", it is possible to generate a duplicated node "He". The antecedent field of the duplicated one will point the original node.

• info: this member variable holds various information of its edge (incoming edge from the parent), and the node content.

The member info consists of two parts: edgeInfo holds information on edge, and nodeInfo has node content. edgeInfo holds the following information.

• relation: this is an instance of data object that represent the dependency relation. The instance holds two variable. One is itsStringRepresentation that holds the relation output of the dependency parser. And the other is type which holds the an enum value that represents dependency relationship. The enum DependencyRelationType closely reflects the adopted UIMA dependency types (the subtypes of type.dependency.Dependency).

The nodeInfo holds the following information:

• word: this string holds the word (token) itself.

• lemma: the lemma of this token

• namedEntity: it holds a enum value that represents a NER class. If this node is not a named entity, this value is null. The enum type NamedEntity closely reflects the adopted UIMA NER types (subtypes of ner.type.NamedEntity).

• syntacticInfo: this variable holds information about POS tag of this token. It holds two variables: the string output of the POS tagger (posTagString) and the normalized POS tag (cannonicalPosTag) that is equivalent to POS types of UIMA CAS.

Note that (not shown in the figure), nodeInfo also provides two methods isVariable() and getVariableID(). isVariable() method returns true if and only if the node represents a variable of a syntactic rule. If the node is a variable, getVariableID() returns ID of the variable as an integer. (The ID is only for human readers and rule visualizations -- actual rule matches and applications are done by bidirectional mappings provided in the rule.)

The BasicNode is implemented with two generic parameters. One is the info type (that holds nodeinfo and edgeinfo), and the other is node type itself (children and antecedent). Generic parameters are omitted in the text paragraph for readability, but they are described in the appendix source code in Section C.8.2, “class BasicNode and related classes”.

A syntactic resource is a collection of syntactic rules. For syntactic rule collections, it is not practical to provide simple access interfaces like getLeftForPOS for lexical knowledge. Due to the exponential number of subtrees in any text and hypothesis, naive querying based on the input is infeasible. Instead, findMatches() method is defined to outline common behavior of platform syntactic rules.

Modern computer hardware is very well prepared for concurrent and parallel processing. Even modest PCs now have multiple cores with multithreading capability. Given users' requirements of quick or even real-time processing, it is important that the EXCITEMENT platform supports concurrent processing.

The EXCITEMENT architecture (as seen in Figure 7, “Common components within EXCITEMENT entailment core.”) can incorporate concurrent processing at two different levels. We will call them concurrent component execution, and concurrent EDA execution. They have different advantages and disadvantages.

Concurrent component execution. Imagine an EDA that requires the results of three sub-components for its entailment decision (a typical situation for TIE or EDITS). Without concurrent processing, the EDA must call three sub-components sequentially and has to postpone its decision to when all three results are available. Concurrent component execution can reduce the total running time by reducing the time needed to produce a single TE decision.

Concurrent EDA execution. Imagine an EDA which must process a set of textual entailment problems. If the time needed to process the set is n, we can reduce the required time to n/m by running m threads for the EDA. This level of concurrency will reduce the time to produce a set of TE decisions by concurrent processing of the selected EDA.

We have decided on the following contract for concurrent processing.

The goal of this interface is to provide an access method for EDAs (and/or EDA wrappers) that can process a set of entailment decision problems concurrently.

List<TEDecision> processDataSetConcurrently (List<JCas> casList)

The interface receives a set of TE decision problems as a list of JCAS. Each CAS object holds an entailment problem (as defined in Section 3.3.4, “Additional Types for Textual Entailment”). The result is returned as a list of TEDecision objects. The interface is not an asynchronous interface: it will block until it process all the given problem list, and return the results only after all processing is done.

The expected behavior of implementation of this interface is to process the given dataset concurrently with a number of threads, using general Java concurrent capabilities like Executors and thread pools. The number of threads and other configurations should be defined in the common configuration. Proper initialization (for example, initializing multiple instances for non thread-safe components) should be ensured by the implementation so that the user does not need to care about sub-components and how they are being run.

This interface can be directly implemented by an EDA (i.e., a class that already implements EDABasic and/or other interfaces). It can be also implemented by a concurrent running wrapper (a "runner") for the EDA, if the implementation of this interface does not lend itself naturally to the internal structure of the EDA.

The WP3 members have agreed that we will approach this issue with incremental steps. We can imagine a set of future improvements including:

Additional concurrent processing supports will be realized as the platform matures.

Recall that a central goal of the EXCITEMENT platform is to provide a testing ground for various pluggable components that decide textual entailment. This is done by providing a set of common interfaces, and common data representations. Another requirement is to provide multilingual support.

From this perspective, compatibility check becomes a necessity, since components do not know in what context they are being called. Each component should provide a minimum level of compatibility check, on its configurations as well as on its inputs. This subsection describes recommended policy of metadata checking for the EDA, the core components, and between the EDA and the core components.

Compatibility checks can be classified into two groups. One is checks at startup time (configuration setting and initialization time). The other is checks at data processing time. The general guideline is that EDAs should do both types of checks, and components should at least do the startup time check.

An EDA must perform a compatibility check at its initialization time. This is a two-step process. First, it should check the provided configuration and check that the provided configuration is compatible with the EDA. For example, the language to be processed is supported, and parameters of the EDA is valid, etc. Second, the EDA must initialize its direct components (components that will be directly called by the EDA). Each component, in turn, will do their own checks, using the information from the common configuration file (or equivalent arguments). Likewise, if a component uses a sub-component, it will also initialize the sub-component.

Thus, two types of metadata check failure can happen at the EDA initialization time. First, the EDA itself can report an exception, like inability to process a certain language, inability to provide a mode (like multi-text entailment), missing parameters, etc. The sub-components of the EDA can report exceptions as well. In this case, the EDA must hand through the exception to the user code. Thus, the successful completion of the EDA initialization must be interpretable to mean that all sub components are successfully initialized and are ready to process input.

An EDA must also perform a compatibility check at the data processing time, when its process method is called. The input to EDA process() is a CAS (as JCas object). EDA must check the validity of the input. At least two things should be checked, namely its language (meta data at document annotation type of CAS), and the existence of the analysis layers required by the components (like POS tagging, or dependency parsing, etc).

Components do not need to do additional checks at the process time. Once a component is properly configured and initialized with a configuration, it is the EDA's responsibility to use the component properly within its capabilities. However, a component may choose to provide meta data checking at processing time.

Some components may need the ability to reconfigure. For example, a user of a distance calculation component may wants to compute 10 distance values, by using 10 different "weighting schemes" using the same underlying distance calculation component. One way of doing this would be to generate ten instances of the same component. However, this might not be desirable (e.g. if the component does not permit more than a single instance, or if the initialization of the component takes a lot of time).

The interface Reconfigurable provides a way to modify the component configuration at runtime. The term "reconfiguration" here means changing some of the configurable parameters of a component, without re-initializing all resources, or changing other settings that may remain constant.

Any entailment core component that supports reconfiguration should implement this capability though this interface. This also includes EDAs. If EDA needs to support online reconfiguration, it should support the capability with this interface. Note that when an EDA provides this interface, the EDA should also update the states of its sub-components according to the configuration method argument. It is the responsibility of EDA implementers to ensure that configuration changes are handed through to sub-components, if applicable. This may mean simply reconfiguring some sub-components (which support this) and re-initializing others (which do not).

The central configuration scheme assumes that each component knows its own name, and it can retrieve corresponding configuration data from common configuration by querying the configuration with its name.

Component name is a String value that is designed to be read and written by users. This name must be able to uniquely identify a component or EDA within the EXCITEMENT framework. This name will be used in the configuration files to denote a section that describes the configuration for the component. All entailment core components and EDAs must have this component name string. Conversely, for each component that is initialized there must be a corresponding subsection in the configuration file. The components share a common way of access the names (String getComponentName). To see the use case of names within configuration interface, see Section 5.1, “Common Configuration ”.

Instance name is a String value that is needed for components that permits multiple instances with different configurations. This name will be used in the configuration files to denote a subsection that describes the configuration for the instance. Conversely, for each instance that is initialized there must be a corresponding subsection in the configuration file. The components share a common way of access the names (String getInstanceName). Note that this string can be null if and only if the component does not permit multiple instances with different configurations.

The specification do not describe how the name are stored internally in the component implementations. It is expected that the platform implementation efforts will come up with a common best practice to keep the names in component implementations.

In order to make an easy combination of EDAs and components possible, the configurations of all entailment core components should be stored and accessed in a uniform manner. A framework for the EXCITEMENT common configuration must meet the following requirements:

This specification assumes that the platform shares a single implementation of configuration access. We start with a set of design decisions that follow from the requirements above.

Figure 12, “Overview of Common Configuration” shows a conceptual overview of common configuration object. On the left hand side, the common configuration object is shown as a big table with hierarchy. It has two depths, sections and subsections. Each section is corresponding to a "region" for a specific entailment component. Actual configurations values are represented as lists of "(Name,Value)" pairs. A set of name-value pairs are called as a name-value table in this section.

Figure 12. Overview of Common Configuration

On the right hand side of the figure, the access of configuration values are described on a conceptual level. A component can access an instance of the common configuration object with getSection() or getSubSection() methods for components and their instances, respectively. A successful call returns a name-value table. Each value associated with a name can be accessed by a get method.

Each section and subsection is represented as a name-value table. Once retrieved as a table, there is no differences between subsection and section. Note that one cannot access subsections of a section from a name-value table. A name-value table only holds the list of name-value pairs. The table is a typed key-value list. Thus, a value (which was written in an XML file as a string) can be accessed with getString(name), getInteger(name), or getDouble(name). The XML strings will be converted automatically into the requested type.

As an in-memory object, all name-value pairs are simply seen as a pair of a name (string) and an associated value. However, in XML file level, they can actually come from variables defined in the XML file (like, <property name="something">&variable</property>). Also, the configuration file should easily encode current system environment values in the XML configuration file (For example, it should permit expressions that uses $PATH variable, like <property name="mypath">{$PATH}/etc</property>). Many configuration management libraries already support such features. (Note. The above pseudo-XML code is given only as an example. The specification does not assume anything about how the name-value pairs are represented in XML.)

For all entailment core components, the configuration is delivered as a whole (single big block contains all pairs) via initialization(). It is the responsibility of the initialization (thus of its implementers) to retrieve and use the relevant sections (by using getSection or getSubSection).

Instance configuration areas (SubSections) are only needed for the components that support multiple instances with different configurations. If all instances of a component have the same configuration, the component do not use instance configuration and uses only the component configuration. For example, knowledge components generally have no need to have different configuration even if they have multiple instances.

However, some components must have multiple instances with different configurations. Let's assume there is a distance calculation component that is shipped with two parameters. One is a path to a static resource (say, a dictionary), and the other is a numeric parameter alpha. Assume that the EDA needs to run two instances: one with alpha as 1.0, the other with alpha as 2.0. By having instance-wise configurations, the EDA can run two instances of the distance calculation component, each with different configuration. In this example, parameter path will stay in the component section, and parameter alpha will be written in the instance subsection.

Note that there is no automatic inheritance of section-level information at the subsection level. That is, component instances will typically have to query both the section level for component-wide configuration and the subsection level for their own instance configuration. (In the above example, parameter path is not visible to the name-value table of the instance subsection.)

This section first outlines the adopted XML configuration format with an example. Then, it iterates a few issues around configuration names and GLOBAL section (PlatformConfiguration).

Let's first see an example file that follows the proposed file format. It has a few sections (components) within it. Each name-values are represented with a property element. The element has one attribute (name), and actual value is written as element value within the property element.

 
<?xml version="1.0" encoding="utf-8"?> 
<!DOCTYPE configuration [
<!ENTITY myVar "Some common #PCDATA that can be reused... "> 
]> 

<configuration>
<section name="PlatformConfiguration">
<property name="activatedEDA">core.MyEDA</property> 
<property name="language">EN</property> 
</section>

<section name="PhoneticDistanceComponent">
<property name="KlingonDictionaryPath">$RESOURCE/VD/KDict.cvs</property>
<property name="beta">0.1</property> 

<subsection name="instance1">
<property name="consonantScore">1.0</property> 
<property name="vowelScore">0.6</property>
<property name="alpha">0.17663311</property> 
</subsection> 

<subsection name="instance2">
<property name="consonantScore">0.6</property> 
<property name="vowelScore">1.0</property>
<property name="alpha">0.17663311</property> 
</subsection> 

</section> 

<section name="core.MyEDA"> 
<property name="myLongKey">&myVar;</property> 
<property name="someOption">PhoneticDistanceComponent, 
                       EditDistanceComponent</property> 
</section> 

</configuration> 

Note that,

If any violation is observed (including those and possible others), CommonConfig must raise an exception while loading the file.

A small note on configuration value names and PlatformConfiguration (Global) region:

This section defines is a minimalist approach to configuration management. We leave possible extensions (like default values or possible ranges) to future versions.

The responsibility for component selection -- i.e., the decision which components to activate for a given run of the engine -- could be assigned to different layers: to the user level code, to the initialization helper, or to the top-level EDA. We make it a responsibility of the EDA, in order to allow EDAs to "hide" some complexity to its users.

Consequently, the selection which components to use forms part of the configuration of the EDA and is specified in the EDA section and/or subsection. We expect EDAs to specify how the set of components to be activated is specified as a configuration option. (Note that currently the configuration does not lists). It is the responsibility of the EDA, not of the user level code, to initialize these components.

This section describes the raw input data. By "raw input data" we mean text-hypothesis pairs without linguistic analysis layers. The RTE dataset formats (RTE1-5) are the most widely used file format for this purpose.

The internal EXCITEMENT entailment problem representation is the CAS representation described in Section 3. The raw input data formats only serve as as basis from which the CAS representations is created. Each linguistic analysis pipeline (LAP) must provide readers for the RTE dataset formats that produce CAS objects from corresponding files. All RTE formats to be supported are listed in Appendix D, Supported Raw Input Formats.

NB. RTE1-5 only support the encoding of "classical" (single H, single T) entailment problems. That is, LAPs are only required to deal with classical entailment problems. Support for multi H-T or T-multi H problems is optional. Each LAP should specify clearly in its documentation which additional problem types it supports and what file format it expects or these problems (such as RTE-6 or later).

The following section describes the RTE data formats, and adds a simple modification.

The RTE challenge data formats are the most well known and widely used data format among textual entailment community. RTE-1 to RTE-5 formats are focused on single T-H pairs.

One problem of RTE format is the lack of language marking. Language designation is important in the EXCITEMENT platform. Thus we adopted a tiny modification to RTE format. The following is the DTD of RTE-5 main task data representation.

<!ELEMENT entailment-corpus (pair+)>
<!ELEMENT pair (t,h)>
<!ATTLIST pair
         id CDATA #REQUIRED
         entailment (ENTAILMENT|CONTRADICTION|UNKNOWN) #REQUIRED
         task (IR|IE|QA) #REQUIRED >
<!ELEMENT t (#PCDATA)>
<!ELEMENT h (#PCDATA)>

It defines an XML format that holds top element entailment-corpus. The element can have one or more pair elements, which have t and h as string data. A pair must have a set of attributes. Here they are id (identifier of the pair, as string), entailment (the entailment decision), and task (as one of information retrieval, information extraction, or question answering).

We modify the DTD as follows:

<!ELEMENT entailment-corpus (pair+)>
<!ATTLIST entailment-corpus
          lang CDATA #IMPLIED
          channel CDATA #IMPLIED>
<!ELEMENT pair (t,h)>
<!ATTLIST pair
          id CDATA #REQUIRED
          entailment (ENTAILMENT|NONENTAILMENT|CONTRADICTION|
                          PARAPHRASE|UNKNOWN) #REQUIRED
          task (IR|IE|QA|SUM) #IMPLIED >
<!ELEMENT t (#PCDATA)>
<!ELEMENT h (#PCDATA)>

Now the top element entailment-corpus has an optional attribute lang and channel. This attribute holds the language identification string according to ISO 639-1 [ISO 639-1]. For example, "EN" for English, "DE" for German, and "IT" for Italian. Since it is an optional (designated as #IMPLIED) attribute, existing RTE-5 XML data will be processed as a valid XML for this DTD. In that case, no language checking will take place. This is the same for channel value. The value records the input channel (source of the input like e-mail, transcribed phone call, etc), and it is an optional value. Also, the possible values of entailment relationship are extended to match those of Section 4.2.1.6, “enum DecisionLabel”.

Support for this xtended RTE-5 format is mandatory. EDA implementers must support processing of the format. Other RTE formats are optional. EDAs (and their LAPs) may support them.

At the current stage (non-decomposed pipelines), each LAP will have to implement its own readers. (It is permissible to have one reader that can deal with all RTE1-5 formats, given the large similarities between the formats.) In the future, we foresee that RTE1-5 readers will be provided as a "collection reader" component of UIMA component, and it is expected that such components will be implemented only once and shared among all EDAs and EDA implementers.

This method gets two strings as input and generates a CAS for an EDA. The resulting CAS must be a well-formed CAS that contains all types can are necessary for a complete description of the entailment problem (i.e. all types from the EXCITEMENT.entailment namespace), as defined in Section 3.3.4, “Additional Types for Textual Entailment”.

The amount of linguistic annotation can differ among implementations (say, EnglishTokenizationOnly(t,h) only returns token with TE annotations, while GermanParsingAndNER(t,h) returns tagging, parsing, POS and NER, etc).

This method gets two File as arguments. First argument is a single input file that follows the raw input XML format (as defined in Section 5.2, “Input file format ”), and the second argument outputDir is the output directory. The method returns nothing. Analysis results are stored as serialized CAS files in the directory. (standard XMI serialization of UIMA CAS).

Again, the generated CASes should be well-formed and must contain all types can are necessary for a complete description of the entailment problem (i.e. all types from the EXCITEMENT.entailment namespace).

The interface is a generic interface that can be called for adding linguistic annotations for any data. addAnnotattionOn method does not produce CASes that are ready for EDAs. It only adds generic language analysis results to the provided CAS. This method provides users with the ability to analyze some text data that is not directly a TE pair.

The method's argument is not a string or text, but a JCas. The user must have some knowledge on CAS to use this interface. For example, they needs to know how to generate a new CAS and add text data to it, before calling this method. This design is intentional. The user needs to know about CAS data access methods beforehand, to open up and manually access the analysis results provided by a pipeline. Moreover, all entailment core interfaces that accept CAS as an argument accept only the EDA-compatible CASes. That is, a specific form and type of CASes that can be consumed by EDAs and components (CASes with EXCITEMENT.entailment.* types, and with TextView and HypothesisViews).

Thus, the addAnnotationOn interface (unlike generateSingleTHPairCAS or processRawInputFormat) produces "generic CASes", which can be non-EDA-compatible CASes. The resulting CASes of addAnnotationOn are meant to be used mainly by the caller himself, when they require to perform linguistic analysis on additional data of some kind. For example, the Transduction layer needs to use this functionality to determine which sentence of a customer e-mail can be regarded as text or hypothesis, respectively.

Note that addAnnotationOn exists in two versions: one that applies to the single view CAS and one that applies just a specific View within the CAS. The pipeline implementer must provide both of them; we recommend to implement addAnnotationOn(JCas) simply as a call of addAnnotationOn(JCas,"_InitialView") (single view CAS is a CAS with one view, that is implicitly named as "_InivitalView").

addAnnotationOn should not add any type that belongs to EXCITEMENT.entailment.* types (types related to text, hypothesis, pair and entailment metadata).

An LAPAccess implementation must provide all three sets of methods defined in the interface.

Write modular modules.

EXCITEMENT platform will adopt some Java code annotations, starting with the followings.

Actual definitions of the Java annotations and their usage will be provided by the implementation team (WP4). Also, the list of the annotations will be expanded along with the implementation effort.