On the Lossless Transformation of Single-File, Multi-Layer Annotations into Multi-Rooted Trees

Introduction

Due to the complexity of this type of textual data, the annotation of linguistic corpora can be seen as a benchmark test for markup languages. In recent years 1 , linguistic analyses became more and more detailed and linguists incorporated the results of these analyses into corpora. Furthermore, linguistic descriptions applied to texts are extremely heterogeneous. This is especially evident if the research belongs to different fields of linguistics, e.g., syntax, semantics, and phonology.

This contribution touches upon several different topics with regard to using XML-based markup languages for linguistic research. Our overall goal is the sustainable archiving of linguistic data. Corpora usually contain multiple annotation layers (morphology, part-of-speech, syntax, semantics, information structure, etc.). We devised a generalised architecture that, among other aspects, requires individual conceptual annotation layers contained in linguistic corpora to be separated from one another. To meet this prerequisite, we have to transform a linguistic corpus (normally represented by a single XML file, i.e., a single-rooted tree) into several XML files (i.e., a multi-rooted tree) so that each file contains one specific annotation layer. After a short introduction to the Generalised Architecture for Sustainability of linguistic data (GENAU; section 2), sections 3 and 4 describe two tools for the purpose of transforming different types of linguistic corpora into multi-rooted trees. While the separation of individual annotation layers can be considered an important and necessary step for the sustainable archiving of linguistic corpora, this process does remove potentially important element nesting information. For a transformation from single- to multi-rooted trees that is 100% lossless, we need to incorporate a mechanism that enables us to store information with regard to element nesting. The approach we introduce in this article is based on XCONCUR, which we briefly describe in section 5.

This contribution reports on work in progress within the project "Sustainability of Linguistic Data", funded by the German Research Foundation (DFG). It is an update to our Extreme Markup Languages 2006 Late Breaking paper [Wörner et al. (2006)]

Generalised Architecture for Sustainability (GENAU)

Since the late 1990s, practically all annotation formats for linguistic corpora have been realised as XML-based markup languages (see [Sperberg-McQueen & Burnard (1994)] , [Lehmberg & Wörner (to appear)] , [Wagner (2005)] for examples). They usually come in two different flavours – traditionally and in accordance with the built-in XML data model, most corpus markup languages form hierarchies that are expressed by nested element trees (for example, for the representation of syntactic constituents or document structures that are in practically all cases modelled based on the OHCO paradigm, i.e., text is supposed to be an ordered hierarchy of content objects, see [Renear et al. (1993)] ).

In stark contrast to hierarchical data formats are markup languages that anchor a data set to a timeline (primarily used for the transcription of spoken language) – this approach borrows heavily from Bird and Liberman's Annotation Graphs [Bird & Liberman (2001)] . In timeline-based formats such as EXMARaLDA [Schmidt (2001)] , the annotator can draw an arc from one anchor to another point on the timeline. However, these upper-level structures are not represented by nested XML element-trees, but with the help of corresponding attribute-value pairs. At the same time and regardless of the hierarchical or timeline-based model, both approaches usually encode several annotation layers concurrently: a certain set of XML elements or attributes represents information on morphology, another set encapsulates syntactic information, while other elements encode linguistic data related to semantic or pragmatic structures. In our sustainability project we have to deal with both hierarchical and timeline-based corpora (examples can be found in [Wörner et al. (2006)] ) and we have to provide the means for enabling users to query both types of resources in a uniform way [Rehm et al. (2007)] . In fact, the original annotation format will be irrelevant to the user, as the user interface and the underlying technology will abstract from any idiosyncrasies and peculiarities of the original corpus data formats.

We devised an approach called GENAU (Generalised Architecture for Sustainability) that is able to cope with the abovementioned difficulties ( [Dipper et al. (2006)] , [Schmidt et al. (2006)] , [Wörner et al. (2006)] ) and that can be compared to the NITE Object Model [Carletta et al. (2003)] . The system architecture and our corpus processing workflow are depicted in figure 1. First, a corpus to be imported into our web-based corpus platform has to be analysed manually. Depending on the XML markup used in annotating the respective corpus, the XML document instance is transformed into multi-rooted trees. Some corpora can be transformed using simple XSLT stylesheets, while other corpora have to be processed using a custom set of tools (Tool₁, Tool₂ etc. in figure 1) with regard to this initial processing stage.

Corpora annotated based on the hierarchical model are analysed by a tool that enables us to map XML elements, attributes and textual content onto one or more annotation as well as primary or secondary data layers (see section 3). As soon as this mapping exists, the n annotation layers identified in the analysis can be exported as n XML document instances. In other words, this tool semi-automatically splits hierarchically annotated corpora that typically consist of a single XML document instance, into individual XML files, so that each file represents all the information related to a single linguistic annotation layer. Furthermore, this approach guarantees that overlapping structures – a notorious problem using single XML documents and, hence, single element trees – can be represented in a straightforward way [Witt (2004)] .

Timeline-based corpora (for example, EXMARaLDA corpora) are split using another tool in order to separate the graph annotations that are also stored in individual XML files (see section 4). Our approach enables us to represent arbitrary types of XML-annotated corpora as individual files that can be conceptualised as individual XML element trees. In a way, these multi-rooted trees are represented as regular XML document instances, but, since a single corpus is comprised of multiple files, there is a need to go beyond the functionality offered by typical XML tools in order to enable us to process multiple files, as regular tools work with single files only.

Finally, these single XML files are imported into a native XML database; currently we use the open source database eXist (see the right hand side in figure 1). A third tool anchors all files to a set of primary data in order to allow query-time coordination between the individual files that represent a single-rooted tree each ( [Eckart & Teich (2007)] , [Rehm et al. (2007)] ).

Figure 1: The two main phases of our corpus processing workflow

[Link to open this graphic in a separate page]

At the same time, the elements and attributes used in the markup languages are analysed and incorporated into an ontology, represented in OWL (Web Ontology Language), that encapsulates knowledge about linguistic terms and concepts. The ontology is used to generalise over the specific and, at times, idiosyncratic names and labels used in the corpus annotation markup languages and to provide a coherent, unified, and homogeneous perspective on the large set of heterogeneous corpora (see the left hand side in figure 1 as well as [Chiarcos (2007)] , [Rehm et al. (2007)] ). The OWL ontology is the main resource within our query interface: users can, for example, search for different combinations of part-of-speech tags. Usually, different corpora use different element and attribute names for encoding part-of-speech information, but the homogenising ontology of linguistic terms and concepts enables us to automatically expand a given query into all matching and appropriate element and attribute names. [Rehm et al. (2007)] describe this approach in detail.

Transforming Single Rooted Trees

Leveler Pipeline

As shown in figure 2 there are two main steps to the Leveler process, each corresponding to one of the goals stated above. The original corpus (C₀) is used as input into Leveler to generate the Step 1 configuration file which contains directives on converting annotation information stored as PCDATA to attribute values. An XSL transformation then takes C₀ and the configuration file as input to generate a text-transformed corpus file, C₁. At this point, if one extracts the PCDATA from C₁, one should just get the straight text of the corpus without annotation information.

C₁ is itself used as input to another part of the Leveler tool to generate the Step 2 configuration file which specifies the different layers of annotation found in the corpus as well as which XML elements belong to them. Another XML transformation takes this configuration file and C₁ to generate C₂, a layer-annotated, text-transformed corpus. Layer information is simply added to the annotation elements as an attribute, and as such, the straight text of the corpus is preserved.

C₂ can then be fed into a final XSL transformation which creates an XML document for each layer, each containing only those nodes which belong to that layer. Each of these files will contain the same PCDATA information, although the branching of the element tree structure may be different. Leveler uses its own dedicated namespace for automatically-inserted elements.

Figure 2: The Leveler pipeline

[Link to open this graphic in a separate page]

The Leveler Web Application

The bulk of the work is done with a web-based tool, Leveler, which takes user input in combination with an XML document to generate configuration files for use in the XSL transformations. Leveler is written in PHP with the SimpleXML extension and presents the user with a series of web forms.

Figure 3: Leveler main screen

[Link to open this graphic in a separate page]

Figure 3 shows the main Leveler web form. The first fieldset, "Basic Information", is used for both Step 1 and Step 2. Here the user specifies the XML document filename and can optionally add Comment, Operator, and Project metadata to the resulting configuration file. To advance to Step 1, the user clicks the "Submit Step 1" button, uploading the XML file for server-side processing.

Leveler: Step 1

In the first step (figure 4), Leveler parses C₀ and returns a table that contains a row for each element in the XML document that has textual information (i.e., contains PCDATA); a list of elements that do not contain text nodes is listed at the bottom of the page for reference. For each element, the user selects one of three transformations by clicking on a radio button (the ∀ link at the top of each column is a shortcut that sets all elements to that transformation type). Upon submission of this form, Leveler generates an XML configuration file which contains a list of elements and how PCDATA within them should be handled.

Figure 4: Step 1 Leveler web form

[Link to open this graphic in a separate page]

The three text transformations are detailed in the following table with examples that show the transformation as applied to the following corpus XML fragment which contains words in orth elements and their parts of speech in pos elements:

Figure 5: Sample XML fragment

<tok id="s119n0">
    <orth>Wir</orth>
    <pos func="HD">PPER</pos>
</tok>
<tok id="s119n1">
    <orth>müssen</orth>
    <pos func="HD">VMFIN</pos>
</tok>
<tok id="s119n2">
    <orth>uns</orth>
    <pos func="HD">PRF</pos>
</tok>
<tok id="s119n3">
    <orth>selbst</orth>
    <pos func="HD">ADV</pos>
</tok>
<tok id="s119n4">
    <orth>helfen</orth>
    <pos func="HD">VVINF</pos>
</tok>
<punc>.</punc>

Types of Text Transformations (Step 1)

Real PCDATA	Identity transformation - all PCDATA nodes remain PCDATA. Note that in the example output the part of speech tags remain PCDATA nodes; extraction of text nodes results in the intermingling of POS labels: "Wir PPER müssen VMFIN uns PRF selbst ADV helfen VVINF ." <tok id="s119n0"> <orth>Wir</orth> <pos func="HD">PPER</pos> </tok> <tok id="s119n1"> <orth>müssen</orth> <pos func="HD">VMFIN</pos> </tok> <tok id="s119n2"> <orth>uns</orth> <pos func="HD">PRF</pos> </tok> <tok id="s119n3"> <orth>selbst</orth> <pos func="HD">ADV</pos> </tok> <tok id="s119n4"> <orth>helfen</orth> <pos func="HD">VVINF</pos> </tok> <punc>.</punc>
Annotation	Text data is converted to an introduced attribute text (in the Leveler namespace) of the containing element. Example: pos set to "Annotation", the rest set to "Real PCDATA". This results in a fragment where the PCDATA is the pure corpus text: "Wir müssen uns selbst helfen .". <tok id="s119n0"> <orth>Wir</orth> <pos func="HD" leveler:text="PPER" /> </tok> <tok id="s119n1"> <orth>müssen</orth> <pos func="HD" leveler:text="VMFIN" /> </tok> <tok id="s119n2"> <orth>uns</orth> <pos func="HD" leveler:text="PRF" /> </tok> <tok id="s119n3"> <orth>selbst</orth> <pos func="HD" leveler:text="ADV" /> </tok> <tok id="s119n4"> <orth>helfen</orth> <pos func="HD" leveler:text="VVINF" /> </tok> <punc>.</punc>
Mixed	Text data remains as PCDATA of the element but is also duplicated as an introduced attribute text (in the Leveler namespace). Example: pos set to "Annotation", punc set to "Mixed", the rest set to "Real PCDATA". This also results in a fragment where the PCDATA is the pure corpus text: "Wir müssen uns selbst helfen .". <tok id="s119n0"> <orth>Wir</orth> <pos func="HD" leveler:text="PPER" /> </tok> <tok id="s119n1"> <orth>müssen</orth> <pos func="HD" leveler:text="VMFIN" /> </tok> <tok id="s119n2"> <orth>uns</orth> <pos func="HD" leveler:text="PRF" /> </tok> <tok id="s119n3"> <orth>selbst</orth> <pos func="HD" leveler:text="ADV" /> </tok> <tok id="s119n4"> <orth>helfen</orth> <pos func="HD" leveler:text="VVINF" /> </tok> <punc leveler:text=".">.</punc>

Leveler: Step 2

In the second step, users enter a list of up to 8 annotation layers (e.g., "morph", "sem", "syn") that are found in the corpus before uploading the XML document (C₁) for analysis (figure 3). After the analysis of the document the user is again presented with a table where each row is a (unique) element as in figure 6. For each element the user can then specify which annotation layer(s) it belongs to (with shortcuts to assign all elements to or remove all elements from a particular layer). Like Step 1, submission of this form results in an XML configuration file which contains a list of layer names and a list of element names with layer membership information (IDREF to a layer ID).

Figure 6: Step 2 Leveler web form

[Link to open this graphic in a separate page]

Because of the number of files and steps involved, a Java utility called LevelRunner provides a simple graphical user interface to running the XSL transformations. Users select which step they are on, then use file selectors to choose a source XML file, a configuration file (i.e., output from the web application), and an output directory. Clicking on a "Run" button shells out to Saxon to run the appropriate transformation. (Note that the actual command used is configurable so users are not locked into a particular processor.)

Transforming Annotation Graphs

Annotation Graphs (AGs, [Bird & Liberman (2001)] ) are used for the representation of multi-layered linguistic annotations. They allow for the declaration of an unlimited number of annotation layers and for an unrestricted linking of points in the text and in the annotation. The formal framework of Annotation Graphs is instantiated in different annotation tools and annotation models. A prominent application of AGs is EXMARaLDA. [Schmidt (2001)] . This section describes the transformation of EXMARaLDA into GENAU.

We developed Splitter, an XSLT-based tool that operates on documents in the time-based EXMARaLDA basic-transcription format and that converts them into a hierarchy-based format called split-transcription. The XML-based EXMARaLDA basic-transcription format uses a time-based data model, the "single timeline, multiple tiers" (STMT) data model ( [Schmidt (2005)] ). Information about speakers' verbal and nonverbal actions, other events, and various annotations referring to these actions and events is contained in several tiers. Each tier is identified by two attributes: The category of data it contains – orthographic transcription of verbal actions vs. phonetic transcription of verbal actions vs. description of nonverbal actions vs. annotation of manner of articulation of actions, and so on – and the speaker whose actions this particular tier contains; tiers of speakerless categories – e.g. for description of noises heard in the background – lack the speaker attribute, of course. Thus, actions of any two different speakers participating in the discourse are physically removed from each other in an EXMARaLDA file by being stored on different tiers, even if their category is the same. Also, actions of the same speaker but of different categories – e.g. Mary saying "I do not know" and shrugging at the same time – don't appear proximate in an EXMARaLDA file. Not even, e.g., a phonetic transcription appears in the vicinity of its orthographic counterpart. Rather, the temporal relation between events as well as the relations between annotations and physical events are expressed using a common timeline, to which all events from all tiers refer. Consider the following example EXMARaLDA basic-transcription (contents taken from [Schmidt (2005)] ):

<?xml version="1.0" encoding="UTF-8"?>
<basic-transcription>
  <head>
    <meta-information>
      <!-- ... -->
    </meta-information>
    <speakertable>
      <speaker id="DS"/>
      <speaker id="FB"/>
    </speakertable>
  </head>
  <basic-body>
    <common-timeline>
      <tli id="T0" time="0.0"/>
      <tli id="T1"/>
      <tli id="T2"/>
      <tli id="T3"/>
      <tli id="T4"/>
      <tli id="T5"/>
    </common-timeline>
    <tier id="TIE0" speaker="DS" category="sup" type="a">
      <event start="T1" end="T3">faster</event>
    </tier>
    <tier id="TIE1" speaker="DS" category="v" type="t">
      <event start="T0" end="T1">Okay.</event>
      <event start="T1" end="T2">Très bien,</event>
      <event start="T2" end="T3">très bien.</event>
    </tier>
    <tier id="TIE2" speaker="DS" category="en" type="a">
      <event start="T0" end="T1">Okay.</event>
      <event start="T1" end="T3">Very good, very good.</event>
    </tier>
    <tier id="TIE3" speaker="DS" category="nv" type="d">
      <event start="T2" end="T4">right hand raised</event>
    </tier>
    <tier id="TIE4" speaker="FB" category="v" type="t">
      <event start="T2" end="T3">Alors ça</event>
      <event start="T3" end="T4">dépend ((cough))</event>
      <event start="T4" end="T5">un petit peu.</event>
    </tier>
    <tier id="TIE5" speaker="FB" category="en" type="a">
      <event start="T3" end="T5">That depends, then, a little bit.</event>
    </tier>
    <tier id="TIE6" speaker="FB" category="pho" type="a">
      <event start="T4" end="T5">[ɛ̃tipø:]</event>
    </tier>
  </basic-body>
</basic-transcription>

Figure 7: Transcript in the "musical score" notation

[Link to open this graphic in a separate page]

The purpose of Splitter is to transform this time-based format into a hierarchy-based format, where the relation between an event and the annotation describing it is not principally one of identical position on a timeline but a dominance relation in an ordered hierarchy. Since in the basic transcription, everything from "tape crackles" to the phonetic transcription of "un petit peu" ([ɛ̃tipø:]) is expressed as event elements, a distinction between "events" in the narrow sense and "annotations" describing them has to be introduced first. The user specifies a parameter called skeletonCategory. The events on tiers of this category are to become the leaves of a multi-rooted tree Splitter produces. Here one would probably choose v, the orthographic category. They will be treated as "events" in the narrow sense, everything else being "annotation".

In the initial step, Splitter takes all the events from all the tiers of the skeleton category and puts them into a single, chronologically-ordered sequence. The physical separation of utterances of different speakers is thereby removed. To keep speakers identifiable, the respective speaker ID is added as an attribute to each event. Temporally adjacent events belonging to the same speaker are treated as a coherent utterance and kept adjacent in the transcription, even if there are events belonging to other speakers during the utterance. As opposed to the score-like visualisation, the XML representation now resembles a script and can intuitively be read line by line:

<event id="1" speaker="DS" start="T0" end="T1">Okay.</event> 
<event id="2" speaker="DS" start="T1" end="T2">Très bien,</event>
<event id="3" speaker="DS" start="T2" end="T3">très bien.</event> 
<event id="4" trans="sync" ref="3" speaker="FB" start="T2" end="T3">Alors ça</event>
<event id="5" speaker="FB" start="T3" end="T4">dépend ((cough))</event>
<event id="6" speaker="FB" start="T4" end="T5">un petit peu.</event>

• overlap: Classic overlap: Y starts during X and continues beyond the end of X.
• sync: X and Y share their start points and their end points (see example).
• sync-shorter: X and Y start at the same time, but Y ends before X.
• sync-longer: X and Y start at the same time, but Y ends after X.
• within: All Y takes place during X. X starts earlier and ends later or at the same time.

<sup> <event id="1" speaker="DS" start="T0"
        end="T1">Okay.</event>
</sup>
<sup value="faster" start="T1" end="T3">
  <event id="2" speaker="DS" start="T1" end="T2">Très bien,</event>
  <event id="3" speaker="DS" start="T2" end="T3">très bien.</event>
</sup>
<sup>
  <event id="4" trans="sync" ref="3" speaker="FB" start="T2" end="T3">Alors ça</event>
  <event id="5" speaker="FB" start="T3" end="T4">dépend ((cough))</event>
  <event id="6" speaker="FB" start="T4" end="T5">un petit peu.</event>
</sup>

Simply continuing like this and adding elements of other categories would go awry in the general case, because annotating events in the source document can overlap freely, whereas XML elements cannot overlap – a strict tree-structure is required. Splitter's solution is to create multiple result documents. Each result document contains annotating elements of just one category – hence the name "Splitter" – but all are completely identical with respect to the sequence of contained events. The set of the result documents can thus be interpreted as a multi-rooted tree, with different annotating nodes descending from different roots, but ultimately dominating a common sequence of leaves.

The result documents (in this case en.xml, nv.xml, pho.xml, sup.xml, and v.xml) are completed by one split-transcription meta-document, essentially the same as the source document, except the tiers are replaced with references to the respective documents:

<?xml version="1.0" encoding="UTF-8"?>
<split-transcription xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns:fn="http://www.w3.org/2005/xpath-functions"
  xmlns:f="http://www.sfb441.uni-tuebingen.de/c2/Kilian/XSLT/Funktionen">
  <head>
    <meta-information><!-- ... --></meta-information>
    <speakertable>
      <speaker id="DS">
        <abbreviation>DS</abbreviation>
      </speaker>
      <speaker id="FB">
        <abbreviation>FB</abbreviation>
      </speaker>
    </speakertable>
  </head>
  <split-body>
    <common-timeline>
      <tli id="T0" time="0.0"/>
      <tli id="T1"/>
      <tli id="T2"/>
      <tli id="T3"/>
      <tli id="T4"/>
      <tli id="T5"/>
    </common-timeline>
    <tier id="sup" category="sup" type="a" href="./split/example/sup.xml"/>
    <tier id="nv" category="nv" type="d" href="./split/example/nv.xml"/>
    <tier id="v" category="v" type="t" href="./split/example/v.xml"/>
    <tier id="en" category="en" type="a" href="./split/example/en.xml"/>
    <tier id="pho" category="pho" type="a" href="./split/example/pho.xml"/>
  </split-body>
</split-transcription>
        

XCONCUR

SGML (see [SGML (1986)] , [Goldfarb (1990)] ) gave authors a facility to create documents with multiple, possibly overlapping, hierarchies in a rather easy way by its CONCUR option. The SGML specification has never been implemented completely, and, as a consequence, there is not a single SGML system we are aware of with full support for CONCUR. XCONCUR (introduced by [Hilbert et al. (2005)] ) provides authors who are familiar with XML a standardized and intuitive way to write documents with overlapping markup. This is achieved by reviving SGML's CONCUR option and by introducing this option into XML.

A set of XML documents with identical primary data ³ can be transformed into XCONCUR, but, unfortunately, there is currently no tool to achieve this in a straightforward way. If, however, the set of XML documents is converted to a Prolog clause database, as defined within the architecture of the Sekimo project (see [Witt et al. (2005)] ), this clause database can be converted to XCONCUR using prolog2xconcur.

<?xconcur version="1.1" encoding="utf-8"?>
<?xconcur-schema layer-id="l1" root="tusneldaCorpus" system="tusnelda.dtd"?>
<?xconcur-schema layer-id="l2" root="tusneldaCorpus" system="tusnelda.dtd"?>
<(l1)tusneldaCorpus version="1.0"><(l2)tusneldaCorpus version="1.0">
  <!-- metadata sections omitted, here just an excerpt of the body -->
  <(l1)body id="SGID0201"><(l2)body id="SGID0201">
    <(l1)div type="unspecified"><(l2)div type="unspecified">
      <(l2)head type="main">
        <(l2)s id="SGID0201.1" n="1">
          <(l1)tok id="SGID0201.1.1" n="1">
            <(l2)tok id="SGID0201.1.1" n="1">
              <(l1)orth>
                <(l2)orth>Kakoj</(l2)orth>
              </(l1)orth>
              <(l1)pos leveler:text="pronoun"/>
              <(l1)desc>
                <(l1)feature type="subpos" leveler:text="interrogative"/>
                <(l1)feature type="tag" leveler:text="pronomen_int_nom_sg_masc_adj"/>
                <(l1)feature type="syntactic type" leveler:text="adjectival"/>
                <(l1)lemma leveler:text="kakoj"/>
                <(l1)case leveler:text="nominative"/>
                <(l1)gender leveler:text="masculine"/>
                <(l1)number leveler:text="singular"/>
              </(l1)desc>
            </(l2)tok>
          </(l1)tok>
          <(l1)tok id="SGID0201.1.2" n="2">
            <(l2)tok id="SGID0201.1.2" n="2">
              <(l1)orth>
                <(l2)orth>socializm</(l2)orth>
              </(l1)orth>
              <(l1)pos leveler:text="noun"/>
              <(l1)desc>
                <(l1)feature type="subpos" leveler:text="common"/>
                <(l1)feature type="tag" leveler:text="substantiv_masc_sg_nom_unb"/>
                <(l1)feature type="animacy" leveler:text="inanimate"/>
                <(l1)lemma leveler:text="socializm"/>
                <(l1)case leveler:text="nominative"/>
                <(l1)gender leveler:text="masculine"/>
                <(l1)number leveler:text="singular"/>
              </(l1)desc>
            </(l2)tok>
          </(l1)tok>
          <(l1)tok id="SGID0201.1.3" n="3">
            <(l2)tok id="SGID0201.1.3" n="3">
              <(l1)orth>
                <(l2)orth>narodu</(l2)orth>
              </(l1)orth>
              <(l1)pos leveler:text="noun"/>
              <(l1)desc>
                <(l1)feature type="subpos" leveler:text="common"/>
                <(l1)feature type="tag" leveler:text="substantiv_masc_sg_gen_gen2_unb"/>
                <(l1)feature type="subcase" leveler:text="gen_2"/>
                <(l1)feature type="animacy" leveler:text="inanimate"/>
                <(l1)lemma leveler:text="narod"/>
                <(l1)case leveler:text="genitive"/>
                <(l1)gender leveler:text="masculine"/>
                <(l1)number leveler:text="singular"/>
              </(l1)desc>
            </(l2)tok>
          </(l1)tok>
          <(l1)tok id="SGID0201.1.4" n="4">
            <(l2)tok id="SGID0201.1.4" n="4">
              <(l1)orth>
                <(l2)orth>nuzhen</(l2)orth>
              </(l1)orth>
              <(l1)pos leveler:text="adjective"/>
              <(l1)desc>
                <(l1)feature type="tag" leveler:text="adjektiv_masc_sg_kurzform"/>
                <(l1)feature type="Adjform" leveler:text="short form"/>
                <(l1)lemma leveler:text="nuzhnyj"/>
                <(l1)gender leveler:text="masculine"/>
                <(l1)number leveler:text="singular"/>
                <(l1)degree leveler:text="positive"/>
              </(l1)desc>
            </(l2)tok>
          </(l1)tok>
          <(l1)tok id="SGID0201.1.5" n="5">
            <(l2)tok id="SGID0201.1.5" n="5">
              <(l1)orth>
                <(l2)orth>?</(l2)orth>
               </(l1)orth>
              <(l1)pos leveler:text="interpunction"/>
              <(l1)desc>
                <(l1)feature type="tag" leveler:text="satzzeichen_fragezeichen"/>
                <(l1)feature type="type" leveler:text="question"/>
                <(l1)lemma leveler:text="?"/>
              </(l1)desc>
            </(l2)tok>
          </(l1)tok>
        </(l2)s>
      </(l2)head>
    <!-- ... -->
    </(l2)div></(l1)div>
  </(l2)body></(l1)body>
  <!-- ... -->
</(l2)tusneldaCorpus></(l1)tusneldaCorpus>

<!-- ... -->
<(s)sentence>
  <!-- ... -->
  <(s)word>
    <(m)morph><(p)syll>ta</(p)syll><(p)syll>ble</(m)morph><(m)morph>s</(p)syll></(m)morph>
  </(s)word>
  <!-- ... -->
</(s)sentence>
<!-- ... -->

# rule 1
($S)word {
  (element[($P)syll] [] element[self]) ||
  (start[self] <= start[($P)syll] && end[($P)syll] <= end[self])
} assert

# rule 2
($S)word {
  (element[($M)morph] [] element[self]) ||
  (start[($M)morph] == start[self] && end[($M)morph] == end[self]) ||
  (start[($M)morph] == start[self] && end[($M)morph] [] element[self]) ||
  (start[($M)morph] [] element[self] && end[($M)morph] == end[self])
} assert

# rule 3
($)word [
  element[($P)syll] // element[($M)morph]
} optional

# rule 4
* {
  element[($P)syll ][ element[($S)word]
} reject

# rule 5
* {
  element[($M)morph ][ element[($S)word]
} reject

  constraint-set ::= comment* rule (rule | comment)*
            rule ::= context "{" expression "}" rule-modifier?
         comment ::= "#" Any character valid for comments
         context ::= "*" | tag-identifier (tag-identifier)*
   rule-modifier ::= "assert" | "reject" | "optional"
      expression ::= basic-expression
                     | expression connector expression
                     | "!" expression
                     | "(" expression ")"
basic-expression ::= operand operator operand
       connector ::= "&&" | "||"
        operator ::= basic-operator | derived-operator
  basic-operator ::= "<<" | "=="
derived-operator ::= ">>" | "<=" | "=>" | "[]" | "][" | "//"
         operand ::= ("start" | "end" | "element")
                     "[" (tag-identifier | "self" ) "]"
  tag-identifier ::= "(" layer-variable ")" element-name
  layer-variable ::= "$" [A-Z] ([A-Z] | [0-9])*
    element-name ::= A QName as defined in Bray et al. (2006b)

GENAU and XCONCUR

The sections “Transforming Single Rooted Trees” and “Transforming Annotation Graphs” demonstrate how different types of linguistic corpora can be transformed to a set of separately annotated, primary-data-identical XML documents. These XML documents serve as the GENAU format. Since the XCONCUR document format can be understood as an interwoven set primary-data-identical XML documents, a transformation from GENAU to XCONCUR can be done trivially and is a lossless operation. Furthermore, an XCONCUR document can again be split up and transformed back via GENAU to its original representation – or even a different one.

The XCONCUR representation allows for the utilisation of XCONCUR-CL to formulate constraint rules, which express the relations between the elements of different trees in the multi-rooted-trees data model. On the one hand, this serves as a mechanism to validate the markup and can improve corpora quality. On the other hand, these constraint rules allow for the formulation of principles which are the result of a deeper analysis.

Figure 12 shows an excerpt of figure 8. The following constraint rules can be applied to ensure that tok elements span over equal ranges:

($L1)body {
  start[($L1)tok] == start[($L2)tok] && end[($L1)tok] == end[($L2)tok]
} assert

($L2)body {
  start[($L1)tok] == start[($L2)tok] && end[($L1)tok] == end[($L2)tok]
} assert

<!-- ... -->
<(l1)tok id="SGID0201.1.1" n="1">
  <(l2)tok id="SGID0201.1.1" n="1">
    <(l1)orth>
      <(l2)orth>Kakoj</(l2)orth>
    </(l1)orth>
    <(l1)pos leveler:text="pronoun"/>
    <(l1)desc>
      <(l1)feature type="subpos" leveler:text="interrogative"/>
      <(l1)feature type="tag" leveler:text="pronomen_int_nom_sg_masc_adj"/>
      <(l1)feature type="syntactic type" leveler:text="adjectival"/>
      <(l1)lemma leveler:text="kakoj"/>
      <(l1)case leveler:text="nominative"/>
      <(l1)gender leveler:text="masculine"/>
      <(l1)number leveler:text="singular"/>
    </(l1)desc>
  </(l2)tok>
</(l1)tok>
<!-- ... -->

Likewise, similar constraints can be applied to an XCONCUR representation of an EXMARaLDA document. An example of an EXMARaLDA document that has been converted to XCONCUR is shown in figure 13. Similar constraints, such as those from the previous example, can be applied. For example, one might want to assert that orth elements should span over the same range as the syll elements.

<?xconcur version="1.1" encoding="utf-8"?>
<?xconcur-schema layer-id="l1" root="body" system="exmeralda1.dtd"?>
<?xconcur-schema layer-id="l2" root="body" system="exmeralda2.dtd"?>
<(l1)body><(l2)body>
  <(l1)orth id="TIE1.a0" s="TLI0" e="TLI1" value="Sí.">
    <(l2)syll id="TIE2.a0" s="TLI0" e="TLI1" value="[CV]">
      <(l1)ts number="1" speaker="CHI" tier="TIE0" n="sc" id="TIE0.sc0" s="TLI0" e="TLI1">
        <(l2)ts number="1" speaker="CHI" tier="TIE0" n="sc" id="TIE0.sc0" s="TLI0" e="TLI1">
          <(l1)ts n="e" id="TIE0.e0" s="TLI0" e="TLI1">
            <(l2)ts n="e" id="TIE0.e0" s="TLI0" e="TLI1">[dʃɪ]</(l2)ts>
          </(l1)ts>
        </(l2)ts>
      </(l1)ts>
    </(l2)syll>
  </(l1)orth>
  <(l1)orth id="TIE1.a1" s="TLI2" e="TLI3" value="Mira.">
    <(l2)syll id="TIE2.a1" s="TLI2" e="TLI3" value="[CV.CV:]">
         <(l1)ts number="2" speaker="CHI" tier="TIE0" n="sc" id="TIE0.sc1" s="TLI2" e="TLI3">
           <(l2)ts number="2" speaker="CHI" tier="TIE0" n="sc" id="TIE0.sc1" s="TLI2" e="TLI3">
              <(l1)ts n="e" id="TIE0.e1" s="TLI2" e="TLI3">
                 <(l2)ts n="e" id="TIE0.e1" s="TLI2" e="TLI3">[mi.ɹɐː]</(l2)ts>
              </(l1)ts>
           </(l2)ts>
         </(l1)ts>
     </(l2)syll>
   </(l1)orth>
   <!-- ... -->
</(l2)body></(l1)body>

Conclusion