Network-Oriented Document Abstraction Language: Structure and Reference for the Rest of the Web

Overview

The Web has created an unprecedented ability to publish and distribute documents and other data and to augment this content with hypertextual references. However, the granularity of these references is, for the most part, limited to the file level. The combination of a need to reuse and/or quote sources smaller than a whole document (e.g. Nelson's transclusion [Nel99] ) and the advent of new technologies built on hypertextual reference (e.g. RDF [RDF98] and the Semantic Web [SemWeb98] has stimulated a desire to provide a means of referencing only parts of a document. In traditional HTML this was accomplished with named anchors, but was thus limited to only those points in a document anticipated by the author. XPointer [XPointer] , XPath [XPath] and XLink [XLink] provide sub-document linking for XML documents but as yet only a small fraction of the Web is available as XML.

Instead of restricting sub-document reference to markup languages, we have designed NODAL, the Network-Oriented Document Abstraction Language, as a simple middleware layer that provides a means of defining URI-referenceable structure for all document formats and other structured data sources. It achieves this by defining a simple data model and schema language that can then be used to define sub-document structure for any document type. An extensible, plugin architecture is used to associate format encoder/decoder pairs with MIME types that in turn identify particular document formats and schemas. This system thus defines a common language for referencing and accessing the structured contents of arbitrary document formats at arbitrary granularity. Moreover, it can be used in conjunction with other sub-document reference schemes without interference.

Finally, the system provides a base for future development and research towards new software architectures and development paradigms that allow for a much more seamless interconnection between data and documents across applications, formats, database structures and distribution protocols.

The Problem: Granular Reference, Content Reuse and Annotation

This kind of scenario is familiar to most hypertext researchers and some of these issues are well described by Ted Nelson in a recent paper [Nel99] summarizing his goals and motivations of the past 40 years and a prescription for “Xanalogical structure”. Unfortunately, like many hypertext systems of the past (e.g. Microcosm [Mcsm90] and Intermedia [Imed88] ), he suggests that the solution lies in an incompatible, new system and data model that will only act as its own kind of information silo. Instead we suggest that the only way that this can be made to work (and have any possibility of wide uptake) is if we strive to develop a system that interoperates seamlessly with existing environments and applications (e.g. as in Chimera [Chim00] ).

In a separate paper [DKC05] , we have analyzed this general class of issues and various kinds of "information silos" (e.g. applications, data formats, operating systems, database systems). We suggested that a change to the assumed application architecture would allow for the deep linking, content reuse and general annotation facilities necessary to build general personal (and group) information management environments. This application architecture (the DKC Model) contains a persistent storage layer at its base (the Data Layer) that forms the basis for sharing, linking, reuse and versioning of data and structure. A Knowledge layer that can be used to imbue this data with semantic meaning, and then a final, independent layer of Context that manages user interface, interaction and modelling. By advocating the independence of view and storage (in the Context and Data layers), we suggest that this model will not only allow personal and group information management to finally begin to deal with cross-platform, cross-application issues but that we will enable greater innovation and adaptation in the user- and task-oriented spaces often considered by HCI and Hypertext researchers.

The NODAL Solution: Document Data Modeling and Reference

The approach we advocate is to separate data structure from syntax (just as pure XML approaches separate representation and presentation) by designing a system around a data model that can represent the data structures internal to a wide variety of file formats. In essence, we represent each data format as an encoding of a structured container and develop a schema language and system architecture to provide a standard means of referencing, querying, reusing and navigating through those data structures. When we combine these schemas with plugins to decode and encode data streams in a wide variety of formats we have a complete system that can enable general markup-like capabilities for any data format. This is similar to the approach of Abiteboul in [Abit93] and [Abit95] which provided a relation database-inspired query and update model for structured files. It can also be traced directly back to Hytime [ISON1920] and its “groves” data model, which was defined in order to provide a reference architecture for the contents of multimedia files. Unfortunately, we believe that the Hytime groves' data model was better matched to representing markup-like files than other data formats, and was sometimes as unnatural to manipulate as relational data is in typical programming languages. Instead, with NODAL, we hope to demonstrate that we can derive and implement a simple data model that naturally represents the data structures and formats of modern programming languages and yet can still form a basis for granular reference and annotation inside of existing file formats. Moreover, we have designed our system around the standard URI document/fragment model for reference and query. This emphasis on generality, simplicity and compatibility in the NODAL design is intended to provide the basis for the development of a standard model for data integration and reference.

The design requirements for the NODAL data model and path language can be largely taken completely from those defined for the XML Schema Language [XSchReq] and XPointer [XPtrReq] , without assuming an XML framework. It must be able to represent unstructured, semi-structured, and structured data in as wide a variety of formats as possible and provide a simple, natural path language that can be easily translated into fragment URIs. The NODAL system thus defines a data model that can be applied to modeling document formats in such a way as to provide a basis for a URI-based fragment references for any kind of modeled document. It is also a superset of the relational model with object-relational (O/R) extensions (making such common data structures as sequences and dictionaries explicit) so it can also be used to provide a direct (and indirect) reference and integration architecture that encompasses both database and document-based information repositories. Moreover, we will show how this approach can also be extended to a wide variety of data access protocols, some database-like and some filesystem-like.

The NODAL Data Model

Structured Data: Collections as Nodes

However, recalling the Abiteboul approach (which actually addresses the issues of database update for structured files [Abit95] ), we do not wish to restrict ourselves to simply static data and structures. To support dynamic, structured data, we then add to these literals a system of structured, modifiable types that we refer to as Nodes. We define the node types N such that for t ∈ N we have t = a i v i , a finite set of attribute/value pairs where a i ∈ A ⁡ N and v i ∈ V i N . Thus, for any node type N , we have a domain A ⁡ N that constrains the attributes of t ∈ N and a domain V i N that constrains the values v i that may be associated with attribute a i ∈ A ⁡ N . By varying the constraints on A ⁡ N and V i N we can define a variety of different classes of node types, while still maintaining this common attribute/value model (in essence, we have defined a data type hierarchy that generalizes to Hytime's property sets [ISON1920] and specializes to common data type building blocks). Below, we describe these constraints with set functions that compose new domains (node types) using existing domains (types). See also the UML diagram in Fig. 1.

Figure 1

[Link to open this graphic in a separate page]

NODAL Data Model. UML inheritance diagram of basic NODAL type system.

Map

The simplest of these is the map M ⁡ A V for two domains A ⊂ T and V ⊂ T whose instances associate any attribute a ∈ A with a value v ∈ V .

Set

In this formulation, a set S t A is a kind of map for which A = V and for which t ∈ S t A → ∀ a i v i ∈ t : a i = v i . In other words, if we define a kind of map for which the attributes and values are always identical, then we can use this as a set.

Sequence

A sequence S q V where the attributes A ⁡ S q V = 0 ... n form an interval and all values are in the domain V ⊂ T .

Record

A record R = R ⁡ A ⁡ V i is defined by a fixed set of names A ⁡ R = a i ∈ name and the domains of the values associated with each name V i R ⊂ T . We refer to the names a i as fields of the record R and the domain V i R is the field type of field a i in R . This is clearly compatible with the existing relational model as outlined in [Rama03] , with our record mapping to a relation.

We further define an inheritance relationship for record types such that if a ∈ A ⁡ R ' → a ∈ A ⁡ R and a ∈ A ⁡ R ' → V ⁡ R a ⊂ V ⁡ R ' , a then we say that R is derived from R ' . Clearly, these conditions ensure that R ⊂ R ' even if R has extra fields a ∈ A ⁡ R such that a ∉ A ⁡ R ' . We consider this model to capture the kind of data structure inheritance available in O/R systems such as PostgreSQL [Post87] and object-oriented languages such as Java and C++.

In essence then an inherited record is a restriction of its ancestors, possibly with new fields. To continue this inheritance as restriction model, we allow a property's value type to be redefined in an inherited record, as long as the newly defined type is a restriction of the inherited type. For example, a Document record has a field mime-type for the MIME-type of document that is a Name that matches a particular regular expression. The record type for XMLDocument includes an override of the mime-type field to a fixed value: "text/xml". Thus the Record types can be used to form an object inheritance hierarchy where kind-of restrictions can be maintained. And example of the declaration of Record types and field shadowing is shown in Fig. 2 , an excerpt from the standard NODAL data model for the basic NODAL types.

Figure 2

 ...
  <!--
  The Document type, an encapsulation of a graph of Nodes
  -->
  <record name="Document">
    <field name="mimeType" type="Name"/>
    <field name="root" type="Node"/>
  </record>

  <!--
  The Directory type, a Document that is simply a mapping of names to
  Documents.
  -->
  <record name="Directory" extends="Document">
    <field name="root">
      <map keyType="Name" valueType="Document"/>
    </field>
  </record>
  ...

An excerpt from the baseTypes.nls schema that defines the basic standard data types in the system. An example of type restriction by record inheritance and field shadowing is shown, with a Directory a kind of Document that contains a mapping from Name to Document.

Node Identity and Metadata

Since the node types are modifiable (and thus not uniquely identified with their contents), we require some sort of external identity to be able to maintain stable references to nodes. These node IDs can be anything unique within an identifiable context (e.g. a database table, a file, a repository). For example, in a native NODAL repository, it is likely that we will have simple node IDs that are unique within the entire repository. In a traditional filesystem-based repository it is more natural to have node IDs uniquely determined within the context of the containing document or file. In a relational database mapping, we can simply use the primary keys of each database table as the unique node ID. The flexibility of this requirement should allow us to find such a system of unique IDs for any of the different kinds of data sources we have proposed supporting within this framework. Finally, given that these nodes are the minimal units of modification, it is also natural to suggest that these nodes be the minimal units for recording change history and access control.

Literal Structures

In order to support a lightweight, compound literal type, we also provide a means of defining record-like literals, which we call a structures: S t r A ⁡ V i . Interpreted exactly as a record type of the same form (with the additional restriction that all of the V i types must be literals), these are extremely useful for modelling data that have natural value semantics but are still decomposable into meaningful components (e.g. RGB pixels in an image). In these Struct types, a set of field names are defined and associated with value types in exactly the same way as with Record types, except that the objects created are immutable and have identity associated with their contents.

Restriction Types

One way of specify the constraints on a particular type is to use a restriction language to define a new type as having a limited value space with respect to an existing type. This is a very powerful concept and is the foundation for the XML Schema type system [XSchema2] and the ISO standard type system from which it is derived [ISO11404] . To this end, we provide just such a type restriction facility for atomic types based on a set of matching functions applied to a base type. Some of the restrictions available are: regular expressions for Name and String types; inclusive and exclusive minima and maxima for any ordered type; a namespace restriction for Name objects; and an enumeration list for any atomic type (including the single-valued enumeration or fixed value).

The NODAL Path Language

In order to provide an external reference system for this data model, we adopt the path language approach of XPath [XPath] and define URI references based on a navigational path through a document graph. A path p ∈ P in NODAL is a chain of path components p = p 1 ... p n . Using the concatenation operator p = p ' / p n = p 1 ... p n - 1 / p n , we can define the tail of a path p as the final component p * = p n and the parent of a path p as the path p ' such that p = p ' / p * . Note that neither of these exist if p is the empty path , but that the parent of a single-component path is the empty path.

To interpret paths, we need some way of interpreting the path components individually and then in sequence. In this formulation, a path component has two aspects, its path normalization function and its binding function. Each of the components p ∈ P has a path normalization function N p : P → P that takes a path and produces a normalized path p for which p = p * → N p p ' = p . Thus, any path component p that appears as the tail of a normalized path has the property that N p p = p / p , Thus concatenation is the standard normalizing function. An example of a component that does not always concatenate is the parent operator .. that extracts the parent path. The normalizing function for the parent component is N .. p = { .. if p = ∅ , p ' otherwise . So, the parent operator can only appear as the first component of a normalized path.

But paths are defined to provide a means of accessing values within a node graph, so we need a means to determine the target value of a path V p . To evaluate this target value, we consider another aspect of a a path component, its binding function V p : P → T , which returns the value that a path with p at its tail refers to. We then define the target of a path with respect to its tail as: V p = V p * p . We distinguish two kinds of path components, the absolute components have values that are independent of the containing path p , while the relative components have dependent values. A relative path is then a path that contains only relative path components, whereas an absolute path contains at least one absolute component. To reduce path redundancy we require that the normalization function of every absolute path component produce a single component path containing itself: if p is absolute then N p p = p , Thus an absolute path has exactly one absolute component at its head. Examples of some of the available components are show in "NODAL Path Components".

So far, these paths are independent of the data model outlined above. To provide some grounding, it will be useful to consider paths within documents. Remember that a document is modelled as the graph of nodes reachable from a root node. (see Sec. “Documents and Node Graphs”). Thus, a reference to a document determines the starting point for navigation within the node graph. An absolute path thus has two parts, the document part and the fragment part. If the document part is not empty, then the fragment part is evaluated relative to the specified document's root node. We can now appreciate one of the main advantages of the unification of the node/collection data model in terms of attribute/value pairs: a homogenization of the standard component for selecting a value in a collection, namely property( g ). From a single absolute root path, we can create paths to all reachable nodes and values with only chains of these property components. The property component is thus the fundamental building block of the relative reference mechanisms in the NODAL path language.

Finally, each path component is expressible as a function or shorthand in text, and a path URI is simply the concatenation of these component expressions with an appropriate separator (in the fragment part of a path, the `/' character is used as a separator). As with XPointer [XPointer] fragment URIs, we use a context frame \#ndl(...) to enclose NODAL fragment expressions. Other fragment expressions are passed to the plugin responsible for the MIME type of the document addressed. Thus, HTML and XML documents can properly handle \#id fragment ids and even xpointer(...) expressions without interfering with the NODAL references.

So, given the components described in NODAL Path Components table above (a subset of the component operators available in NODAL) we can describe a number of NODAL path expressions as examples:

The NODAL Architecture

This data model and path language are, of course, only useful in the context of a system to interpret them. The NODAL prototype is a Java middleware implementation of the data model and path language with a small set of proof-of-concept format plugins (text, HTML, XML, the NODAL schema language). This prototype is currently able to interact with data access protocols file:, http: and imap:. All of these provide documents as bitstreams and thus need to pass through document decoders in the format plugins before being presented via access APIs. (see Fig. 3 ).

Figure 3

[Link to open this graphic in a separate page]

The NODAL system architecture. Note that the API presents nodal database view of the information whether it is coming in from a database or a filesystem source. The connection between the database and filesystem(s) is via a set of data format plugins that encode and decode a variety of data formats.

Future Directions and Potential

We are currently testing the system in this data consumption mode and formulating a number of pilot projects to assess the ability to build working applications on top of the NODAL APIs. The NODAL type system is already implemented based on an interpretation of the data model described in the baseTypes.nls schema excerpted in Fig. 2. One of the most interesting possibilities is to create a personal information management environment that can operate by building semantic indices not only to local and remote files but also to their contents to provide granular annotation.

Future work is planned on the following items:

• Database interaction: Automatically extracting database schema and allowing the NODAL APIs to access relational and object database systems was one of the design goals and must now be implemented and tested.
• Read/write functionality for all data sources: Currently we can consume and reference data from a wide variety of sources but can as yet only write to local file systems.
• Query: It is important to provide mechanisms for data discovery that go beyond the simple exploratory metaphor provided by filesystem and link following. We must be able to search data based on content and structure.
• Versioning: As was stated above, the best unit for attaching metadata and managing version control is the Node. Each node has a specific structure and a limited number of possible changes. We can already generate change records and associate node versions with each transaction. The difficulty comes layering this functionality on top of inextensible data stores such as local filesystems.
• Access control: If versioning is best done at the node level, then perhaps access control is too.
• Synchronization: Once a version management is available, then it should be possible to enable CVS-like [CVS90] synchronization between local copies and remote repositories. We suggest that it will be easier to automatically extract difference charts between local and remote modifications, since most of the node types have a very simple set of modification operators. In fact, the Simias Collection Store [Sim04] being used by the iFolder project has a very similar structure to this one and it is completely designed for synchronization.

With the NODAL data model and path language, we have demonstrated a new paradigm for opening up all Web-accessible content (not just HTML and XML) to the advantages of hypertextual information management. We hope to extend it so that it can become a generic Data layer that can be the foundation for a next-generation of fully interoperable, collaborative end-user applications.