XML Query (also known as XQuery) is now a W3C Recommendation, as of January 2007; in other words, it is recognised by governments and others as an international standard. There are many implementations, and already XQuery is being used in industry. Among the implementations certain patterns have emerged: sets of features, positioning, maturity and more.
This paper is intended as an introduction not to XQuery but to talking about XQuery with potential users: it introduces a vocabulary, suggests a simple taxonomy of implementations based on the emergent patterns, and gives ways to help people choose from among those implementations, whether Free, open source or closed.
Products and specific implementations are mentioned by name in this paper in order to give specific examples. This is not in any way an endorsement of those products: the author of this paper has no information about the reader's situation. One might reasonably assume that every implementation is good for something, even where it is not immediately clear what. An implementation that is weak in one area might be strong in another.
A consultant might find that this paper provides useful ways to communicate with a client about how to use XQuery, and how to choose software. A programmer or query writer might also find it useful to have a wider perspective on how XQuery is used; an implementer might like to consider the scenarios given in the paper as possible features for the future.
XQuery 1.0, An XML Query Language, was published as a W3C Recommendation on the 23rd January 2007 [XML Query] together with a number of supporting documents. There are now some fifty or more implementations of the XQuery language listed on the XQuery Public Web page at W3C [XQuery Web Page], but that page, at the time of writing this paper in the Spring of 2007, gives little or no guidance to people trying to choose which software to use.
Since the number of implementations is so large, one is naturally tempted to partition those implementations into groups, to look for patterns, and to categorise them. The result may be a clearer view of how XQuery is being used, and one might also hope for insights that will help in designing system architectures that incorporate XQuery engines.
This paper discusses in general terms the sorts of features that appear to characterise the main XQuery implementations, and suggests some ways to choose between them. Specific products are not endorsed in this paper. With XML in use in places as diverse as automobile engines and encyclopedias, the most important part of investigating suitability to task of an XML tool is often the tool's intended usage environment. It is not unreasonable to suppose that most of the XQuery implementations are useful for at least something.
A query is an expression that, when evaluated in the context of a given data set, returns some or all of that data set. In the case of XQuery, the expression is of course written in the XML Query notation. The technical difference between implementations, then, is in where the query can appear, how the data set is selected, in what form the results can be generated, in optional features supported, and in performance characteristics and other restrictions.
The purpose of this section is to suggest to the reader some of the most significant ways in which implementations differ.
Some XQuery implementations are intended to be very general-purpose. Others are intended for specific uses, such as debugging queries; still others are aimed at specific application areas, such as deploying Web services or message queueing. If you try to take an XQuery engine optimised for fast response over large sequences of small documents, and use it to search a single multi-terabyte file, you may find that it keeps the whole document in main memory, and fails. Software intended for use in embedded systems may be unable to communicate errors back to a user, but, conversely, software aimed at debugging and development might not do sufficient optimisation for production use.
For example, MarkLogic sells a commercial XQuery server intended for large collections of data; Oracle and IBM also both include native XQuery support in database servers (Oracle and DB2) that can handle large amounts of data. MXQuery is intended for use at the other end of the spectrum, on mobile devices, as is xbird. Products such as StylusStudio and Oxygen include debugging support for XQuery, and are intended for use by developers of queries and applications, as well as by document authors, but not for use on a server (the makers of StylusStudio do have a separate server product). BEA has a product that's intended for enterprise or middleware use, and handles streaming data: that is, running queries over a single but indefinitely large document.
Do not be afraid to ask vendors and implementors whether their software will be appropriate for a particular usage scenario: an honest no can save you, the client and the vendor a lot of problems, and the next time you encounter a scenario that is appropriate for that implementation, or are asked to suggest one, you will remember.
If the XQuery implementation is embedded in another language, there may be restrictions on where XQuery expressions (queries) can occur, on the data that can be queried, and on the format of the results (for example, a reference to an in-memory tree might be returned). Similarly, calls from XQuery to external functions may be possible, but are almost always restricted to being implemented in particular programming languages. On the other hand, if you are using a relational database or other system with an embedded XQuery engine, the chances are that you've already selected the system itself and have little or no choice about the XQuery implementation.
IBM DB2, Microsoft SQL Server and Oracle all include XQuery support. The Microsoft implementation was released first, and, at the time of writing, conforms to a draft version of the XQuery language; they appear to have taken the approach of avoiding implementing anything they felt likely to change, which makes their implementation incomplete but will probably help their users to migrate to a more recent version, and the subset they have chosen appears very useable in their environment. The IBM and Oracle implementations were released more recently. Probably work was begun on all three implementations at around the same time, but the three companies happened to be at different points in their release cycles. The result was that IBM and Oracle released engines conforming to the Recommendation (or possibly to the Candidate Recommendation; there are some very small differences). In addition, some or all of these implementations have support for full text searching, even though at the time of writing the Full Text extensions to XPath (and hence to XQuery) are not yet a standard. Full text searching can have significant performance benefits, but you should be aware that you will want to move any queries you have written to the final standard syntax, when it is approved.
In the open source/libre/Free Sotware world, the Monet project at CWI in Amsterdam has implemented MonetDB/XQuery, and claims to have both a full implementation and a strong emphasis on performance.
A related property is whether an implementation is intended for use in research, or in production, or in development; this is also mentioned elsewhere in this paper. One might also ask if a product is intended for use in an enterprise environment, where there might be full-time systems staff devoted to it, or in smaller organisations that have fewer resources available. Of course, research products often turn into commercial products later, but the question then becomes one of release cycles, support and ongoing development. Software written for a specific research goal, for example to show whether XPath axes can be rewritten efficiently in terms of other XPath expressions, might not implement features that were not needed for the research, and might not be supported after the end of the project. Such implementations often get onto public lists such as the one on the W3C Web site because they help to build up the XQuery community, but might not always be very useful in a production environment.
Software is generally developed and released in cycles. Since specifications are also developed, released, updated, amended and sometimes superseded, software is often not quite synchronised with the corresponding specifications it claims to support.
A number of shipping XQuery implementations in fact adhere, as of Spring 2007, to an earlier draft of the language, because the vendor had to ship software. Although there were relatively few changes to XQuery between 2005 and 2007, there were some changes to the query prolog, to the syntax of function and variable definitions, and to some of the built-in type names: the xdt built-in prefix for some types was removed, and the types instead are in the XML Schema (xs) namespace instead, for example.
One way to test this is to ask whether the implementation can run the XQuery use cases [XML Query Use Cases] unmodified. If not, there may still be good reasons to use the implementation, but you should understand that tutorials and examples might not work without some small (but systematic and easily understood) changes.
In a few cases, XQuery implementers have submitted the results of running the XQuery Test Suite [XQuery Test Suite]. Since the test suite is very large and covers all the optional features of XQuery, most implementations will not give 100% scores, but the tests are a good indication that the vendor is anxious to be seen to conform closely to the specification. At the time of writing, test results have been reported for fourteen different implementations,
The others are listed here in alphabetical order, with a few brief notes intended to give the reader a feel for some of the diversity of these implemntations.
Another possible source of information is to ask for XMark benchmark results.
Another related issue is how frequently the vendor makes new versions available, and under what terms; it is generally difficult to assess the turnaround time for support questions, although with open source projects in particular that information is often in the public record. Even if full released are infrequent, there may still be bug fixes and other updates.
Some XQuery implementations work only on stand-alone XML files; others work on data stored in some form of database, or in a proprietary format. In addition, some XQuery implementations can run queries over multiple documents, or collections, which may in turn be stored in a database (whether relational or XML-native) or may be a file in the file system, either in XML (perhaps a directory containing XML files) or in a proprietary application-specific binary format. The application-specific formats are usually designed to optimise performance in specific aspects.
For example, Saxon uses only files, but can treat a directory of files (a folder, if you prefer) as a collection. The files are loaded into memory as XML and parsed each time the query is run, so there are performance limits to this approach, but in practice it can be useful. MarkLogic, by contrast, and eXist, are both examples of XML-native databases that pre-index the documents into a proprietary binary format, and have support for collections.
One of the strongest features of XQuery is surely that it provides a unified interface to querying many diverse forms of data. A secondary benefit is that, since the language is standardised, you can you use the same language in multiple applications; this can reduce staffing and training needs as well as reducing support costs. Most XQuery implementations do support queries against more than one data source. For example, Qizx/open can query relational datbases as if they were XML, as well as supporting fetching any form of XML via a URI with the standard XQuery document function.
When implementations rely on pre-indexed formats, they sometimes cannot load external XML documents at query time, whether from files or from a supplied URI; this can sometimes be an advantage (providing a closed world) or a disadvantage. For example, Berkeley DB XML can only fetch external data in its own pre-indexed binary file format, so that it cannot directly open an XML document on a hard disk, or call a Web service.
A number of implementations use a relational database as a backing store; others can present existing relational database tables to the XQuery interface as if those tables were in fact an XML document, sometimes as efficiently or more so than the SQL interface to those same tables. In some cases, however, relational data support is interpreted to mean the ability to store and query over XML documents stored as XML data in individual cells. In that case, the database may or may not support a view of a relational table as XML from an XQuery expression. IBM DB 2, for example, lets you store an entire XML document in a single cell, and descend into it with XQuery, either separately or as part of a SQL expression; Oracle appears to do something similar.
Even where some form of database is used, whether relational or XML-native, an implementation may or may not support the optional XQuery Update feature, which is only a W3C Working Draft at the time of writing. Furthermore, an implementation might (as already mentioned) support fetching an XML document, pointy brackets and all, over HTTP, using the XML Query document function; in this case, a CGI script or servlet can be written to provide access to database tables if they are not available natively, at a cost in performance. If this is hidden from the query-writer behind a set of functions you define in an XQuery module of your own, a subsequent change to the implementation to support direct access to the database tables might not have significant impact on queries you have already written except of course to enable them to run much faster.
Regardless of the backing store used, many XQuery implementations support access to relational databases using JDBC or ODBC, even if that means connecting to an external and possibly remote database system.
There is a lot of variation in the way that XQuery engines provide access to relation data. As a result, as with any other extension or implementation-specific feature, you can easily use it in a way that makes it difficult to move between implementations. To prevent this, define a set of accessor XQuery functions so that queries can be written in terms of the problem domain rather than in terms of a database storage schema. The functions can be contained in their own namespace, and in most implementations can be stored in a shared module, reducing the danger that programmers will copy the functions and change them slightly.
To give an example: suppose that you have a relational database storing information about all of the acrobats in your circus troupe, including their names, birthdays, salary, height, weight, skills and preferred sock colour. Of course, you must ensure that the preformers do not wear clashing hosiery, but rather than writing code like:
for $a in jdbc:get_rows("circus.performers") where ok_together($a/sql_row, $candidate/sock_colour)
or, in another system.
for $a in jdbc:get_rows("circus.performers") where ok_together($a/performers/sock_colour, $candidate/sock_colour)
you could isolate the differences:
for $a in circus:getperformers() where circus:oktogether($a, $candidate)
Now you need to define oktogether, but if you decide to keep a record of how well a particular pair of acobats perform together, rather than choosing performers based on the colour of their footwear, you won't later have to look through all of your queries and update them. If you switch from (say) Qizx/open to Saxon, or to IBM DB2, you will probably need to change the definitions of the functions in your circus namespace, but not the code that uses them. The oktogether function might include lines like
coloursOKtogether( circus:getsockcolour($a), circus:getsockcolour($b) )
and you would define those lower-level functions in terms of the primitives provided by the XQuery implementation you were using and the specific SQL database schema you used.
If it seems that a lot of space has been devoted to a simple example, consider that the ability to define functions in this way adds a considerable flexibility to choosing an implementation. For example, if you only very occasionally refer to a relational database, and then only for a small amount of returned data, you might be able to have a separate program that creates an XML file containing the data when it changes. If the data depends on the query, you might even be able to have a Web interface to your data (presumably that's accessible only from the computer running the XQuery implementation), which could be as simple and brute-force as a CGI-based Perl script or a Java servlet that generates XML on the fly.
Most (but not all) XQuery engines can resolve a URI and run queries against the resulting XML (see the Web Integration section below). If a particular implementation does not offer a feature you need, ask yourself whether you could implement that feature easily yourself and hide that behind some functions, so that if the feature later becomes available natively there is no need to change actual queries.
The XQuery document function takes as its argument a URI; some implementations do not support this, however. Some that do support it require that the returned resource be in a proprietary, non-XML format. This is particularly unfortunate for implementations that do not support database access, as the method described above to use the document function to invoke a CGI script is then unavailable.
The feature described here is the ability for a Query to go out over the Web and fetch something. This is not the same as the ability to embed a fragment of XQuery in an HTML document and have a Web browser do something with it; that ability is desvribed under Ajax Hype Compatibility below.
Most XQuery implementations can fetch remote XML documents. Of course, there may be performance issues, both because the network access is likely to be slower than the local file sysetm and because the returned data must be parsed from pointy-bracket normal form into an in-memory representation; the fastest XQuery implementations use indexes for efficienct, but an externally loaded douemnt will generally not be in any index.
Not all implementations of XML Query can access XML files directly; they may be restricted to documents that have been indexed using some external loading program, or that already reside in a database, or to files converted to some proprietary format.
XQuery does not guarantee access to directories or file listings; some implementations (e.g. Saxon) support opening a directory using the XQuery collection function, and then allow queries over all the XML documents found in that directory, but this is a non-standard extension.
There is no direct way to invoke a Web service in XQuery 1.0. As a result a number of incompatible approaches have arisen; if Web services are important to you, check whether the implementation supports them. If it does not, you may be able to get there by writing an extension function, as described above under Relational Integration, or even by invoking a servlet or CGI script, as mentioned above (Web integration).
A disconnect between XQuery and Web services is that XQuery functions use positional parameters and Web services relies on named parameters. A good implementation might retain static type checking of function arguments, perhaps based on WSDL information.
A number of XQuery implementations can call Web services directly, and any implementation that has the abililty to call user-defined functions in a native language (C, Java, OCAML etc) will also be able to do the same. The reader should be aware that XQuery has an entirely synchronous processing model. This means that a query that calls a Web service will generally have to hang until it gets either a reply or an error. A more efficient model might be to use separate fragments of XQuery that are called by another language when the response from the Web service is received.
How can you tell if an XQuery implementation can be used in this way? It must be accessible in one of the following ways:
Even if none of these is possible, there might be a sample program included with the API documentation that runs a canned query and can be modified to read the query as input.
Some implementations of XQuery are intended primarily to be development environments in which one writes queries and explores data; the queries are then, one imagines, deployed against some other XQuery implementation. A useful debugging tool will give helpful error messages on syntax errors, might support the syntax both of the Recommendation and the previous Working draft (or Candidate Recommendation), and will generally let you use multiple different XQuery engines directly from the debugger.
There are XQuery plugins for development environments such as Java Eclipse, and there are XQuery-aware XML tools such as Oxygen and StylusStudio. Many text editors have some sort of syntax highlighting for XML and XSLT, but the XQuery grammar is somewhat unusual and is harder to support.
There are several ways in which XQuery implementations support W3C XML Query:
Firstly, an implementation can use an external validator to determine whether loaded instances are valid, and can invoke the same schema processor at runtime to handle XQuery validate expressions, serialising into XML to do so when needed. Such implementations will not in general be able to mark nodes in the data model with their schema types, and will be limited in their ability to perform static typing.
Secondly, an implementation might use an external validator but receive back an annotated version of the XML document (the so-called PSVI [Post-Schema Validation Infoset]). Such an implementation may usefully support static type checking of user-defined types.
Thirdly, an implementation might use a built-in validator. This has the same effect as the second case, but is generally likely to have better performance. In particular, there are implementations that use knowledge of document structure when they compile a query.
Saxonica has a commercially-available version of Saxon that has Schema support, and makes use of the validation and schema information for XQuery optimization. MarkLogic also has Schema support. Galax is a more research-oriented system that has Schema support; Anglo may be another (the author has not at the time of writing been able to verify this). Microsoft SQL Server has some static typing support, but apparently not full XML Schema support. There are many others.
Some implementations support a transform function to call an XSLT engine; most often this is Saxon, an open source XSLT processor with a commercial version available that has W3C XML Schema support. Not all implementations identify which version of XSLT is supported, nor how the feature is implemented.
In a tight integration it might also be possibly to call XQuery functions directly from within an XSLT stylesheet.
The W3C XSL Working Group and the W3C XML Query Working Group have been developing a full-text search facility for use with XPath 2 and XQuery. At the time of writing (Spring 2007) there are no published plans to incorporate this into XSLT, but there is a public working draft of the specification [Full Text]. Because the final syntax is still not decided, some implementations offer the functionality through a set of extension functions. Others implement the draft syntax, or some other syntax, or do not offer the feature.
Although it is possible to search collections for instances of natural-language words or phrases using the XQuery contains function, this is unlikely to be efficient, and phrase search becomes sensitive to spaces and punctuation in inconvenient ways. Furthermore, full text support generally relies on an inverted index, and will have close to constant speed regardless of an implementation's optimisation strategy or of the database size.
It is too early too highlight particular implementations that are known to support full text queries.
Once a query has been used to select items of interest, one typically wants to perform operations other than merely printing the result. In particular one often wants to change the resulting data, and have the changed data stored back where it came from. The XQuery Update Facility is a draft specification to allow XQuery expressions to do exactly that [XML Update] , and implementations of it are starting to appear.
It is too early too highlight particular implementations that are known to support update.
No specification is likely to anticipate all possible uses. People writing XQuery expressions might need to access the local file system, or to call mathematical functions, or any of a host of other things that can't be done directly. XQuery provides a mechanism through which an extension function can be registered and then called. The functions can be written in XQuery (in which case they are usually contained in XQuery modules), or they might be written in some other language entirely, such as Perl, OCAML, Scheme, C++ or Java. Not all implementations support user-written extension functions, but, when they do, most implementations support the functions in only one programming language.
The simplest form of XQuery implementation to understand is where the user invokes a program on a query, and the program runs the query, prints the results and exits. This is the most general form, and often the most easily integrated into other systems, but there can be obvious efficiency questions, and it is not ideal for development or debugging.
Examples include Saxon, Qizx/open, Galax, and many more.
Some XQuery implementations are applications with editing windows in which one can enter a query, and which bring forth new windows containing results. These are typically intended for developing with XQuery. They can be very useful, but at the expense of having to use the built-in text editor. This is perhaps less onerous for users of Windows Notepad than for users of vi or emacs.
The best-known examples are probably Stylus Studio and Altova XMLSpy.
Most major relational databases now include support for XQuery, and typically do so at the storage level alongside SQL, rather than by translating XQuery into SQL and executing it indirectly. This means that these implementations can be very efficient, although that efficiency is highly dependent on the optimiser.
Examples include Microsoft SQLServer, IBM DB2, Oracle and MonetDB.
A variation of the stand-alone implementation, a server sits listening for queries to be sent to it over a network (or possibly only on the local machine) and replies to them as needed. This avoids having to wait for the query engine to start up and initialise itself for each incoming query, at the expense of increased complexity.
Examples include MarkLogic, Cognetic XQuantum, Ispras Modis Sedna, Data Direct XQuery Server, and more.
Some implementations of XQuery are not meant to be used directly; the XQuery engine is used primarily by the developers of the application, or has a graphical front end that hides the syntax. For example, there are computer desktop search engines that use XQuery, but that do not expect the user to be familiar with XQuery or even XML.
Implementations in this category are often tied to their environment and cannot easily be reused. Eamples include Apple Sherlock, as well as ETH mxquery, BEA AquaLogic Data Services Platform,
People have many different reasons to use XQuery, and these reasons are by no means mutually exclusive. Purpose influences strategy. A clear understanding of the benefits one expects can dictate the course one must follow.
Data mining, searching for trends and patterns, finding possible new products, researching, all of these are amenable to XQuery, and indeed XQuery enables them over XML data as nothing has before. XQuery also empowers the user to correlate relational and XML data.
The user in this case may be most interested in either a standalone implementation of an application with a graphical user interface.
A common deployment pattern is to encode business logic in XML, and to use XQuery and/or XSLT to generate Web pages on the fly based on a combination of page templates, business logic, and encapsulation of user input or state. In this environment one typically needs a stand-alone implementation, a server or a development library one can incorporate into a server.
It should be noted that the XML Query specification does not directly support HTML (or XHTML) serialisation; most implementations provide this as an obvious extension. It is not sufficient to serialise results using the XHTML DTD (or Schema), because there are additional syntax constraints: requiring the space in <br /> for example. An alternative is to use XQuery to generate an intermediate document that is then processed with XSLT to create the final Web page; whether this is useful depends mostly on the backgrounds of the individual people maintaining the system.
Since Web services are encoded in XML it seems natural to try to use XQuery to process them. In practice there are some obstacles. Implementations do often offer extensions to surmount these obstacles, and the W3C XML Query Working Group is also considering some of them. The main issue is that XQuery has no way to mark an external function as having side-effects, and no way to control the sequence of evaluation. It is a declarative, functional language. This means, for example, that an implementation can use memoization [Michie 1968] to store the result of calling a function, and use that stored result instead of calling the function again in the future. A function such as random could thus always return the same result, even if the first time it was run it called an external hardware random number generation facility.
Even where functions are called every time they are encountered, XQuery allows an implementation to reorder expressions and to evaluate them in whatever way is most efficient. There is also no standardised ability to trap errors or handle exceptions. As a result, it is probably best to call XQuery from some other environment that handles these things, or to use an implementation with the necessary extensions.
XQuery is being used today in a wide variety of situations and environments. Selecting an implementation to use can be difficult. However, there are common usage patterns that are widely supported, and common extensions that can help deployment. The intent of this paper is to help the reader to choose among these implementations, and the author would greatly appreciate feedback.
A later version of this paper will (after the conference) be published at http://www.w3.org/XML/Query/ if it appears to be useful.
[Michie 1968] Donald Michie, Memo Functions and Machine Learning, Nature 218, pp. 19-22, 1968.