From Word to XML to Mobile Devices

David Lee

Abstract

Epocrates® provides clinical references on mobile devices. Key constraints for storing content on mobile devices are small memory footprint and limited CPU power. Key attributes of clinical content is 'rich data' requiring structural and presentation markup to be displayed. We have standardized on an efficient form of binary XML (XText) which is designed for efficient parsing and display natively on mobile devices while allowing authoring in any environment that can produce XML.

A problem arises because clinical content authors are familiar with authoring using Microsoft® Word and are not comfortable authoring in native XML editors. Furthermore, content is not only authored by Epocrates employees, but also by third parties who we do not have the ability to require provide us with XML. The vast majority of content authors, both internal and external, are only trained and comfortable authoring in Microsoft Word. Even when provided with other methods of data entry, such as web enabled applications or native XML editors, the authors initially create content using Word and then copy and paste into the application. This 'impedance mismatch' causes much frustration, inefficiency and errors (both human and machine).

Rather then fighting against this installed base of skills and habits, we have created a workflow and tools to enable authors to create content in their environment of choice (MS/Word) and then translate these documents into our XML schema which is then used to create the binary XML content which is deployed to mobile devices.

This paper is a case study of our experiences to develop a workflow to handle smooth translation from MS/Word formats to XML and then to device formats. This includes lessons learned in both the human and technical realms and can serve as a design pattern for others attempting to achieve similar goals. Automated portions of the workflow include an XML pipeline based on XQuery. Human and usability consideration is given to what types of editing within Word people are most comfortable with and what kinds of errors and mistakes are common. The entire workflow is implemented using no commercial components except for Microsoft Office, and is based on Java, scripting and the Saxon implementation of XQuery.

Keywords: Office Documents; XQuery; Transforming

David Lee

David Lee has over 20 years experience in the software industry responsible for many major projects in small and large companies including Sun Microsystems, IBM, Centura Software (formerly Gupta.), Premenos, Epiphany (formerly RightPoint), WebGain. As senior member of the technical staff of Epocrates, Inc., Mr. Lee is responsible for managing data integration, storage, retrieval, and processing of clinical knowledge databases for the leading clinical information provider. Key career contributions include Real-time AIX OS extensions for optimizing transmission of real-time streaming video (IBM), secure encrypted EDI over Internet email (Premenos), porting Centura Team Developer, a complex 4GL development system, from Win32 to Solaris (Gupta, Centura), optimizations of large Enterprise CRM systems (Epiphany), implementation of ecommerce systems for on-demand digital printing and CD replication (Nexstra).

From Word to XML to Mobile Devices

David Lee [Epocrates, Inc.]

Extreme Markup Languages 2007® (Montréal, Québec)

Copyright © 2007 David Lee. Reproduced with permission.

Introduction

What is Epocrates ?

Epocrates develops clinical decision support software that can be used on mobile devices. More than 500,000 healthcare professionals currently use the applications.

Common Terminology

There are a few terms which are difficult to avoid, so they are defined here.

PDA [Personal Digital Assistant]

In this paper refers to mobile devices, running a general purpose operating system, specifically the Palm OS® or Windows Mobile® (previously know as "Pocket PC" and "Win/CE").

Palm

A PDA device running the Palm OS® operating system

PPC

A PDA device running Microsoft Pocket PC or Windows Mobile operating system.

PDB

From the Palm/OS terminology, a "PDB" is a "Palm Database". A very simple variable length record format with a single 16 bit key index.

Syncing

The process of synchronizing a server's database with a PDA's database. New records on the server are inserted into the PDA database, modified records updated and records which no longer exist on the server are deleted from the PDA.

Sync Server

The on-line servers in the Epocrates data center responsible for managing syncing to the PDA's.

KOL [Key Oppinion Leader]

From clinical publication terminology, refers to a person who is considered an expert in their field and who's comments and opinions can be trusted.

Core Application

The Core Application supporting the technologies and workflow desribed in this paper is the MRC [Mobile Resource Center] application which provides a constantly updated mobile reference to clinical publications. See figure 1.

Figure 1: Home screen of the Mobile Resource Center Application
[Link to open this graphic in a separate page]

Background "The Problem Space"

The "problem" attempting to be solved is the 'impedance mismatch' between clinical authors and XML content. Our experience is that the authors of clinical content authors are familiar with authoring using Microsoft Word and are not comfortable authoring in native XML editors. Furthermore, content is not only authored by Epocrates employees, but also by third parties who we do not have the ability to require provide us with XML. The vast majority of content authors, both internal and external, are only trained and comfortable authoring in Microsoft Word. Even when provided with other methods of data entry, such as web enabled applications or native XML editors, they initially create content using Word and then copy and paste into the application. This 'impedance mismatch' causes much frustration, inefficiency and errors (both human and machine).

Rather then fighting against this installed base of skills and habits, we have created a workflow and tools to enable authors to create content in their environment of choice (Microsoft Word) and then translate these documents into our XML schema which is then used to create the binary XML content which is deployed to mobile devices. This workflow leverages the common skills and experiences of the authors by allowing them to continue to use Word as their authoring software, but constraining them in ways which makes the translation of Word to our XML schema fairly simple.

Often, the more difficult problems are not technical problems, they are human problems. Somtimes forcing people to work differently so they fit well within your ideal technical model simply doesn't work.

Characteristics of the application

The characteristics of the mobile application, with respect to XML usage, help drive the design requirements for XML processing on the device and in content creation. The application runs on mobile devices which have very limited memory and typically slow CPU. Parsing XML on the device is not generally fast enough to provide a good user experience, instead an efficient binary XML encoding of the document is deployed to the device which can be parsed and displayed quickly and with minimum memory requirements. This means that all XML parsing, and conversion to device formats needs to be implemented on the server or desktop, not on the device; this allows the use of tools which are not easily portable to devices.

Workflow Outline

An intermediate XML format is used that represents the full structured data. This intermediate format can be generated from any number of sources, not only Word documents and then itself serves as the source of possibly multiple output formats. The relevant workflow involves editing content in Word, then translating first to the intermediate XML format (MRC [Mobile Resource Center] schema) then finally to a display format (HTML, PDF, XText).

Human Workflow

The human workflow is involves many people in the publishing process, the conversion to Word to XML is only a small part if the process. See Figure 2.

Figure 2: Human Workflow
[Link to open this graphic in a separate page]

XML Conversion workflow

The automated part of the workflow converts Word or other document content into the intermediate format (MRC XML) and then into an output display format. For mobile devices this is an XML schema called XText [LEE-XML] which is then binary encoded to an efficient format for the devices. See Figure 3.

Figure 3: XML Conversion workflow
[Link to open this graphic in a separate page]

Deployment workflow

The deployment process is initiated after the final QA of the content. Device formated content is pushed to the database server and made available to the "Sync Servers" in the form of both complete data (full record images) and incremental data (changed data since the last update). When devices connect to the data center during synchronization, the appropriate content is deployed to the device to bring it up to date. See Figure 4.

Figure 4: Deployment workflow
[Link to open this graphic in a separate page]

MS Word Authoring

At Epocrates we have had years of experience with how typical content authors, at least in the clinical publishing field, work within the Microsoft Word environment. Somewhat surprising, to us at least, is that many of the sophisticated techniques used by the popular Word to XML conversion tools actually work extremely poorly in the 'real world' of clinical authoring. We have learned about what features of Word can be exploited to aid in XML translation, and what features are too easily abused.

Word Styles

Early attempts at enforcing structure within Word documents made heavy use of Styles. The idea is to use Word styles to convert directly to XML elements or to tag locations in a document corresponding to sections in an XML document. In theory this seems like a perfect match between Word and XML, and is the technique used by several major commercial Word to XML conversion products

However in practice we have found this to be an unsuccessful strategy. It is very difficult to enforce consistent use of styles in the Word environment even if we supply the template (Word DOT files). Word does not have much ability to constrain use of styles and which styles are allowed where in the document. Authors frequently will apply the wrong styles, copy styles from other documents, delete or re-arrange the positions of tagged markers in template documents. Also common practice is authors choosing to not use styles at all but rather use the font characteristics to achieve the right "look" without actually marking the document with styles.

Tagged Sections

A common goal in extracting structure from Word documents is to identify sections of text which belong within XML elements. We created template documents and marked sections with either styles or bookmarks. For example, we create an "Abstract" section, and mark it by using the word "Abstract", or using a style or bookmark, expecting authors to enter the abstract text directly after this.

Unfortunately our experience is that authors would edit the tagged sections, sometimes deleting them entirely, reorder them, or copy and paste sections from other documents making parsing of the document impossible.

Form Fields

We experimented briefly with Word Form Fields. The idea is that these are the natural candidates for entering tabular data. Surprisingly people had a great deal of problems with them. The form fields only allow "plain" text (no markup), this caused some people to actually delete the form fields and paste in structured text. Furthermore, parsing the data of the form fields from the Word Object Model was tedious. In general we found form fields to be of very limited usefulness.

Formatting and symbol characters

A very common problem is authors using Word formatting, and 'smart' features such as "Smart Quotes", auto conversions to symbol characters such as copyright, em-dash, en-dash and manually inserting Unicode characters such as units (e.g. micro), Greek characters, and mathematical symbols. The set of symbols which are portably supported on mobile devices is very limited and these 'special' characters are a constant cause of problems. It is difficult at the time of authoring to prevent people from inserting unsupportable characters.

Since it is impossible to stop insertion of any text into a Word document, special care must be used to strip out or translate unacceptable characters, especially the "smart" characters that Word is so insistent on using

Word Macros

The Word macro facility (Word VB Script) is a very powerful feature of Word. When used carefully it can assist in the authoring process, scan documents for errors, correct common mistakes, and even save out an XML representation of Word documents. However when overused it can cause problems for the author by getting in the way, hitting unanticipated runtime errors or not being activated properly. The macro language itself, while powerful, is cumbersome to program complex operations and difficult to maintain and deploy consistently.

It is difficult, and sometimes impossible, to enforce the use of macros. Security settings on individual workstations can completely turn off macros. Use of multiple versions of Word can cause compatibility problems. Unanticipated copy and paste of content across documents can strip documents of attached macros. Keeping a distributed authoring environment up to date with code changes is also a challenge.

Our experience with clinical authors suggests that deployment of macros to an authoring staff is reasonable for some purposes as long as some basic design rules are followed.

  • Macros should never be required for authoring. They can provide useful error checking help during the authoring phase but should not be an absolute requirement. If a user somehow disrupts the functioning of a macro in a document they should be able to complete their tasks, even if it means that some errors are undetected at authoring time.
  • It is acceptable for the production staff to be required to use macros because their work environment is more controlled and can expect timely assistance from IT if problems occur.
  • Macros should be kept as simple as possible due to complexities of writing, debugging and maintaining macros.
  • Macros should not interact with every keystroke, but rather should be invoked at the end of an editing session to help catch and fix common errors.

Getting XML out of Word

We evaluated many methods of extracting XML from Word documents including several commercial products. We found no single 'off the shelf' method suitable to our needs for either cost, functionality, or usability reasons. The free products were not featureful enough for our needs, and the commercial products have very restrictive licensing agreements not appropriate to our distributed work-flow (very high per seat license fees) and still required a large development effort to customize the products which diminish their "off the shelf" advantages. Furthermore, we found the core techniques used by commercial products often run counter to our own usability experience.

We decided to implement in-house our Word to XML conversion.

Word to XML Conversion strategies

There are multiple strategies for converting Word to a usable XML format.

Convert through RTF

Many commercial and free software applications convert Word documents to XML by first storing as RTF. Then a post processing phase parses the RTF and generates XML in some predefined schema.

Convert through HTML

This strategy involves "Save as HTML" then post processing the HTML to XML. Visual fidelity is good but structural elements are lost (such as bookmarks, styles tags).

Native Word 2003 XML format

Word 2003 can save natively as Word XML [MSXML]. This is a very powerful feature but it requires use of Word 2003 or later and stores a very complex XML format which is tedious to parse.

Convert through Macros

Word Macros (VB Script) can access both the Word Document Model, and the MS XML COM object. This allows a Word macro to store Word documents as XML in any schema. However, doing a good job of this is very difficult due to the complexity of the Word Document Model, and the use of VB Script. The advantage over "Save As XML" is that it does not require Word 2003 and can interactively present the user with error messages as problems are found.

Selected Input Strategies

We have two separate products which currently need conversion from Word to XML, developed by different teams. One project, which has been in production for several years, used the "Convert through Macros" technique to extract and produce the final XML format. This project is where we uncovered many of the early mistaken assumptions about common editing usage. It also uncovered problems attempting to convert a Word doc through macros to a final XML format. A mixture of conversion and business logic all in one Word Macro created very complex code that was difficult to maintain and enhance, and impossible to re-use for other projects.

A new project gave us the opportunity to start over and reexamine our design. For both projects we standardized on a Word usage "design pattern" [DESIGN] for minimizing data entry problems.

Word Tables

We have found to-date that the most error resistant and easiest to use design pattern for entering data into Word which is to be extracted as XML is to make use of Word Tables. For both projects we construct one or more 2 column tables in the 'template' document for distribution to content editors.

In column 1 we enter field header names or descriptions like "Title" or "Article Date", and create a Word bookmark with an internal field name. In addition we add descriptive text such as the date format 'YYYY-MM-DD' or other suggestive text such as "Enter the title up to 40 characters".

In column 2 we either leave the cells blank, or include sample data such as "Put the article title here". We have found this very simple structure is highly resilient to a variety of common human errors and editing practices. People very rarely corrupt tables in Word like they do in free-form or style tagged text. Even copy and pasting from other documents does not corrupt the controlling structure. Changing style tags doesn't effect things. Even moving or inserting rows doesn't break the structure as long as the entire row is moved. See Figure 5 for a sample Word document in this format.

Furthermore this structure is relatively easy to parse and extract meaningful XML consistently.

While in this project, the schema is fairly flat, we have experimented with more complex schemas which have a deep structure and have found that nesting tables within tables to represent nested XML elements works just as well in practice.

The biggest difficulty using this approach is handling of repeating elements. Our experiments having authors insert blank rows to tables have shown we cannot count on the authors ability to do this correctly. Fortunately in our case, the use of repeating elements is limited to a small number of occurrences so we pre-create blank cells for the maximum number of occurrences. We have not yet found a good solution to handling of an arbitrary number of repeating elements.

Figure 5: Example Word document
[Link to open this graphic in a separate page]

Word Macros

We have found that careful use of Word Macros can provide valuable and early error correction which is difficult to provide further 'downstream' in the pipeline. Once data is extracted from a Word file its almost impossible to backtrack that data and provide meaningful errors to the author about where the data came from. By using a Word macro many errors can be detected at point of entry and meaningful feedback given such as positioning the cursor at the error. We also made use of some automatic corrections including translating smart quotes to ascii quotes, and converting unsupportable common Unicode characters.

There are still problems with using macros that are difficult to eliminate completely such as security settings, users copying data into other files without copying the macros, installing into the "Normal.DOT" template on a users system etc. These issues are difficult or impossible solve perfectly so its important to consider use of Word macros as an "aid" but not a requirement for the editing process, especially when the authors are remote, in unmanaged environments, or especially when outside of the organization.

Selected XML Extraction strategies

The original project used Word Basic to extract XML from Word documents, for the new project we went with Word 2003 'Save as XML' and store only the XML version of the Word document. In both cases we defined an intermediate XML schema to represent the basic structural elements we needed to extract from the Word documents. This schema (we call it 'Epocrates Word Schema' ) is composed of the following very simple element set. This is intentionally simplistic and is designed to be the bare minimum needed for our applications.

Elements in the Epocrates Word Schema

Tables

Captures the structure of table, row, and cells.

Paragraphs

Captures paragraph breaks including the Word Style associated with the paragraph

Line breaks

Encodes hard line breaks.

Formatting

Captures basic character formatting including bold, italics and underline.

Bookmarks

Capture bookmark sections. Words bookmark-start/bookmark-end parse are converted into a single bookmark tag which identify a point in the document.

Markup tags

We needed the ability to 'pass through' XML-like markup. To accommodate this we recognize freeform text that conforms to simple XML syntax and encode it as TAG_START and TAG_END tags.

Plain text

Everything else is plain text. Some Word special symbols are translated into more usable Unicode representation.

Word Basic

With one project we use Word Basic to parse the Word document and write it out to the intermediate XML format. The main advantage to this strategy is that it works with versions of Word prior to Word 2003. Furthermore the XML extraction can be part of the same Word Macro which performs the error checking.

The disadvantages are that Word Basic is a tedious language to develop complex software, and that it is actually quite complicated to parse through the Word Object Model to extract the necessary data.

Word XML

In the new project we made use of the Word 2003 feature which allows you to store a Word document with full fidelity as an XML file. You can do this with "Save As" (either manually or in a macro) or by changing the default file type to xml. The resulting file can be opened within Word and re-saved as a .doc file with no loss of information. We use an XQuery [XQUERY] script to convert the Word XML file to the same "Epocrates Word" intermediate XML format.

By converting first to an intermediate format, it not only decoupled the input from direct dependency on Word XML (and hence an architectural dependency on Word 2003), but also allowed vastly simpler code further down the pipeline.

XTool Application

An "XTool Application" is an application targeted for mobile devices which is entirely specified as a set of XML files. These XML files compose the configuration (meta data), indexing, and the content. The Content is the bulk of the data and is represented in "XText" schema XML Files.

XText is a schema designed for describing documents optimized for mobile devices. Its a schema similar to XHTML in concept but much simpler. It removes most of the device assumptions inherent in XHTML (such as font sizes, RGB colors, pixel metrics) as well as simplifies many of the core elements for efficient rendering, and in particular rendering appropriate to the characteristics of mobile devices. There is also inclusion of some concepts particular to the Epocrates mobile architecture, such as inter-application linking and enforcement of consistent look and feel.

On the device, due to severe resource constraints (CPU, RAM and storage), a binary encoded and compressed XML representation is used. This improves resource usage significantly and allows for quick real-time parsing and rendering even on the most constrained devices.

On the server, the serialized text representation of the XML is encoded into the binary format and packaged into the device specific database format ("PDB" files). The application is deployed to the devices as part of a synchronization.

XML Pipeline

The entire process of converting a set of Word documents, to the final output "PDB" format was implemented as an XML Pipeline, primarily in XQuery, but with some java code as well. The end point of XML in this pipeline is an XML file describing the binary layout of the PDB file, which is then converted to binary PDB format by using java code to produce a text file used by an off-the-shelf tool "pdbc" which generate the palm database files. The content of these PDB files is primarily binary encoded XML documents (in XText format) designed and optimized for presentation on mobile devices, along with some indexing information.

The pipeline makes heavy use of intermediate formats. This design is intended to allow multiple insertion points along the pipeline to accommodate content originating from various sources, not only Word documents. In addition, the intermediate formats allow document generation at various stages where appropriate.

The whole pipeline looks something like this Figure 6.

Figure 6: XML Pipeline
[Link to open this graphic in a separate page]

The implementation of the pipeline infrastructure was written in java, making heavy use of Saxon [SAXON] XQuery [XQUERY] implementation for all XML transformations. Various pipeline implementations were considered but discarded because none were a good fit for this process due to the necessity of passing in some computed runtime values. The pipeline process could have been easily implemented as a sequence of individual processes in a scripting environment.

XQuery was used instead of XSLT [XSLT] as a proof of concept that XQuery is well suited for general purpose complex XML transformations in a pipeline. Furthermore, I found that the XQuery language and syntax to be much simpler write, read, and debug then XSLT.

The following outlines the steps in the pipeline.

Doc Conversion

Input files in DOC format are converted to WordXML format using a Word macro. Files already in WordXML file are imported without conversion.

MRC Compiler

The MRC [Mobile Resource Center] Compiler component is responsible for converting content in "MRC" format into "XTool" format. "MRC" format consists the files required to define a "Mobile Resource Center", which is a collection of articles and supplemental information surrounding a single clinical topic.

The output of the MRC compiler is the collection of files in "XTool Format". "XTool" format is the a set of XML files which constitute a single deployable application to the device. The XML files include the documents, indexing, and meta data which describes a single deployable application.

Epoc Word Creation

An XQuery script parses each WordXML format file and produces the "Epocrates Word" XML format.

The resulting files are validated against the Epocrates Word schema.

Documents already in Epocrates Word schema may be imported at this step.

Epocrates Word Schema

The following is the Epocrates Word Schema


<?xml version = "1.0" encoding = "UTF-8"?>
<xsd:schema xmlns:xsd = "http://www.w3.org/2001/XMLSchema">
	<xsd:group name = "text">
		<xsd:choice>
			<xsd:element ref = "B"/>
			<xsd:element ref = "LINK"/>
			<xsd:element ref = "BR"/>
			<xsd:element ref = "BOOKMARK"/>
			<xsd:element ref = "TAG_START"/>
			<xsd:element ref = "TAG_END"/>
		</xsd:choice>
	</xsd:group>
	<xsd:element name = "DOC">
		<xsd:complexType mixed = "true">
			<xsd:choice minOccurs = "0" maxOccurs = "unbounded">
				<xsd:element ref = "P"/>
				<xsd:element ref = "TABLE"/>
			</xsd:choice>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "P">
		<xsd:complexType mixed = "true">
			<xsd:sequence minOccurs = "0" maxOccurs = "unbounded">
				<xsd:group ref = "text"/>
			</xsd:sequence>
			<xsd:attribute name = "style" type = "xsd:string"/>
			<xsd:attribute name = "list" type = "xsd:string"/>
			<xsd:attribute name = "number" type = "xsd:string"/>
			<xsd:attribute name = "level" type = "xsd:integer"/>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "TABLE">
		<xsd:complexType>
			<xsd:sequence>
				<xsd:element ref = "ROW" minOccurs = "0" maxOccurs = "unbounded"/>
			</xsd:sequence>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "ROW">
		<xsd:complexType>
			<xsd:sequence>
				<xsd:element ref = "COL" minOccurs = "0" maxOccurs = "unbounded"/>
			</xsd:sequence>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "COL">
		<xsd:complexType mixed = "true">
			<xsd:choice minOccurs = "0" maxOccurs = "unbounded">
				<xsd:element ref = "P"/>
				<xsd:element ref = "BOOKMARK"/>
			</xsd:choice>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "B" type = "xsd:string"/>
	<xsd:element name = "LINK">
		<xsd:complexType>
			<xsd:simpleContent>
				<xsd:extension base = "xsd:string">
					<xsd:attribute name = "href" use = "required" type = "xsd:string"/>
				</xsd:extension>
			</xsd:simpleContent>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "BR">
		<xsd:complexType/>
	</xsd:element>
	<xsd:element name = "BOOKMARK">
		<xsd:complexType>
			<xsd:attribute name = "name" use = "required" type = "xsd:string"/>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "TAG_START">
		<xsd:complexType>
			<xsd:sequence>
				<xsd:element ref = "ATTRIBUTE" minOccurs = "0" maxOccurs = "unbounded"/>
			</xsd:sequence>
			<xsd:attribute name = "name" use = "required" type = "xsd:string"/>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "TAG_END">
		<xsd:complexType>
			<xsd:attribute name = "name" use = "required" type = "xsd:string"/>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "ATTRIBUTE">
		<xsd:complexType>
			<xsd:attribute name = "name" use = "required" type = "xsd:string"/>
			<xsd:attribute name = "value" type = "xsd:string"/>
		</xsd:complexType>
	</xsd:element>
</xsd:schema>

				

The following is an excerpt from the Microsoft Word document after being saved as XML. The full document is too large to include in this paper. This excerpt shows some of the structure, including one table, and part of a second table. See reference [MSXML] for complete schema documentation.


<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?mso-application progid="Word.Document"?>
<w:wordDocument xmlns:w="http://schemas.microsoft.com/office/word/2003/wordml"
	xmlns:v="urn:schemas-microsoft-com:vml"
	xmlns:w10="urn:schemas-microsoft-com:office:word"
	xmlns:sl="http://schemas.microsoft.com/schemaLibrary/2003/core"
	xmlns:aml="http://schemas.microsoft.com/aml/2001/core"
	xmlns:wx="http://schemas.microsoft.com/office/word/2003/auxHint"
	xmlns:o="urn:schemas-microsoft-com:office:office"
	xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882"
	xmlns:st1="urn:schemas-microsoft-com:office:smarttags"
	w:macrosPresent="yes" w:embeddedObjPresent="no"
	w:ocxPresent="no" xml:space="preserve">
<o:SmartTagType o:namespaceuri="urn:schemas-microsoft-com:office:smarttags"
	o:name="country-region"/>
	<o:SmartTagType o:namespaceuri="urn:schemas-microsoft-com:office:smarttags"
	o:name="State"/>

...




<w:body><wx:sect>
<w:tbl>
 <w:tblPr>
  <w:tblW w:w="10080" w:type="dxa"/>
	<w:tblInd w:w="-612" w:type="dxa"/>
	<w:tblBorders><w:top w:val="single" w:sz="4" wx:bdrwidth="10"
		w:space="0" w:color="auto"/><w:left w:val="single"
		w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
	<w:bottom w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0"
		w:color="auto"/><w:right w:val="single" w:sz="4" wx:bdrwidth="10"
		w:space="0" w:color="auto"/>
	<w:insideH w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
	<w:insideV w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
	</w:tblBorders>
  </w:tblPr>
  <w:tblGrid><w:gridCol w:w="2160"/><w:gridCol w:w="7920"/></w:tblGrid>
  <w:tr>
   <w:tc><w:tcPr><w:tcW w:w="2160" w:type="dxa"/>
  	</w:tcPr><w:p><w:pPr><w:pStyle w:val="Heading1"/>
    <w:rPr><w:sz-cs w:val="28"/></w:rPr></w:pPr><w:r><w:rPr>
	<w:sz-cs w:val="28"/></w:rPr><w:t>Field</w:t></w:r></w:p></w:tc>
   <w:tc><w:tcPr><w:tcW w:w="7920" w:type="dxa"/></w:tcPr>
   <w:p><w:pPr><w:pStyle w:val="Heading1"/>
   <w:rPr><w:sz-cs w:val="28"/></w:rPr></w:pPr>
   <w:r><w:rPr><w:sz-cs w:val="28"/></w:rPr>
   <w:t>Value</w:t></w:r></w:p></w:tc>
  </w:tr>
</w:tbl>
  <w:p/>
<w:tbl>
   <w:tblPr><w:tblW w:w="10080" w:type="dxa"/>
   <w:tblInd w:w="-612" w:type="dxa"/><w:tblBorders>
   <w:top w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
   <w:left w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
   <w:bottom w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
   <w:right w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
   <w:insideH w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
   <w:insideV w:val="single" w:sz="4" wx:bdrwidth="10" w:space="0" w:color="auto"/>
   </w:tblBorders><w:tblLayout w:type="Fixed"/>
   </w:tblPr><w:tblGrid><w:gridCol w:w="2160"/><w:gridCol w:w="7920"/>
   </w:tblGrid>

  <w:tr><w:tc><w:tcPr><w:tcW w:w="2160" w:type="dxa"/></w:tcPr>
   <aml:annotation aml:id="0" w:type="Word.Bookmark.Start" w:name="mrc_id"/>
   <w:p><w:pPr><w:rPr><w:b/><w:b-cs/></w:rPr></w:pPr>
   <w:r><w:rPr><w:b/><w:b-cs/></w:rPr><w:t>MRC ID</w:t></w:r>
   <aml:annotation aml:id="0" w:type="Word.Bookmark.End"/></w:p>
   </w:tc>
   <w:tc><w:tcPr><w:tcW w:w="7920" w:type="dxa"/></w:tcPr><w:p>
   	<w:r><w:t>eo00</w:t></w:r></w:p>
   </w:tc>
  </w:tr>
   <aml:annotation aml:id="1" w:type="Word.Bookmark.Start"
   	w:name="article_id"/>

  <w:tr><w:tc><w:tcPr><w:tcW w:w="2160" w:type="dxa"/>
  </w:tcPr><w:p><w:pPr><w:rPr><w:b/><w:b-cs/></w:rPr>
  </w:pPr><w:r><w:rPr><w:b/><w:b-cs/></w:rPr><w:t>Article ID</w:t>
  </w:r><aml:annotation aml:id="1" w:type="Word.Bookmark.End"/></w:p>
  </w:tc><w:tc><w:tcPr><w:tcW w:w="7920" w:type="dxa"/></w:tcPr>
  <w:p><w:r><w:t>1022</w:t></w:r></w:p>
  </w:tc>
  </w:tr>

   ....

	

As you can see, this is a very tedious XML to follow. A particularly interesting difficulty is the Bookmarks. Bookmarks are identified by start and end tags which can span tables representing an overlapping structure on the document.

The following is the XQuery used to transform a Word document into the Epoc Word Schema. Note that this is not a complete transformation of everything in the Word document, but rather is specifically implemented to extract out as simple a representation as possible that still contains the necessary information for our product. Included in the extraction is parsing of plain text content to identify what "looks like" XML markup and convert that text into actual XML markup (TAG_START, TAG_END). This processing takes up a significant part of the XQuery code.


declare namespace w = "http://schemas.microsoft.com/office/word/2003/wordml";
declare namespace wx = "http://schemas.microsoft.com/office/word/2003/auxHint";
declare namespace aml = "http://schemas.microsoft.com/aml/2001/core";


(: Allowed symbols/chars which are allowed - can be translated to XText :)
declare variable $chars_allowed as xs:integer* :=
( 32 to 126 , 160 to 255 , 338 , 339 , 352 , 353 , 376 , 402 , 710 , 732 ,
   946 , 956 , 960 , 8211 , 8212 , 8216 , 8217 , 8218 , 8220 , 8221 , 8222 , 8224,
   8225,8226 , 8230 , 8240 , 8249 , 8250 , 8364 , 8482 , 8765 ,
   8804 , 8805, 9674, 9826 , 9827 , 9825 , 9824 ,
	937 ,  934 , 916 ,931 , 8592 , 8593 , 8594 ,8595
   );



(:
    Construct an element tag with a name attribute
	Elements can look like
	<TAG>							-> TAG_START
	<TAG/>							-> TAG_START TAG_END
	<TAG attr="attr1" ... >			-> TAG_START
	<TAG attr="attr1"/>				-> TAG_START TAG_END
	</TAG>							-> TAG_END
:)

(: Return a tag string as 4 strings :)
(: "</" , "TAG" , attrs , "/>"  :)

declare function local:split_tag_attr( $tag as xs:string ) as xs:string+
{
	let
	$s := replace( $tag , "^(</?).*$" , "$1" ),
	$e := replace( $tag , "^</?[^/>]*(/?>)$" , "$1" ),
	$t := replace( $tag , "</?([A-Za-z0-9]+).*$", "$1" ),
	$a := replace( $tag , "</?[A-Za-z0-9]+ *([^/>]*)/?>$" , "$1")
	return
		( $s , $t , $a , $e )


};
(: Translate a body like   'foo="bar" spam="bletch"' into a set of ATTRIBUTE elements :)
declare function local:tag_attrs( $body as xs:string ) as element(ATTRIBUTE) *
{
	if( $body eq "" ) then
		()
	else
	for $vs   in tokenize( $body , """ +")
	let $attr := tokenize( $vs , "=" ),
	    $name  := $attr[1],
	    $value := replace( $attr[2] , """" , "" )
	return
		<ATTRIBUTE name="{$name}" value="{$value}"/>
};

(: convert a string which 'looks like' an XML tag into a TAG_START or TAG_END element :)
declare function local:tag_element( $tag as xs:string   ) as item()*
{

	let $data := local:split_tag_attr( $tag )
	return
	if( $data[1] eq "</" ) then
		<TAG_END name="{$data[2]}"/>
	else
	 (
		element TAG_START {
			attribute name { $data[2] } ,
			local:tag_attrs( $data[3] )
		} ,

		if( $data[4] eq "/>" ) then
			<TAG_END name="{$data[2]}"/>
		else
			()
	 )


};


(:
  Return the contents of a single <w:r> element  that has  no links
  if w:rPr/w:b exists (indicates a bold element) then construct a bold element
  otherwise convert to simple text
:)
declare function local:r( $r as element(w:r) )
{
    let $b :=  $r/w:rPr/w:b,
	  	$v := $b/@w:val
	return
	if( not(empty( $b )) and ( empty($v ) or not( $v = "off" ) )) then
	   <B>{local:text($r/w:t)}</B>
	else
    if( fn:exists( $r/w:br ) ) then
	  (  <BR/>,local:text($r/w:t) )
	else
      local:text($r/w:t)
};




(:
 Split a single string containing "<" into multiple strings where every
 	tag-like string is a single string
 e.g.  "foo<B>ar" -> ("foo","<B>" ,"ar")
 :)


declare function local:splittags( $s as xs:string ) as xs:string+
{
   let	$a := substring-before( $s , "<" ),
        $b := substring-after( $s , "<" ) ,
	    $c := substring-before( $b , ">" ),
		$d := substring-after( $b , ">" )
	return
		( $a , concat( "<" , $c , ">" ) ,

		if( exists($d) and contains( $d , "<" ) ) then
			local:splittags($d)
		else $d
		)


};


declare function local:value-except (
  $arg1 as xs:anyAtomicType*,
  $arg2 as xs:anyAtomicType*) as xs:anyAtomicType*
{
  fn:distinct-values($arg1[not(.=$arg2)])
};

(: Format a list of codepoints as a readable character string for purposes of error display :)
declare function local:formatchars($s as xs:integer* ) as xs:string
{
   for $i in $s
   return
   		concat( $i , ":" , codepoints-to-string( $i ) , " ")

};


(: Fix any "funny" characters in a text string replacing them with good
	chars or throwing an error :)


declare function local:fixtext( $t as xs:string ) as xs:string
{
	(: Translation table
	   em-dash  -> "-"
	   word-space(xF020) -> " "
	:)


	let $fixed := translate( $t , codepoints-to-string(( 8722,61472 ))  , "- " ),
	 	$bad := local:value-except( string-to-codepoints( $fixed ) , $chars_allowed )
	return
	   if( empty($bad) ) then $fixed
	   else
	   error( xs:QName('Error') ,
	      concat('Invalid characters found: ' , local:formatchars( $bad ) ) )


};




(: parse text nodes by looking for embedded tag references like <TAG arg="foo"/> :)

declare function local:text( $t as node()? ) as node()*
{
	if( empty( $t ) )
	then
	 	()
	else
	(: If text contains a "<" but doesnt itself equal a "<" then parse as a tag :)
	(: Need special case because a &lt; sign shows up as "<" by itself in a <t> element :)

	if( matches( $t/text() , "</?[a-zA-Z_0-9]+/?>" )  ) then
		for $arg in local:splittags( $t/text() )
		return
			if( matches( $arg , "^</?[a-zA-Z_0-9]+/?>" ) )then
				local:tag_element( $arg )
			else
				text { local:fixtext($arg) }
	else
		text { local:fixtext($t/text()) }

};


(: construct bookmark tags from annotations :)
declare function local:annotation( $a as element(aml:annotation)* ) as node()?
{
	if( $a/@w:type="Word.Bookmark.End") then

	   <BOOKMARK name="{$a/root()//aml:annotation[@aml:id=$a/@aml:id]
	   		[@w:type="Word.Bookmark.Start"]/@w:name}"/>
	else
	   ()
};

(: create a <link> element from a w:hlink word element :)
declare function local:hlink( $a as element(w:hlink)* ) as node()*
{
  <LINK href="{$a/@w:dest}">{local:text($a//w:t)}</LINK>
};



(:
  handle all children of w:p elements
   Special case, handle repeated sequence of w:r elements in order to
   parse out text with embedded XML like syntax
   Word splits up <X>A</X> into  3  w:r elements so we have to find all
   contigous w:r elements and process them together.

   for other elements process them individually then pass the remaining list
   recursively to local:ps

:)

declare function local:ps( $ps as element()* ) as node()*
{
	for $e in $ps
	return
	   typeswitch( $e )

	   case $r as element(w:r)  return
		     local:r( $e )

		case $a as element(aml:annotation)
		   return ( local:annotation( $a )  , local:ps( $e[ position() > 1 ] ) )

		case $l as element(w:hlink)
		   return ( local:hlink( $l )  , local:ps( $e[ position() > 1 ] )  )

		default
  			return $e//text()

};

(:
  Return the attributes for a paragraph
   if style is specified add style="style"
   if listtype is specified then add list="listtype"
   if listtype is specified then add level="level"

 :)

declare function local:p_attrs( $style as xs:string? ,
		$listtype as xs:string? , $level as xs:int? ,
		$number as xs:string?) as node()*
{
     ( if( empty($style) ) then () else attribute { "style" } { $style } ,
	   if( empty($listtype) ) then () else attribute { "list" } { $listtype } ,
   	   if( empty($listtype) ) then () else attribute { "level" } { $level } ,
	   if( $listtype eq "numbered" ) then attribute { "number" } { $number } else ()

	   )

};

(: return the list style, either "bullet" , "numbered" or () :)

declare function local:list_style( $lpr as element(w:listPr) ? ) as xs:string?
{
   if( empty($lpr) ) then
      ()
   else
  	let
		(: Find the <w:list> associated with this paragraph :)
	    $list := $lpr/root()//w:list[@w:ilfo=$lpr/w:ilfo/@w:val] ,

	    (: find the list def associated with the list :)
		$listdef := $lpr/root()//w:listDef[@w:listDefId=$list/w:ilst/@w:val],

		(: find the list text format for level 1 for this list :)
		$listtxt :=  $listdef/w:lvl[1]/w:lvlText/@w:val

		return
		if( empty($listtxt) ) then ()
		else
		(: if format contains a % then assume numbered lists :)
		if( fn:contains( $listtxt , "%" ) ) then "numbered"
		(: otherwise bulleted :)
		else "bullet"

};


(:
  Return the contents of a <w:p> element
  If a style is given then include the style=  attribute
  If a list type then include the list= and level= attributes
  If an "Ignore*" type style then ignore entirely
:)


declare function local:p( $p as element(w:p) ) as element()?
{
    let $style   := $p/w:pPr/w:pStyle/@w:val,
	    $listtype  := local:list_style($p/w:pPr/w:listPr),
		$level   :=  $p/w:pPr/w:listPr/w:ilvl/@w:val ,
		$number  :=  $p/w:pPr/w:listPr/wx:t/@wx:val
	return
		(: if style starts with "Ignore" then ignore this entire paragraph :)
	   if( fn:starts-with($style,"Ignore") ) then
	      ()
	   else
	     (: construct a <p> with optional attributes :)
	     element {"P" } { local:p_attrs( $style , $listtype , $level , $number ) ,
		 	local:ps($p/*)}
};

(: Return a table column :)
declare function local:col( $c as element(w:tc) ) as element()
{

   <COL>{local:content($c/*) }</COL>
};

(: Return a row element :)
declare function local:row( $t as element(w:tr) ) as element()
{
   <ROW>{

   		for $tc in $t/w:tc
		return local:col( $tc )


   } </ROW>
};

(: Parse and return a table element :)
declare function local:table( $t as element(w:tbl) ) as element()
{
    <TABLE>
	  { for $tr in $t/w:tr return local:row( $tr ) }
	</TABLE>
};

(: Parse any block content :)
declare function local:content( $s as element()* ) as element()*
{
   for $e in $s
   return
   typeswitch( $e )
   case	$p as element(w:p)
        return local:p( $p )
   case $t as element(w:tbl)
        return local:table( $t )
   case $ss as element(wx:sub-section)
        return local:content( $ss/* )
   case $a as element(aml:annotation)
		   return local:annotation( $a )
   default return ()

};




(:
  Top level XQuery element, returns a DOC element with all its contents
 :)

<DOC>
	{
	for $s in /w:wordDocument/w:body/wx:sect
	return
	local:content($s/*)

	}
</DOC>

This is an example of the Word document in Figure 5 after it is converted into the Epocrates Word Schema.


<?xml version="1.0" encoding="US-ASCII"?>
	<DOC>
		<TABLE>
			<ROW>
			    <COL><P style="Heading1">Field</P></COL>
				<COL><P style="Heading1">Value</P></COL>
			</ROW>
		</TABLE><P/>
		<TABLE>
			<ROW>
				<COL><P><B>MRC ID</B><BOOKMARK name="mrc_id"/></P></COL>
				<COL><P>eo00</P></COL>
			</ROW>
			<ROW>
				<COL><P><B>Article ID</B><BOOKMARK name="article_id"/></P></COL>
				<COL><P>1022</P></COL>
			</ROW>
			<ROW>
				<COL><P style="Heading1">Version<BOOKMARK name="version"/></P></COL>
				<COL><P style="Heading1">1</P></COL>
			</ROW>
			<ROW>
				<COL><P style="Heading1">Last Update Date<BOOKMARK name="last_update_date"/>
					(YYYY-MM-DD)</P></COL>
				<COL><P style="Heading1">2007-06-11</P></COL>
			</ROW>
			<ROW>
				<COL><P style="Heading1">Source<BOOKMARK name="source"/></P></COL>
				<COL><P style="Heading1">MedPage Today</P></COL>
			</ROW>
			<ROW>
				<COL><P style="Heading1">Sub-Source<BOOKMARK name="sub_source"/></P></COL>
				<COL><P style="Heading1"/></COL></ROW>
			</TABLE>

			....

				

Article creation

An XQuery script converts each article file from Epoc Word format into the MRC Article format. The resulting files are validated against the MRC Article schema.

Article Schema

The Article schema represents a single MRC article in a direct structured format. This schema is fairly simple and allows for allows for a minimum of mixed text, in addition to the structural markup

The following is the Article schema


<?xml version = "1.0" encoding = "UTF-8"?>
<xsd:schema xmlns:xsd = "http://www.w3.org/2001/XMLSchema">
	<xsd:group name = "para">
		<xsd:choice>
			<xsd:element ref = "P"/>
			<xsd:element ref = "M"/>
			<xsd:element ref = "L"/>
		</xsd:choice>
	</xsd:group>
	<xsd:group name = "text">
		<xsd:choice>
			<xsd:element ref = "B"/>
			<xsd:element ref = "LINK"/>
			<xsd:element ref = "BR"/>
		</xsd:choice>
	</xsd:group>
	<xsd:element name = "ARTICLE">
		<xsd:complexType>
			<xsd:sequence>
				<xsd:element name = "FULL_TITLE" type = "xsd:string"/>
				<xsd:element name = "SHORT_TITLE" type = "xsd:string"/>
				<xsd:element ref = "EXPERT_COMMENT"/>
				<xsd:element ref = "SHORT_DESC"/>
				<xsd:element ref = "PDA_TEXT"/>
			</xsd:sequence>
			<xsd:attribute name = "article_id" use = "required" type = "xsd:integer"/>
			<xsd:attribute name = "clinical_significance_rating" use = "required">
				<xsd:simpleType>
					<xsd:restriction base = "xsd:integer">
						<xsd:maxInclusive value = "3"/>
						<xsd:minInclusive value = "0"/>
					</xsd:restriction>
				</xsd:simpleType>
			</xsd:attribute>
			<xsd:attribute name = "content_category" use = "required" type = "xsd:string"/>
			<xsd:attribute name = "date_of_article" use = "required" type = "xsd:date"/>
			<xsd:attribute name = "last_update_date" use = "required" type = "xsd:date"/>
			<xsd:attribute name = "target_epoc_publish_date" use = "required"
					type = "xsd:date"/>
			<xsd:attribute name = "mrc_id" use = "required" type = "xsd:string"/>
			<xsd:attribute name = "version" use = "required" type = "xsd:integer"/>
			<xsd:attribute name = "source" type = "xsd:string"/>
			<xsd:attribute name = "sub_source" type = "xsd:string"/>
			<xsd:attribute name = "url" type = "xsd:string"/>


		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "EXPERT_COMMENT">
		<xsd:complexType>
			<xsd:sequence>
				<xsd:element ref = "P" minOccurs = "0" maxOccurs = "unbounded"/>
			</xsd:sequence>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "SHORT_DESC">
		<xsd:complexType>
			<xsd:sequence>
				<xsd:element ref = "P" minOccurs = "0" maxOccurs = "unbounded"/>
			</xsd:sequence>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "PDA_TEXT">
		<xsd:complexType>
			<xsd:choice minOccurs = "0" maxOccurs = "unbounded">
				<xsd:group ref = "para"/>
			</xsd:choice>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "P">
		<xsd:complexType mixed = "true">
			<xsd:sequence minOccurs = "0" maxOccurs = "unbounded">
				<xsd:group ref = "text"/>
			</xsd:sequence>
			<xsd:attribute name = "list" type = "xsd:string"/>
			<xsd:attribute name = "number" type = "xsd:string"/>
			<xsd:attribute name = "level" type = "xsd:integer"/>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "B" type = "xsd:string"/>
	<xsd:element name = "LINK">
		<xsd:complexType>
			<xsd:simpleContent>
				<xsd:extension base = "xsd:string">
					<xsd:attribute name = "href" use = "required" type = "xsd:string"/>
				</xsd:extension>
			</xsd:simpleContent>
		</xsd:complexType>
	</xsd:element>
	<xsd:element name = "BR">
		<xsd:complexType/>
	</xsd:element>
	<xsd:element name = "M"/>
	<xsd:element name = "L"/>
</xsd:schema>

					

The following is a sample of the same article from Figure 4 after conversion to the Article schema


<?xml version="1.0" encoding="US-ASCII"?>
<ARTICLE
	mrc_id="eo00"
	target_epoc_publish_date="2007-06-19"
	clinical_significance_rating="2"
	article_id="1022"
	content_category="Conference Highlights"
	date_of_article="2007-05-29"
	version="1"
	last_update_date="2007-06-11">
	<FULL_TITLE>ATS: XDR-TB Gains Ground Around the World</FULL_TITLE>
	<SHORT_TITLE>ATS: XDR-TB Gains Ground</SHORT_TITLE>
	<EXPERT_COMMENT><P>It should be emphasized that the total number of
		XDR TB cases in the US
		from 1993-06 was 49 or about 3 per year.
		Of these, 25 (52%) were foreign born.
		The big problem with XDR TB is in countries where TB is endemic -
		especially in those where HIV rates are also high.
		- John Bartlett - </P>
	</EXPERT_COMMENT>
	<SHORT_DESC>
		<P>Rates of XDR-TB continue to increase, and the disease now
		requires a global effort for control.</P>
	</SHORT_DESC>
	<PDA_TEXT>
		<P>(MedPage Today)</P><P>May 29, San Francisco -
			Extensively drug resistant tuberculosis

	...

Figure 7: Sample MRC Article page on the mobile device
[Link to open this graphic in a separate page]

XTool Compiler

The XTool compiler takes a set of XML files describing an "XTool Application" (an XML config file, and XML XText content files) and creates the binary device "PDB" files. It does this through a pipeline that first encodes the content files from the XText schema into a binary encoded XML format, which is then output as XMLPDB format. The XMLPDB format is an XML document which describes the exact binary record layout for the PDB files. This format is then converted to "pdbc" format which is the text format required by the pdbc [PDBC] compiler, and then compiled into binary PDB files.

XText content

Each page of the XTool application data is specified in XML using the XText schema. The following is the XText XML content of the same article in Figure 4 after conversion from the Article format to the XText format used by XTool. Figure 7 shows what this page looks like when deployed to the mobile device and rendered by the MRC application.


<?xml version="1.0" encoding="US-ASCII"?>
<XTEXT lr_hpad="0" tb_vpad="0">
  <HEADER lr_hpad="0" tb_vpad="0">
	<COMBOBOX name="topnav">
		<OPTION value="Resource Center Home" page="index"/>
		<OPTION value="Conference Highlights" selected="1" page="conference"/>
		<OPTION value="Clinical News" selected="0" page="news"/>
		<OPTION value="Scientific Abstracts" selected="0" page="pubmed"/>
		<OPTION value="MobileCME Links" selected="0" page="cme"/>
		<OPTION value="Resources" selected="0" page="resources"/>
	</COMBOBOX>
	<HR/>
	</HEADER>
	<PAGE lr_hpad="1" tb_vpad="0">
		<P align="left">
			<FONT color="black" bold="true">
				ATS: XDR-TB Gains Ground Around the World
			</FONT></P><HR/>
		<P align="left">
			<IMGBUTTON popup="star" src="two_star" name="star"/>&#160;29May07</P>
		<P>Rates of XDR-TB continue to increase, and the disease now requires a
			global effort for control.</P>
		<HR/>
		<P>(MedPage Today)</P><P>May 29, San Francisco -
			Extensively drug resistant tuberculosis (XDR-TB) continues to be

	...

PDB Compiler

The XMLPDB files are transformed into "pdbc" format. Then pdbc is executed to create the binary PDB files from the pdbc text files. An application configuration file (app.xml) is also created to pass on the meta data required for deploying an XTool application.

The resulting files are the final binary format used to deploy to the sync server and synced to the devices.

Postmortum - Lessons learned.

The design outlined in this paper were evolved as much by our failures as our successes. Our failures were both human and technical.

On the human side were incorrect assumptions about how authors really use Word processing tools. The best technical solutions failed miserably when the people don't use them in the expected manner. Adopting the technical solutions around how people actually work, rather then trying to get them to change their work habits has been the most successful strategy.

On the technical side, using complex Word macros (VBScript) turned out to be a big mistake. Especially when the macros are a required part of the authoring environment. Attempting a monolithic single program which converted Word to the final XML format was also very challenging, difficult to debug and left no room for future expansion. Implementing the process as a pipeline, instead of a single transformation, resulted in a much better solution. The use of XQuery exclusively for the XML transformations worked very well. I was tempted in many occasions to use the Saxon feature of calling into native Java code in the XQuery scripts, but found that it was unnecessary. XQuery alone provided 100% of the needed functionality for every phase of the pipeline. The java and script code used to integrate the pipeline steps was only needed to gather input from the environment, and to call the processes necessary for the final conversion from XML to the device format.

Conclusion

Epocrates has successfully adopted an XML oriented pipeline architecture bridging two processes which are frequently difficult or inefficient to integrate with XML - authoring in Microsoft Word and rendering rich content on mobile devices. Effectively addressing two "ends" of the pipeline are critical for a successful XML implementation, as fundamental arguments against using XML technology are difficulty in creating the content in XML format, converting content to XML and rendering XML data in formats specific to the application. In the problem space for Epocrates, allowing content to be authored in Word solves the problem of content authoring and conversion, and efficient encoding and rendering of XML on mobile devices addresses the output concerns. This allows the entire workflow to be based on an XML pipeline architecture which has proven to have great advantages.

As an implementation "proof of concept", it has been demonstrated that XQuery can serve as a general purpose XML Transformation language in a production environment, where otherwise XSLT would have been considered as the only candidate.


Bibliography

[DESIGN] Erich Gamma, Richard Helm, Ralphy Johnson, John Vlissides, "Design Patterns, Elements of Reusable Object-Oriented Software", Addison Wesley Longman, Inc., 1995

[LEE-XML] David A. Lee, "XML encoding techniques for storing XML data on memory limited (mobile) devices", http://2006.xmlconference.org/programme/presentations/13.html, Dec 2006, XML2006 conference,

[MSWORD2003] Microsoft, "Microsoft Office Word 2003 XML Software Development Kit", http://msdn2.microsoft.com/en-us/library/aa223586(office.11).aspx, Referenced June 2007

[MSXML] Microsoft, "XML in Microsoft Office Word 2003", http://msdn.microsoft.com/msdnmag/issues/03/11/XMLFiles, Referenced June, 2007

[PDBC] Eric Obermuhlner, "PDBC (Palm DataBase Compiler/Decompiler", http://obermuhlner.com/public/Projects/Palm/PDBC/ Referenced June, 2007

[SAXON] Michael Kay, "SAXON The XSLT and XQuery Processor", http://saxon.sourceforge.net/ Referenced June, 2007

[XMill] Hartmut Liefke, "XMill An Efficient Compressor for XML"http://www.liefke.com/hartmut/xmill/xmill.html, Referenced June 2007

[XQUERY] W3C, "W3C XML Query (XQuery)", http://www.w3.org/XML/Query/, Referenced June, 2007

[XQUERY-EXP] Don Chamberlin, Denise Draper, Mary Fernandez, Michael Kay, Jonathan Robie, Michael Fys, Jerome Simeon Jim Tivy, Philip Wadler, Howard Katz Editor, "XQuery from the Experts, A Guide to the W3C XML Query Language", Pearson Education Inc., 2004

[XSLT] W3C, "XSL Transformations (XSLT) Version 1.0", http://www.w3.org/TR/xslt, Referenced June, 2007



From Word to XML to Mobile Devices

David Lee [Epocrates, Inc.]