Mail Markup Language, The Back Story

Preface

The following story is written by Mail Markup Language inventor, Austin Cheney, on 23 April 2014 about his experiences immediately leading up to the invention of the language.

Graduation

In May of 2006 I was employed as a web designer for a local firm in the small, and somewhat isolated, town of Stephenville, TX and graduated from Tarleton State University. Shortly after graduation I was laid off because the business was failing. At this time I had no awareness there was a concept called separation of concerns and all my HTML code was primarily written to create marketing presentation and visual layouts. This was typically accomplished by abusing HTML tables, often through deeply nested tables, and mutilation of fonts to create beautiful presentation at the cost of everything else.

At this time the web browser Internet Explorer (IE) 6 owned 85% of market share. In 2001 the combination of IE5 and the new IE6 dominated with close to 97% market share. The common practice when I started learning web design and presentation in late 1997 was to design for IE and then fudge it just enough so that it did not fail in Netscape. By 2006 it was common for web developers to only design for IE6. It is necessary to keep in mind that web browsers at this time were far less uniform than today and cross-site experiences took a substantially larger effort to complete.

In past years I had spent a fair amount of time in online text-based chat with IRC, particularly the Open Projects Network (renamed to Freenode.net). I had been away from IRC in roughly the 2 years immediate, but since I no longer had employment and was struggling to find a job I had time to explore IRC again. It is in this time that I became aware of Cascading Style Sheets (CSS) and how to fully separate presentation from HTML. More importantly still, in separating presentation from HTML I learned that HTML could become an expressive content description mechanism. I had been writing HTML for years but had never really thought about it a purely semantic expression.

Digital Alchemy

In September 2006 a small marketing firm called Digital Alchemy (DA) took a chance on me. DA is a consumer resource marketing firm specializing on the high-end 4 start and above hospitality industry. At the time all their clients were extremely high-end vacation resorts or luxury urban hotels. Most of their revenue was generated from data mining property reservation data and returning that data back to the property in a more meaningful way that allows their property management to better understand their clientèle.

DA was interesting for a number of reasons. At this time DA was still operating in start-up mode even though it was founded 7 years prior. Shortly before my hiring the company signed its largest contract to date with Rosewood Hotels, which allowed the company to start planning its movement from a start-up into a thriving business. More interesting still is that almost all of their client products were HTML documents designed for transfer across email.

Email is a challenging medium compared to the web. Firefox was an emerging web browser that had climbed to about 8% market share. Safari existed, but Safari had no market share since not many people were using Macs at this time and the first IPhone was still months from release. Although it might have been more challenging to write cross-browser products for the web at this time there were really only two web browsers that demanded consideration. Furthermore, on the web you only had to design for the browser. If the document worked properly in a particular browser version one time then it was fairly safe to think it would continue to perform in a like manner upon repeated viewings. These assumptions often proved false in email. At any given time there are several dozen email clients on the market with each supporting a wild diversity of standards and technology.

Upon my hiring at DA it was clear what my initial mission was: secure the Rosewood account. DA had a designer already, but he was not a dependable employee and his work was of substandard quality. Rosewood used IBM's Lotus suite for all their email and messaging. If the presentation of email documents did not look good in their software then DA was going to lose this account. It should be noted that Rosewood Hotels was the biggest name in the business. Reputations were made and broken by servicing that brand. DA could not afford for this contract to fail and so far nobody had been able to solve this problem.

After establishing a remote access connection to a computer with their email software/system and doing some research to determine the conventions supported by the Lotus software I was able to design a solution. It was possible to design and code documents with their branding in a way that presented perfectly in their software. The solution was to separate presentation out with CSS, using only a selected set of safe CSS properties, and then delivering the CSS inside the document inline on the HTML elements using a style attribute.

In the Rosewood case the solution was a separation of concerns and then a reintegration of those concerns in a manner the software accepted. This solution would prove to be only half acceptable as we later discovered with web mail. It should be noted email clients, and in some cases even email servers, mutilate HTML documents each time the documents are received for processing. This is especially true and catastrophic for web mail where the document is mutilated on a web server to ensure that it displays in an email client that is then embedded within a web page sent to a web browser. The web mail problem took a large amount of trial and error, but eventually I was able to solve for this problem as well.

In DA I learned to write mostly accessible HTML with fully separated CSS for layout and visual presentation. Email proved to be the school of hard knocks. Designing for the web would later proved to be effortless child's play after solving the challenges of email. I learned that even in more challenging environments I did not have to sacrifice document semantics or accessibility to achieve a desired presentation. This continually reenforced principle would become a primary design consideration of Mail Markup Language.

Travelocity

A college buddy hooked me up with a design position at Travelocity. My first day was 1 October 2007. On 3 October I was handed my first assignment: code an internal email newsletter for the CEO. This was technically simple since the visual layout was clean and primitive. The hard part was not anything related to actual skilled labor. The primary challenge was expressing that email is not the web. There are different constraints and different challenges.

In my prior experience I had learned there was an amazing potential for HTML/CSS code to be corrupted and that the depth of this corruption was directly related to the differences and quantity processing software agents. I attempted to educate my Travelocity design boss that even though everybody in Travelocity were basically using the same email server and same email client the document would be corrupted anyways so long as the document was sent with any software that attempted to process the contents therein (and still possibly corrupted due to processing on the email server). Most of these warnings were ignored, because this document contained web technologies and clearly the web does not work this way.

When the CEO sent this newsletter document from her personal email client the layout was in fact corrupted. Fortunately this corruption was minor and did not impact the content in the document.

Once my new boss identified this corruption she immediately questioned why I failed to design the document properly. Although I provided sufficient warnings that such corruption was probable if not sent properly and fully articulated the reasons why this corruption occurred I still failed and it was clearly my fault. All technical merits of the prior stated warnings were ignored and all explanations of the corruption were also ignored. At this moment I had an epiphany.

This episode was deeply frustrating. I had just started a new job at a major corporation with a first project of incredibly high visibility and it flopped despite sufficiently warning others of the risks. The problem and the answer became immediately clear: HTML is designed for a sessionless broadcast protocol (HTTP) and is therefore incompatible with email (SMTP).

Writing a Language

Effort and Inspiration

I churned on my Travelocity failure for about a week until 12 October 2007 when I put hands on keyboard with a mission to write a new language. I wrote this language on my own time during weekends and nights over a span of a year and half. One weekend a month I would drive the 300 miles from my apartment in Haltom City, TX to my Army Reserve job in San Antonio. I remember using this commuting time to reflect heavily upon my design decisions and critically challenge my approaches.

The more I worked on this language and made design decisions about how to solve various structure or descriptive problems the more frustrated and angry I got at HTML. I was still writing HTML full time for my day job and each and every frustrating moment became yet another cause for critical examination in my own language. Many of my design decisions were inspired by learning what to do from examination of taxonomies described by HTML and various XML languages and similarly by learning what not to do by looking at the practice upon those technologies.

The primary intention of this language was to describe content under the different transmission context of the SMTP protocol than is available with the web's HTTP protocol. However, after a bit of effort, two ultimate goals emerged.

Conversations and Email Threads

My first goal was to account for the complexity of conversations as presented by email threads. The current practice of email was to include previous communication instances at a lower point in the document when replying or forwarding a communication. This allowed a conversational context, but it also made the document more complex in a way not supported by XML or HTML.

Existing markup languages were designed to represent a single data context where everything inherits from a root node. This is problematic because a conversation comprises multiple contributions where each comprises a unique data context and data fragments uniquely described by their immediate context. This is further problematic in that if the data may be described by separate contexts then enhancements to that data, particularly presentation, must be separately described in a manner that does not interfere with other communication instances.

To solve this problem I created a new means of separation. The idea is to specify context separations by a given tag name in the markup code. The approach allows a collection of isolated data islands as siblings in a single document under a single root node. The approach provides processing separation in a manner similar to separate web pages, somewhat like an HTML iframe element, but without the need for separate documents. The solution allows interpretation of data by context across a plurality of connected communication instances without need to expose this data to a third party. This enables any user to conveniently perform heuristic analysis with extreme precision against their own data without exposing that data to other people, software services, or other intermediaries. This analysis may be applied to data resident in a given document and also be used to reason about data that is not present, such as models of data flow between users in a conversation, and still maintain privacy in a way other markup languages cannot.

Solving for Accessibility

The second design goal was to solve the problem of accessibility. In HTML accessibility is a challenge for three reasons:

I solved for the final bullet point, sloppiness, by defining the language against a schema and requiring validation against the schema prior to transmission. This was absolutely necessary to prevent document corruption due to poorly written markup from a single contributor, which would have second and third order consequences. Through establishing of a firm foundation from well-formed syntax and the structure requirements of the schema all manner of other data extensions become possible, where they are otherwise entirely impractical or unavailable on the web.

I solved for the first two bullet points by forming my own solution to structuring content. I created a three tier system. All content description elements would be assigned to one and only one of those tiers, which are: complex-blocks, simple-blocks, and inline elements. The definitions for these tiers is intentionally specific and short. A complex-block is any element capable of containing itself, other elements in the complex block group, or simple block group elements per the element's definition. A complex block element will not contain text content or inline elements. A simple block element can only contain text and inline elements. Inline elements can only contain text.

The three tier system allowed for a structuring of elements in way that separates text content from the structure describing it, but more effective descriptions are needed than what is available on the web. I needed to create elements that could describe text content in a manner that is open to extension but simultaneously limit that description to as few elements as possible to prevent confused interpretations. I chose to define elements that described a facet of linguistic or data table structures in the broadest sense as well as a few elements that described text in a manner that conveys linguistic utility. Everything more elaborate could be supplied to a role attribute to supply more specific contexts.

I discovered through practice in writing this language and by exercising element structures according to my three tier system it is possible to solve the accessibility problem for visually impaired. In fact, this problem became trivial to solve in a manner that is immediate, persistent, durable, and enforceable. I also found that this solution can be easily reproduced in other languages and approaches so long as the magic three ingredients were supplied: a three tier structure, adherence to a well formed schema, and well defined elements that describe only a taxonomy without being too specific.

The Emergence of HTML5

While I was writing my small language a group called the WHATWG began making noise about a language they were writing called HTML5. Their language was not based upon anything programmatic. At least the previous HTML specifications were based upon SGML which served as a reenforcing construct to resolve ambiguities of syntax. HTML5, however, based upon plain text and focused most heavily upon media.

The development of HTML5 was particularly painful and agonizing to watch. The editors would openly flame any critical challenges that were raised in the most public manners available with little regard for even the smallest shred of professionalism. It immediately became clear that the hostility from the HTML5 leadership drove field experts far away, which means decisions by committee whose members were completely unprepared for the decisions before them.

Through the decaying decision making process the direction of this language became increasingly clear. Through the addition of features the objectives of HTML5 appeared to target marketing and media. While these qualities may certainly be business objectives of the web they certainly are not objectives of the technology. Even more infuriating is that HTML5 made little or no effort to fix the existing problems of HTML that were already identified by the W3C, and many cases made these problems substantially worse.

I was fortunate to be writing my language at that exact moment, because now I had another example of what not to do and this other example gained a substantial following quickly. I could see quickly enough this language had fallen into the same traps that earlier versions of HTML had fallen into. This was a language of design by committee that served the interests of the editing few and favored the loudest ideas without merit or examination of those ideas in practice. While I was being careful and moving slow to carefully and critically examine foundational approaches the HTML5 movement could be characterized as a fast and feature rich movement.

Through the experiences I had gained in examination of semantic technologies and language formation I realized at around the time I was offered patent protection that I could make predictive assessments upon the technology, as well as competing technologies like HTML5. I am writing this document 5 years after the language was submitted for patent protection. The patent application is still pending at this time. This time has allowed me the opportunity to learn to program, which is a skill I did not have when I started writing this language. This time has also allowed me the opportunity to test my predictions and opinions of HTML5 implementation in the wild.

History of Protection

When I started writing this language I was new to the corporate world. I had no idea of matters relating to intellectual property, protection, or the rationale behind such. Fortunately, shortly after I started writing this language Sabre, the parent company of Travelocity, hired an intellectual property attorney. He met with my design team in the spring of 2008 to educate the team about protecting the company's IP from accidental disclosure.

Shortly after this meeting I with the attorney I asked what should happen with my pet project. The attorney immediately offered to supply a provisional patent to protect the technology upon realized I had not showed to anybody. I was afraid to show the language to anybody for fear the ideas would be stolen and abused before the language grow organically in the marketplace. I knew in this disclosure that I was sacrificing ownership to Travelocity, but I figured they could probably take ownership anyways. At least this way the ideas in the technology would be protected. I predicted my idea of separation would eventually adopted into the web, where it is harmful to HTTP, for use to either increase traffic to advertisement or request advertisement assets in a manner of greater ease for lesser competent web developers. I also accurately predicted that my accessibility solution would be completely ignored by the web.

Only a few months later the economy collapsed into massive recession, sometimes called the little depression. My attorney was laid off, as well as many other people. Had I not met with the attorney when I did the language would likely have never been protected at all.

In spring 2009 the provisional patent was set to expire. I made a plea to the quarterly Sabre patent committee for funding of full patent application and legal fees. There were three technologies evaluated by the committee that quarter before my little language. My invention was the only one to win protection that quarter and the voting was unanimous. The application was filed with the USPTO on 19 April 2009, which is also the date of expiration on the provisional patent. On 22 October 2009 the patent application was published online by the USPTO. The language remains under examination at this time and can be reviewed online.