A Standards Quality Case Study: W3C

Since I gave a presentation on this topic at the OFE Conference in Geneva at the end of February I have meant to post something about it here. As some of us stated before, if anything, the OOXML debacle has achieved one thing: raising awareness for the need for higher quality standards and standards development processes.

Introduction

Having been primarily involved in W3C both as a staff member and a member company representative I had grown to expect a certain quality level which has led me to be genuinely baffled by the whole OOXML experience. I just didn’t know how superior the W3C process was compared to that of ECMA and ISO/IEC. I just didn’t know those organizations had processes which are so broken that they would allow such a parody of a standards development to take place and such a low quality specification to be eventually endorsed as an international standard.

There have been discussions within the W3C for a long time as to whether it should seek to become a PAS submitter and adopt a policy to systematically submit its standards to ISO/IEC. I used to think it should. I no longer think so. The W3C process is so superior to that of ECMA and ISO/IEC, it’s these organizations that need to learn from W3C and those who are working for the W3C standard label to be recognized at an international level in its own right have all my support.

Ecma’s value proposition vs W3C’s core principles

Let’s look at what differentiates W3C from these organizations by first having a look at Ecma’s stated value:

A proactive, problem solving experts’ group that ensures timely publication of International standards;

Offers industry a “fast track“, to global standards bodies, through which standards are made available on time;

Balances Technical Quality and Business Value:

• Quality of a standard is pivotal, but the balance between timeliness and quality as well: Better a good standard today than a perfect one tomorrow!
• Offers a path which will minimize risk of changes to input specs
• Solid IPR policy and practice

Ecma can be viewed as a reconfigurable hub of TCs

The insistence on time, fast track, business value, minimal risk of changes over quality certainly strikes me as odd. Contrast this with some of the key characteristics of W3C taken from various parts of its documentation:

Mission: To lead the World Wide Web to its full potential by developing protocols and guidelines that ensure long-term growth for the Web.

W3C refers to this goal as “Web interoperability.” By publishing open (non-proprietary) standards for Web languages and protocols, W3C seeks to avoid market fragmentation and thus Web fragmentation.

A vendor-neutral forum for the creation of Web standards.

W3C Members, staff, and Invited Experts work together to design technologies to ensure that the Web will continue to thrive in the future, accommodating the growing diversity of people, hardware, and software.

Although W3C is a consortium which for a large part is funded by its members, the staff led by Tim Berners-Lee has a clear understanding that its mission goes far beyond that of merely satisfying its members. It is working for the benefit of all with a long term vision.

Because of this W3C is more open than many other organizations. One such evidence is the notion of invited experts that was introduced very early on and that allows non members subject-matter experts to participate in the development process. Because of this it also favors quality over time, knowing that while publishing standards faster might serve some short term financial interests it is typically detrimental to the overall stability of the web and contrary to a smooth evolution that will benefit the greater community in the long term.

This is of course not without creating some tensions between its staff and its members at times but, to its credit I think the staff has been mostly successful at balancing the various forces at play so that no single interest takes priority over general interest. This was true for instance when it adopted a patent policy which favors Royalty Free licensing, forcing major vendors, often more stuck in their old ways than necessarily fundamentally against it, to reconsider how they manage their IP with regard to standards.

W3C’s standards development process

Looking at the W3C standards development process also reveals some key characteristics that are fundamental to achieving its greater mission. The typical development of a W3C “standard” – officially called “recommendation” – looks something like this:

1. Member or Team Submission
2. Development of a charter / Creation of a Working Group
   • Vote from Members + call for participation
3. Publication of Member-only and Public Working Drafts (WD).
4. Last Call announcement.
   • WG believes all requirements are fulfilled
5. Publication of a Candidate Recommendation (CR)
   • Call for implementations
6. Publication of a Proposed Recommendation (PR).
   • Call for review
7. Publication as a Recommendation (REC).

It is particularly important to note that contrary to Ecma, submissions to W3C in no way constrain what is eventually produced as a standard, and that no guarantee is given regarding how much can be changed. In fact, quite the opposite is said to be expected. Yet, I’ve never heard anyone claim that any W3C standard developed from a submission didn’t turn out to be better than the original submission.

It is also worth noting that several phases of the process stress the need for reviews by various interested parties, going from a fairly small group to an ever bigger community as the level of confidence increases over time and the specification gets closer to final approval.

Also worth noting is the “Candidate Recommendation” stage. I’m happy to say that I, along with Lauren Wood then chair of the DOM Working Group, am at the source of the introduction of this step in the W3C’s standards development process. The idea behind it is simply to stress the need for implementation experience and to ensure that specifications do not move forward unless they are backed by actual implementations demonstrating that the specification achieves its stated goal.

When first introduced, the success criteria for this phase merely relied on having for each feature of the specification a couple of vendors reporting successful implementation. Over time the bar has been raised time after time to now going as far as holding “interop fests” during which implementations from various vendors are tested against each other.

Contrast this with Ecma and ISO/IEC publishing international standards without even a single claim of successful impementation from anyone…

More striking yet, is the alternate paths a specification may follow within W3C:

• Alternate ending
  Working Group Note
• Return of a Document to a Working Group for Further Work when:
  • the Working Group makes substantive changes to the technical report at any time
  • the Director requires the WG to address important issues raised during a review or as the result of implementation experience.

When not enough implementation experience can be gathered after a while the specification is basically parked aside and recorded as a “Note” rather than let through as a “recommendation” or standard.

Any time significant changes occur or issues are found the document is sent back to the beginning. This is simply because it is well understood that 1) all the checks that were made all along may be jeopardized by any significant changes, and 2) any issue found may require significant changes leading to 1). In practice this doesn’t always mean a lot more time being spent. Indeed, if the changes turn out not to raise any particular problems the document will go that much faster through every step the second time around. But this way, no chances is taken.

Contrast this with the ISO/IEC Fast Track process which allowed OOXML to be modified in ways no one could even fully understand and which went its merry way to final vote without even a final document to show for.

W3C’s decision process

Another key differentiator of W3C is its decision process which I’ve talked about in my blog entry called Can you live with it?

• “Consensus is a core value of W3C.”
• Vote is a last resort when consensus cannot be reached.
  • Everyone has one vote (including invited experts)
• Consensus sets the bar higher than a majority vote.
  • Not only ask whether people agree but also whether anybody dissents
  • Practical way to judge the latter is to ask: “Can you live with it?”
  • Can lead to opposite decision

While the notion of “consensus” isn’t that unique I think W3C differentiates itself from other organizations claiming to make decisions by consensus in the way it defines and assesses whether consensus has been reached.

From what I’ve heard of what went on with OOXML many claims of decisions made by consensus I believe would have failed in W3C.

W3C’s constant evolution

Beyond the core principles on which it is founded, the W3C differentiates itself in that it is constantly looking for ways to further improve its process to better achieve its goals.

• Process is constantly evolving to increase quality and openness
• More and more Working Groups are public
• Technical Architecture Group (TAG)
• Based on the belief that the larger the community the greater the standards produced
• Patent policy evolved from RAND to RF

I’ve already talked about the introduction of the “Candidate Recommendation” phase to ensure greater quality. The introduction of the TAG with the mission to ensure that all W3C recommendations follow some key architectural principles and that the sum of all of them constitute a consistent set is another example of how the W3C evolved for the better.

I’ve already talked about the notion of invited experts ensuring greater input and more openness. Allowing its Working Groups to be opened to the public was yet another bold move from W3C. This was feared to be detrimental to ensuring a sustained number of members for which one of the incentives of being a member is to do just that: participate in Working Groups. But here again the W3C favored greater openness over its own self interest and from what I understand it is being rewarded in that more and more WGs are becoming public without having generated an hemorrhage of members.

Contrast this with ISO/IEC’s process which, from what I’m told, has been left untouched for many years, save a few changes to reduce the amount of time allocated to each phase of its process…

(True) Open standards development process increases quality

It is now well understood that the power of open source development comes from its community-driven approach to problem solving. Because open source communities can include people with very different geographical and cultural backgrounds, they are inherently richer than what any single organization can afford. As a result the sum of community innovations thus created far exceeds what any single vendor could create. The same applies to standards development.

• The benefits of open development apply to standards just as well
• Greater community input with different background, expertise, culture, interest leads to better standards
• Example: SOAP
  • SOAP 1.1 submitted to W3C in 2000 by several members
  • SOAP 1.2 Recommendation published in 2003
  • SOAP 1.2 is recognized by all to be superior

As previously stated and demonstrated in the example of SOAP, specifications going through true open development improve. Progressive companies that have understood this embrace this openness rather than fight it or pretend, simply because they’ve realized that when everybody benefits from it so do they.

Conclusion

Not all standards development organizations are the same. Looking forward, I believe that competition between standards organizations will increase and established de jure organizations will be further challenged. In this context, quality will become a differentiator between standards organizations and, just as it is true in the corporate world, standards organizations that do not strive to improve will become irrelevant over time.

The number of ad hoc. community-driven organizations will increase and more standards will be created the way OpenID was: by a group of interested individuals that share a common interest and decide to solve it swiftly in a somewhat informal way using the internet to its full advantage.

Customers will learn to differentiate products, solutions, and services based on quality open standards or seek unbiased counsel from firms and partners who can help them tell the difference between good quality and good marketing.

Ultimately, reliance on traditional de jure standards will probably decrease. In the meanwhile, if they care to survive standards development organizations will need to start a serious introspection of their processes and look to adopt some of the principles set by exemplary organizations such as W3C.

While no organization is perfect and there always is room for improvement, W3C has indeed set itself apart from the pack by showing the way to much greater quality and openness for the benefit of all.

It only makes me more proud to have its name on my resume. 🙂