Talkstandards » Blog Archive » The Pros and Cons of SSO Ranking

The Pros and Cons of SSO Ranking

posted by Damien Geradin on Thursday, April 29th, 2010 at 2:59 pm

Rankings can usefully contribute to inform users/consumers in a large variety of areas about the options that are available to them. They can also help them to make a well informed choice. To be useful, rankings need to be objective and informative, and rely on sound methodology and data. The methodology and data used to rank people, companies, institutions, countries, etc will often be criticized. Ranking of academic institutions are for instance highly controversial. Besides the fact they are not necessarily accurate, they may also induce universities to focus on issues that will improve their ranking rather than those that may ameliorate the welfare of their students and professors. There are therefore pros and cons about rankings.

Ranking standard-setting organisations (SSOs) would thus raise significant challenges. The first immediate question would be to determine the criteria that would be used to rank SSOs. Such criteria could comprise a large variety of factors, such as participation, openess, fees and costs, adoption speed, successful implementation, etc. A second question would be to decide the relative importance to be given to these factors. Fees and costs may not be an issue for some companies, but it might be for others. Successful implementation is the ultimate goal, but it may not be easy to measure as people may disagree over what “success” means. These challenges would be very serious indeed and may prove unsurmountable.

But even if one could agree on methodology, it remains to be seen whether such rankings would do more good than harm. Well ranked SSOs may become very popular, while poorly ranked SSOs may disappear. This would clearly affect diversity and competition between SSOs. This lack of competition might in turn strengthen the position of firms that dominate a given standard.

For these reasons, a preferable approach would be to develop guidelines as to what how a SSO should ideally operate, but such guidelines should be sufficiently flexible to allow diversity, which is very beneficial in itself.

Back to the forum

recent DISCUSSION

A. Rebentisch says:

We know what the best standard from the rationale of a market system, a standard which leads to perfect competition, in other words enables free markets and free trade. SSOs can be ranked by their ability to deliver these results. Simple. Of course individual agents in the standard process normally do not pursue a perfect competition agenda, e.g. want to use patents to gather advantages. The task of a SSO is here to set its rules and provide the mechanisms. Both invisible and visible hand so to speak.
Daniel Dardailler says:

The question is not whether or not it’s good to have criteria, we know it’s good to analyse things to improve them, it’s whether or not we all use the same, and therefore allow for more interoperability of this SSO ecosystem.

Having worked all my professional life in this area, but I believe standards allow for innovation, by design, and only in rare occasions they slow down progress, even when they fail. They provide a stable platform for other to run on top.

The SSO ecosystem is currently unstable, with, to mention one aspect, the ESO and ISO sections receiving millions in staffing and support and Internet and Web consortia almost nada; same goes in terms of government recognition, how stable is a situation where goverments cannot help W3C, since we’re not “official”, nor can they reference our specs, but where they have to use our production anyway since there is no other choice ? More transparency and understanding of the situation, using a common set of criteria, could help foster the creations of many more consortia, giving them access to grants by virtue of them following a given level of criteria. Today, they have to lobby to get funding, and it’s always in the way of their real work, which is to produce standards.
Alistair Buttar says:

Earl makes a good point that SSOs listing their own attributes might encourage competition and evolution without the downside of ranking – in fact, most SSO have some kind of a manifesto on their websites which makes clear aims of the fora, but views from the outside are likely to diverge wildly and depending on preferred business models, will also result in very different rankings. Ken also brings up a good point concerning internal reviews of SSO – most SSO do continually review the internal situation, sometimes impacted with external events, and internal changes can be made through an internal governance procedure. In some cases internal policies can be changed, and in extreme cases, the forum could be merged with another similar forum or simply dissolved if the aims of that forum have been completed or have been surpassed by events in the market place. But this is internal ranking based on market realities, and attempts from the outside to perform such ranking would certainly be open to abuse. And Steve also makes the point about innovation – external pressure would surely limit diversity and enforce “blinkers” which would polarize approaches and impose regimes reminiscent of those characteristic of the Middle Ages and fiefdoms – surely we don’t want to see ourselves back in those times!
Steve Mutkoski says:

Great comment by Earl, with an excellent example. I’ve stressed elsewhere the importance of maintaining a diversity of approaches that we currently have and I think you point out the important and logical extension, that we need to allow for innovation going forward to ensure we have that diversity in the future. Ajit did a nice job of explaining this in his post here
http://www.talkstandards.com/standards-for-standards-is-the-best-way-to-predict-the-future-to-standardize-it/
Ken Krechmer says:

Perhaps we could break this question into two parts. One, ranking SSOs from the outside. Two, ranking SSOs from the inside. I am not a supporter of ranking the SSOs from the outside for the reasons already given. But I think the SSOs do need to review their own policies to see if the level of openness is sufficient for their stakeholders.

Perhaps what is needed from the outside is an impetus to the SSOs to rank themselves.
Damien Geradin says:

I concur with both Earl and Ken.
Ken Krechmer says:

Ranking SSOs will always be ay least somewhat qualitative. So any ranking will have a political component. This is undesirable in my view. In addition ranking raises the issue of what is the “best.” As always the “best” is in the eye of the beholder.

Considering the standardization process a complex adaptive system where different SSOs will jockey for relevance is more appropriate. No ranking is necessary, although the SSOs should do much more to present how they achieve openness.
Earl Nied says:

Another challenge of ranking will be to allow room for novelty and new ideas. By its nature, ranking must follow criteria set by some group but also keep in mind that this criteria is from a snapshot in time. If not constantly updated and subject to the highest levels of open and active involvement, it runs the risk of stifling innovative solutions from ever being introduced.

Example: I chaired the ANSI Patent Group at a time when ANSI was reviewing accreditation renewals for two SSOs who revised their IPRs. Each had a different and novel approach to facilitate the disclosure of patent license terms (generally considered an increase in openness and transparency). However, one policy had strict rules that some people considered impractical and some key companies had already left that SSO over those rules. So, an operative question became: “Is this new policy really more open if some key contributors are leaving?” The counter argument was that other companies were continuing to join and the additional information offset the loss of a few contributors. ANSI chose to approve accreditation of the SSO policy with some minor changes/clarifications to their policy. Despite many Patent Group members being skeptical of the effectiveness of the proposed policy, ANSI did not pass judgment and allowed the policy to evolve in the market.

I believe the above example came to the right conclusion. The draft IPR policy had its own pros and cons that were unique to that situation and not discernable to an outside party. This needed to evolve in the market. It might succeed as proposed. It might evolve. It might fail.

I personally feel it would be hubris to think that anyone could create a ranking criteria for all situations. Rather than a ranking criteria, perhaps a listing of attributes could help facilitate people’s decision to use a standard. I note that this would also need to be accomplished in a neutral and equitable manner. Perhaps allowing SSOs to list their own attributes might encourage competition and evolution without the downside of ranking.
Damien Geradin says:

I think that the criteria I cited were fairly uncontroversial (costs, speed, openess, participation, etc.), but other criteria may be much more sensitive. Should the fact that standards involved / do not involved IPRs be taken into consideration. Companies / entities may have very different views about that one.

But even if you manage to agree on a set of criteria, this still doesn’t mean that the ranking will work optimally. Most ranking of academic institutions are for instance deeply flawed, and they give terrible incentives to universities, which might for instance decide to fire some of their students to improve their teacher / student ratio.

I still remained to be convinced not only about the feasibility of a ranking of SSOs, but also about the usefulness of such rankings.
Daniel Dardailler says:

Hello Damien

I’m always surprised by the comments like “Such criteria could comprise a large variety of factors”.. and therefore it is too hard..

As I said in another post, we’re talking about a dozen criteria max, you name off your head half of them, and I could do the same, so what is large for you ?

Consider the Health Care/Life Science SemWeb group at W3C, they have to consider standardizing the meaning of things from molecules to brain diseases.

I did such a matrix of SSO comparison a few years ago, it took me a couple of hours to design and a bit more time to fill it using google. What will take us time is agreeing on the criteria, of course, but with some principle requirements laid out clearly, I think this is achievable in a few months of committee work, say a year. Plus, they are several framework already available, like Ken’s work on the various users of standards, the rights those people have, etc.

If we don’t do that, it won’t be because it’s too hard to do, it will be because people don’t want to see it happening.
Damien Geradin says:

The outlook is bleak. In any event, I think that ranking SSOs is a bad idea.
Niklas Lindblom says:

Damien, your parallell to the ranking of uni’s is very striking. There certainly is a risk of “conforming” SSOs if a ranking system is introduced. Given your focus on methodology, how do you consider the outlook of ever reaching any agreement on how rankings should be performed? //Niklas

The Pros and Cons of SSO Ranking

recent DISCUSSION

Comment on this post

About the Author

Damien Geradin

Talkstandards.com

An active online community where developers, researchers, policymakers and other interested parties can share ideas and collaborate on the global standards system.