recent DISCUSSION

• Daniel Dardailler says:

  Ken, you wrote “To summerize all these discussions (at least for me): The desire is to rate the standard not the standardization organization as each standard from a standardization organization might rate quite differently ”

  Not quite.
  Only looking at the products, the standards themselves, is not enough, as it doesn’t give any information as to which stakeholders where part of the design. You may argue that the standard text itself is reflexive of its stakeholders, and therefore that by examining it, we should be satisfied (or not), but investors in standards want to know up front what their engagement in a given SSO is going to produce in the end, not in terms of prophecy, but in terms of probability of ending up in the results they expect.

  Of course each standard needs to carry its own set of metadata, but out of a dozen criteria, I would image that 10 will be the SSO default, and a few will be override this default.

  example:
W3C
criteria (patent) = RF
criteria (copyright) = free to copy, not to change
criteria (test suite available) = not guaranteed
criteria (open participation) = members + invited expert
IETF
criteria (open participation) = public
criteria (patent) = FRAND
etc


  and then, for each standard, default to where they come from, or override

  
W3C.XML
criteria (test suite available) = yes
W3C.HTML
criteria (open participation) = public
IETF.SMTP
criteria (patent) = RF


  etc

  In addition, there are criteria that are not really relevant to the specs, like

  
W3C
criteria (organization type) = non profit, private, multiple hosts, paid staff
criteria (government rep) = no, they can join as member


  but which nevertheless may help investors in standards making their choice.

  There are also criteria that are only specific to a spec, e.g. deployment, acceptance of a standard, how long it took to make, which can be averaged, etc back into a SSO criteria.

  You also write “The standardization organization would create the rating of the standard.”

  What is the rating here, is that the ranking, or the criteria ?

  I think the set of criteria allowing for ranking/rating to be developed by the market needs to be done once, not by each SSO.

  Note that the logic I present above is exactly what the ISO PAS transposition is about: first you describe your organization to get a PAS submitter status, and then for each PAS/spec sent to ISO for ballot, you override the criteria of the org.
• Ken Krechmer says:

  To summerize all these discussions (at least for me): The desire is to rate the standard not the standardization organization as each standard from a standardization organization might rate quite differently (e.g., SIP, an IETF RFC, would get high ratings for adaptability). The standardization organization would create the rating of the standard. A rating system that is well vetted should be applied by all standardization organizations within a specific area (type or geographic). E.g., ANSI in the US, all organizations submitting standards for PAS to ISO, ASTM and related organizations, etc.

  I do think the ratings system will have to be different for different standards successions.
• Daniel Dardailler says:

  Hello Matias

  I’m not saying that we should standardize a ranking system, but standardize the objective criteria that allows for ranking systems to be developed.

  Ranking systems may be good or bad, too simple or too complex, I’m happy to let the market decide which one should succeed, on a sector by sector basis, country by country level, and the WTO criteria would just be another mapping/filter applied by the WTO, for instance.
• Mattias Ganslandt says:

  Daniel!

  Interesting points. Some of these issues need to be discussed in more detail. While there is generally a great fascination for metrics that tries to reduce multi-dimensional qualities into a one-dimensional measure it is much less certain that such metrics are helpful for optimization and efficiency. Why would you like to reduce a problem that involves a fundamental trade-off, e.g. between inclusive decision making and speed to market or between IP and access, into a single, universal measure? In fact, the optimum is likely to depend on the specifics of the case.

  Constructing an index (ranking) that is useful is all about measurement and weights. I doubt that this can be done with reasonable precision to be useful in the SSO context.

  I do agree with your core principle that transparency is fundamental for ensuring effective decision-making. This is generally true – in politics, in business and in standard setting./Mattias
• Daniel Dardailler says:

  Hello Ajit

  wrt funding for the SSO standard work, and hosting such an initiative, I don’t have an answer. There would clearly be interest in groups like DCOS at the IGF/UN but IGF itself is not designed to produce anything (but an avenue for discussions and a yearly f2f plenary). I’m sure part of the industry would be interested in sponsoring such an effort, as it costs them a lot to track and adapt to our jungle. IBM made statements to that effect recently. Governments clearly need more clarity so they will help as well.

  Regarding the cross domain aspect, it is just another dimension in our grid. I can very well imagine than a non RF standard be acceptable in a cross Web/OilGasIndustry interface protocol, where no Open Source community is present and industry is happy and able to pay lawyers, but not in a pure Web environment where we run on Open Source.

  What is missing is a standard saying exactly that kind of things and explaining the consequences, not something saying RF is good.
• Ajit Jaokar says:

  Hello Daniel

  Good to see you here.
  Yes w3c and ietf have made some great contributions to web so your insights are valuable

  Some comments:

  1) As you said, w3c and others have smaller budgets. So the qs is: who has the budgets to create such a ranking for all standards and what would be the motivation?

  One could think of governments – but governments may not have the budgets themselves and immediately we have one guideline per government!

  2). Increasingly, standards will be ‘cross domain’ ex smart grids, e health etc making the act of ‘rating’ them much more complex especially since standards bodies will also evolve and hence we could end up rating a moving goalpost. Thoughts?

  3) Re:
  ——
  These SSO evaluation criteria to be developed should be inclusive of the ways W3C or IETF have been operating, given that it has brought us the benefits of the Internet and the Web as we know it (i.e. developed using a transparent and consensual process, open to all stakeholders, and that are free for all to implement and redistribute).
  —-
  Free to implement and redistribute works well with the web but not in all domains because the cost of production of pure digital content is zero( ex on web). However, in other domains like mobile technologies, cross licensing of ipr is almost a rule rather than an exception.
  Hence free will not work esp because companies have spent some costs creating the entities in the first place.

  4) Re
  ——————————————————-
  To the question:

  * What are the pros and cons of ranking standard setting organizations e.g. fora and consortia and, if done, how should rankings be used?

  More valuable thank side-by-side comparisons of organizations, establishing agreed to goals would allow all organizations to understand avenues for improvement that they may prioritize according to resources and other considerations.
  —————————————————–
  A ranking is a ‘side by side comparison’ ie we already have a range of goals and discussions about standards themselves and these are being debated. I thought that the ranking was the extra step being proposed. So, the qs is – what is missing in the agreed to goals for standards at the moment?

  Many thanks
  Kind rgds
  Ajit
  PS- I am travelling so responses when I land in london on fri
• George Willingmyre says:

  Daniel mentions the TBT code GTW Associates offered recently to review the BSI DRAFT PAS 98 Code of practice for the establishment and management of consortia
  http://www.businessstandards.com/In_Brief/091130_BS29_Consortia according to the ISO/IEC Guide 59: 1994 Code of good practice for standardization (Excerpts from which are at http://gtwassociates.com/answers/Guide59.Excerptspdf.pdf So far the the BSI work remains within the closed committee. I was involved in creating the ISO code which regrettably IMO sits on a shelf. I highlight on the web page the IPR text
• Daniel Dardailler says:

  Kai, I don’t buy the “there is too much of it for us to handle”, we’re talking about a dozen criteria max and their individual vocabulary (e.g. copyright regime: “free to copy, not free to change”, or “all rights reserved”, etc). It doesn’t have to be number of course.

  In terms of ontology complexity, if we think this is too much, we may as well stop thinking about the Semantic Web ;)
• Daniel Dardailler says:

  Niklas, yes we’ve got our failures as well, and I’m fine documenting those, why they fail, what where the costs, the benefits. In fact, I’d welcome external people studying that (all our discussion archives our available), and the steps we took or should have taken to improve the situation.

  Smaller consortia would probably have a success/failure rate higher than ours, just because they have less production, but this kind of metrics would have to be taken contextually, or not (up to whoever is using this metadata).

  At the same time, we should look at the presence of things like test suites, end-users implication, which are usually sacrified by budget constraints. I’d be so happy that W3C gets a bad rating in that area, believe me, this way, our stakeholder would add more water to the mill just for doing that: more testings, interop labs, conformance review, etc.

  I’ve been a programmer for years, and anything that helps writing code once, that is, with interoperability, is good.
• Niklas Lindblom says:

  Daniel, Thank your for the insight into the W3C point-of-view. However, your answers raises a few additional questions. For instance, you mention that a ranking system would allow W3c to gain additional recognition for its successes, but have all W3C standards ever produced become successful? How should the fact that even the “best” SSO can produce standards that never become accepted by the market be handled in a ranking system? And what about SSOs of which only a minority of standards become successes? Should those standards never even get the chance to reach the market? //Niklas
• Kai Jakobs says:

  I pretty much agree with Daniel re the question “Is a formalized review of SSOs fostering or impeding efficient standardization?”; the point where I disagree is out of topic for this discussion.

  Re: “What are the pros and cons of ranking standard setting organizations e.g. fora and consortia and, if done, how should rankings be used?”

  We’ve got way too many rankings anyway …. And dependiong on what I want to standardise I will be looking for different qualities in an SSO. For instance, wide participation and far-reaching consensus will be important for infrastructure technologies. In contrast, speed will be important for application standards. Technical quality and/or low licensing fees may be important for ‘stand-alone standards’ (e.g., codecs). A ranking would only serve to pretend to be able to make an ‘objective’ choice.