These tools, however, require users to know explicitly what they are looking for while sorting; current tools do not necessarily enable discovery of unique information and may inadvertently keep highly salient but rare text hidden. Existing software allows for the annotation of text, but it generally must be performed by users who know what they are looking for.

The “Sifter” web-based platform, currently in development by Texifter, closes the gap between automated text processing and human annotator judgments. A key aspect of Sifter is its ability to automatically cluster and ”sift”collections of text to allow networks of human users to asynchronously annotate their findings. The platform scales up allowing rule-writers to “crowdsource” the task of tracking down relevant information in public comment collections.

Additionally, Sifter allows users to form credential-based peer networks with other system users in order to better incorporate expertise on specific rules and distribute the analytic work widely. Try the PCAT beta and let us know what you think.

Back to the Forum

Data.gov, the public catalog of diverse, high value, machine readable US government datasets is perhaps the best current example of information-based transparency. Recovery.gov has a similar transparency goal, but a different approach. It focuses on one topic, accounting for $787 billion in stimulus funding, by presenting summary level analysis of thousands of quarterly reports from the organizations that received funding.

There is a lot to like about these programs. They are easy to find and they offer a lot of valuable information. They are supported from the very top of the government and were implemented quickly. All executive departments participate. Still, they are far from perfect.

In concept, data.gov is great for expert data analysts, but not so for most people who lack the tools or expertise to use raw data. Moreover, many of the datasets lack standardized (or any) metadata, vary widely in the quality of data presentation and organization, and have data accuracy and consistency problems. Recovery.gov has similar weaknesses, especially when aggregating across time and many data sources.

Remember, though, that much of this data was not originally intended for public use. Making it all public automatically invites criticism. But criticism can lead to significant quality improvements, especially if direct feedback mechanisms are added. In addition, these initiatives represent an opportunity to adopt good transparency practices regarding collection, validation, analysis and presentation of information for public use. These are at least as important as technical standards for data organization and management for achieving the transparency goals of open government.

Back to the Forum

That is one of the key conclusions of the World Economic Forum Global Agenda Council on the Future of Government on which I serve. Agree or disagree, one thing is clear: Government has to change to keep pace with the 21st Century.

The digital tsunami pouring over every aspect of the global economy, society and culture has crashed around government. The lifeline that could save government from drowning in irrelevance? Openness…Open Data, Open Access, Open Dialogue, Open Government.

In the current global economic and political environment, the public is demanding transparency and accountability from government. Citizens are demanding that government engage them in open dialogue using new digital, mobile and social media communications tools. Businesses, citizens and media alike are keen to see and “touch” the mountains of information and data collected by government over the years to see what they can make of it. And in the new digital world, if users don’t get what they want or need, they will go elsewhere.

As compelling as the demand side of the “openness” equation is, the opportunities on the supply side of Open Government should alone be enough for governments of all types and sizes to open up now. Early Open Government adopters will not only stave off irrelevance, but enjoy a whole new type of relevance that will fundamentally transform government and governance in the 21st century. Complete and utter openness is the future of government.

Back to the Forum

In order for Open Government initiatives to produce results that will truly affect political and cultural change, we need rich, usable, and USEFUL solutions that will add real value to citizens and agencies… and that doesn’t happen by accident or merely through community enthusiasm.

The success of Open Government initiatives isn’t just about getting the data right, it will depend heavily on execution across these five elements: Data, APIs, Builders & Tools, Domain Experts, and Incentives – as well as a tangible strategy and formal mechanisms to bring them all together.

Full Article:
Aggregating and exposing information is the initial hurdle to compliance with the Open Government Directive, and is the basic prerequisite for everything else. But the Open Government Directive isn’t just an exercise in publishing data – the intent, and expectation, is that the data will be an enabler for a new dimension of Government service and citizen participation.

That said, the heavy focus in the technical community on getting the data has obscured some of the downstream requirements for getting the overall vision right. Sometimes it seems there’s a perception that just exposing the data is enough – and there’s an expectation that useful applications will start to “magically” appear. I’ve tried to avoid the cliché analogy to the “Field of Dreams” movie mantra: “If you build it, they will come…” – but it’s just too tempting in this case.

Unfortunately for most initiatives, it’s likely that “they” (consumers and citizens) will NOT come if we limit the overall focus to data publication. The fact is, most people will not have the time or the interest to download datasets and scour through them for usable tidbits… nor do most people have the context to fully understand the data or the time or interest to attempt to gain the proper insight.

In order for Open Government initiatives to produce results that will truly affect political and cultural change, we need rich, usable, and USEFUL solutions that will add real value to citizens and agencies… and that doesn’t happen by accident or merely through community enthusiasm. There are a set of moons that need to align “just right” in order to produce something that will be significantly valuable.

This set of elements that must coalesce includes the Data, the APIs, the Builders & Tools, the Domain Experts, and the Incentives.

• The Data – As mentioned above, this is the cornerstone to it all… without good data, nothing else matters. The good news: as a result of the Open Government Directive, we should start to see rapidly increasing volumes of Federal information – and many States have also implemented their own data publishing initiatives.

• The APIs – The data needs to be hosted somewhere and published in a way that makes it easily consumable. The minimal expectation should be a structured format for download, such as XML (nothing burns my britches more than getting structured data in a PDF file). The problem with downloading datasets, of course, is that someone has to be responsible for ongoing synchronization… and if many organizations are using the data, that means there will be a tremendous amount of duplicate work. The best solution: a single source of the data, exposed via open standard APIs.

• The Builders & Tools – Okay, you have the data and you have the APIs… the next thing needed are developers and designers who know how consume the data and how to create interfaces that meet the needs of Citizens and Agencies. You also need good tools and technologies as a platform for the solutions. The challenge here is getting the time and focus from developers (which I’ll talk about in my last point). There is good news here, though – as there are some great tools available.

• The Domain Experts – My favorite dataset up on Data.gov is the “2005 Toxics Release Inventory data for Guam”… it just sounds very interesting and cool. Unfortunately, I have no idea what a Toxic Release Inventory is, I’d probably be challenged to find Guam on a globe, and I have no idea how to decipher the information in the dataset. However, there is likely someone out there who completely understands the information and knows how it can be used. It is critical to engage such Domain Experts as we embark on the development of solutions – otherwise, most of the data publishing work will be a useless exercise.

• The Incentives – this one is the kicker… how do we actually get people to write applications?? Yes – there will be some that are built out of interest, curiosity, and even civic responsibility. But I think it would be unreasonable to expect those things, alone, to develop an entire ecosystem of solutions that truly meet the tenets of Open Government. There needs to be a carrot that helps incent people to spend time and money building such solutions, and that typically means that that folks need visibility into how they’re going to monetize their investment – either by selling solutions, using an advertising platform, gaining visibility in the market, providing consulting opportunities, etc.

Thus, success of Open Government initiatives isn’t just about getting the data right, it will really depend heavily on execution across the above five elements (Data, APIs, Builders & Tools, Domain Experts, and Incentives) – as well as formal mechanisms and a tangible strategy to bring them all together.

Original Article posted here: http://www.dankasun.com/archive/2010/02/09/open-government-starts-with-open-data.aspx

Back to the Forum

Today of course transformation of governments is a big theme.

But where to start that reorganization from?

There are a few possibilities: Data, Technology, Standards.

Of course these are all true but I think the starting point could be process.
Here is why:

I saw US CIO Vivek Kundra’s recent talk (video link HERE) about the future of Government, and the themes that come across consistently are process related. For instance: Cut waste in services, Social security admin not as easy to get as booking a table on OpenTable; Blackberry allocation based on seniority; Veterans benefits (simplifying); Patent and trademark offices workflow – (speeding up) etc etc.

But if the starting point is ‘process’ then this has two components.

On one hand we have internal processes (like speeding up veterans benefits) and on the other hand we have what I call ‘unpredictable processes’. These unpredictable processes need data to be liberated so that ordinary developers can create new services.

Today, there is a myth/emphasis on external/unpredictable (Web 2.0) processes and less on the core internal processes like speeding up veteran’s benefits. These internal processes are not related to liberating data but have far reaching benefits.

Are we getting too caught up in the Web 2.0 hype and focusing on liberating data but perhaps not giving as much coverage to optimizing internal processes?

Interestingly enough, the Vivek Kundra talk seems to be more balanced in this regard – which is good to see.

Back to the Forum

The USA has many such federal government data liberation initiatives – for instance California Governor Arnold Schwarzenegger’s initiative to reduce data centre footprint, NASA’s Nebula cloud computing environment, apps.gov and others.

In the UK, Tim Berners-Lee unveils government data project. London Mayor Boris Johnson unveils a London data store.

Sites like Openly Local and Mash The State are supposed to use that data and create services through mashups i.e. combining data from two or more external services to create a new service.

This is all good, but don’t we need standards for mashups?

There are two issues: firstly, not all data may be ‘combinable’ with other data and the rights and revenue sharing availed by the mashup may be unclear. Secondly, the mashups themselves need a common framework (standard).

It turns out that there is such an industry led mashup standards initiative called the Open Mashup Alliance which is designed to promote an open schema and language for mashups called Enterprise Markup Mashup Language (EMML).

This is a positive step and it is industry led. It recognizes that ‘data liberation’ is only the first step – we need standards for mashups to make use of the ‘liberated data’.

Back to the Forum

What does Open Government imply for standards?

Let’s take a step back. Prior to 1999, I used to work for an ERP vendor. ERP (Enterprise Resource Planning) is a complex class of software that is typically intended to manage all the functions of a company (such as Accounts Payable, General Ledger, Billing and so on). Inspite of their complexity, there was a mad scramble to install ERP systems which was mainly motivated by the Y2K deadline.

However, did ERP really make a difference in terms of innovation?

Not really.

ERP was all about internal functions of an organization, the ‘plumbing’ so to speak. It made very little difference to the customers as to what billing system a company used or what General Ledger the company used.

Last week, the White House adopted Drupal, the Open source content management system which has a strap line of ‘community plumbing’. Whitehouse.gov is now running on Drupal. However, like ERP, if it is used only for internal functions (aka ‘plumbing’) – it should make no difference to anyone at all…

After all, Whitehouse.gov is a pretty structured website in terms of its content; photos of the President, videos, some blogs, information on legislation etc. And that content is mostly structured (apart from comments). One could argue that Drupal has many extensible libraries that third parties can build. But so do many other platforms.

Thus, the adoption of Drupal should be viewed more as a symbolic gesture for the use of Open Source; and in that role lies its greater significance and implications for standards.

Tim O’Reilly discusses the wider impact of the Drupal announcement under the concept of Gov 2.0.

The introduction of the terms Open Government/ Gov 2.0/ ‘Web 2.0 for Government’ adds to two already complex terms i.e. ‘Open Source’ and ‘Open Standards’. As viewed by Tim O’Reilly, Open Government is the concept of ‘Government as a platform’ which could also be seen as ‘Web 2.0 for Government’.

Web 2.0 itself is a philosophy i.e. a broad concept based on the foundation of Data, which many people wrongly equated to specific technologies (like the AJAX programming language).

Similarly, Open source (the technological foundation) is not necessarily needed for Open Government (which could be viewed as the concept/philosophical foundation of ‘Government as a platform’).

The question of whether Governments should mandate specific standards or technologies is more complex. Today, in the minds of most people we have a benevolent administration in the White House. But administrations can change every five years, and often they do. And consequently, enthusiasm for Government led ideas can wane.

The philosophy of Openness is correct, as is the idea of ‘Government as a platform’. However, just like Web 2.0, its technological implementation may be varied.

The standards for Open Government may be independent of the technology that is used to implement its internal systems as long as the philosophy of Openness and the idea of the ‘Government as a platform’ is maintained.