The future of the federal statistical system in an era of open government data was the subject of the recent Association of Public Data Users Conference (APDU). It gave me the unique opportunity to pose three questions about the ironic state of federal statistics to an august panel of experts.

The panel included: Connie Citro, Director of the Committee on National Statistics, National Academy of Science; Robert Groves, Director, US Census Bureau (until recently when he became the Provost at George Washington University, that allowed him to speak freely as you will see below); and Shirin Ahmed, Assistant Director for Economic Programs, US Census Bureau.

My three questions were:

Q: Why, when the current administration has spoken so much about data, hasn’t the federal statistical community leadership (e.g. OMB Chief Statistician Kathy Wallman) spoken up?

Answer: Because she is only an SES (Senior Executive Service member) and not a political appointee. (Connie Citro)

Q: Why didn’t a data agency like the Census Bureau get the job of hosting and managing Data.gov?

Answer: It fell victim to (former federal CIO) Vivek Kundra’s federal IT program and federal IT and statistics live in two different worlds. (Robert Groves)

Q. Why doesn’t the federal statistical community maintain FedStats.gov when my analyses shows that it still has better quality data and metadata than Data.gov does?

Answer: The current administration and political candidates are not talking about statistics and there needs to be a statafacts web site for this. (Robert Groves)

It is important to understand both the history of federal statistical data and the perception that statisticians have of open government data to understand the different worlds of the federal IT, Data.gov and the federal statistical communities FedStats.gov

I thought Connie Citro expressed it best with the title of her presentation: “The Federal Statistical System – A Crown Jewel, But Its History Makes It Harder to Meet Today’s Challenges.”

Shirin Ahmed articulated one of the key problems: There needs to be a legal basis for statistical agencies to share data and passage of the Data Synchronization Proposal by Congress will help.<

Then, Census Historian Margo Anderson, co-author of Encyclopedia of the US Census asked: Haven’t We Been Here Before? Historical Perspectives on the Federal Statistical System.

Anderson described the evolution from our basic founding principles to know who we are, to how are these data critical for job creation, poverty alleviation, and policy making. She described how the federal statistical system has periodically attempted to integrate administrative and operational records into the statistical system and how open data proposals before us now are different from the past.

Open data includes operational and administrative data used now by public agencies for statistical purposes. It’s changing the world of data users and producers alike.

Statisticians refer to their data as design data (data from survey’s designed to capture data that can answer questions with statistics with confidence levels) and open data as data that just turns up and becomes “big data” because it has no end, which a survey does, like the every 10 year census that lasts several months and then is done until ten years later.

So in answer to the question about bringing these different worlds of data together: Yes we have been there before, but it is definitely different from the past and in the words of Margo Anderson: “it is both brash and exciting.” Keep reading →