Brand Niemann

 

Posts by Brand Niemann

I have been trying out the General Services Administration’s new portal for Governmentwide Acquisition Contracts, or GWACs.

There is a lot of useful information here, but the user experience remains uneven and in my experience, there are available tools that could improve the ability to analyze the data.

The new GSA web site says:

A Governmentwide Acquisition Contract (GWAC) is a pre-competed, multiple-award, indefinite delivery, indefinite quantity (IDIQ) contract that agencies can use to buy total IT solutions.

The GWAC program has taken their data to the next level by creating an interactive tool that allows GWAC stakeholders to view and segment GWAC information to make better decisions.

Users have the ability to:

  • Explore GWAC data by contract family, federal agency, and industry partner
  • Build customized reports and download them to your computer
The GWAC Dashboard is compatible with Internet Explorer 8 and 9 using Flash Player 14.4.X. If you are using any other version, you may experience usability issues.

The Users Guide contains the footnote:

The Governmentwide Acquisition Contract (GWAC) Dashboards are intended for informational purposes. The data contained within may not be fully accurate.

The contacts page says:

Request for accessible dashboard content may be directed to herman.lyons@gsa.go.

That may be important to some users, since they, like I experienced, may not be able to download the complete data set. That’s what happened to me, so I contacted the GSA contact, but did not get a response so I downloaded each agency separately and merged them into one spreadsheet 19,168 rows and 18 rows. I also created a data dictionary spreadsheet.

Keep reading →

“If people do not emit or discuss their behavior, it’s hard to find out what they are going to do,” declared Director of National Intelligence Jim Clapper, speaking recently at the huge annual conference of intelligence professionals called Geoint.

The U.S., he made clear, did not have tactical warning of the Benghazi attacks. He also noted that anti-American protests were taking place in 54 countries when the attacks occurred, clearly implying the intelligence community was hearing a lot of noise that day.

While it’s true, it’s hard to predict what people will do, it’s also increasingly true that people worldwide do emit and discuss their behavior through online social media and that can be monitored and analyzed in nearly real-time.

Companies such as intelligence analysis contractor Recorded Future, which is backed by Google, the CIA, and others for this very reason, offer examples of that on a routine basis. Recorded Future, for instance, tracks protests occurring around the world by extracting references to those events and their locations from online media.


Evidence offered at the first ever Recorded Future User Network (RFUN) conference at the Newseum in Washington, DC, and by a recent Washington Post story, clearly suggest that protests are rising worldwide, according to this visualization produced by the Recorded Future.

One of their measurements records “intensity” of references to protests in online media and social media, with color coded indicators signifying heavy discussion of protests, little or no discussion at all. One can also see the list of terms associated with protests in a selected country during the selected week and protests planned for the future.

There are certainly caveats to be kept in mind with these data sets and visualizations. The protest metric is not perfect, and not all protests are comparable. Much has been made of the role of social media in facilitating protest movements in the Middle East and China, but it could also be amplifying their impact.

Dr. Melissa Flagg, a senior Department of Defense manager at the RFUN event noted how analysts are starting to use Recorded Future to guide future DoD science and technology investments by tweeting: “We have to tell the story of the long term to the people who only care about the next three minutes.”


Recorded Future CEO Chris Ahlberg and inventor of Spotfire, says that Recorded Future is the only comprehensive source of past, planned and predicted events on the web and the world’s first Temporal Analytics Engine. His goal is to eventually provide access to everything on the Web in nearly real-time. This of course would be an example of really big data analysis, using the cloud like the National Security Agency has talked about.

Recorded Future is also stressing the importance of data science and data scientists in all of this. Their event featured Drew Conway (@drewconway) speaking about the joys, challenges, and power of data science. Among other insights, Conway showed some results that suggest the peaks in country protests are related to food shortages.

As a data scientists having also worked with Recorded Future on things like Visualization Of The Osama Bin Laden Letters, I could not resist checking and building on their analysis of trends in global protests (see table below.)

It shows, for instance among the top 10 countries for protest intensity, registered in the Recorded Future data set for October 1, 2011, to October 11, 2012, the United States is the most frequently mentioned term – showing up in five of the top 10 instances.

The chart also includes comments worth noting.

Keep reading →

Jonny Goldstein, a veteran media producer, artist, trainer, and speaker, created the above visual notes for the recent Digital Government: The Transformative Power of Communications, captured and disseminated fittingly via Instagram.

The tweet: “Check out @jonnygoldstein visual notes from (the) #GovD12 event on Digital Government: the power of communications!” captured the sentiment of many that attended this excellent event, which featured the book Little Bets, by Peter Sims and a series of government agency success stories including the following:

  • Small Business Administration – As a result of a campaign focused on government contracting, visits to the SBA government contracting page have jumped 72% since the campaign began — and a 255% increase in traffic to entry-level courses like Government Contracting 101.
  • U.S. Census Bureau – A national campaign to encourage the public to participate in the 2010 Census saved $1.87 billion in taxpayer money. Census also saw more than 175,000 visits to its FAQs, helping to increase the number of subscribers the agency can reach directly.
  • Centers for Medicare & Medicaid Services – In just eight months, CMS has seen more than 850,000 visits to its FAQs, helping to drive an increase of more than 80,000 subscribers to over 2,400 topics.
  • United States National Guard Bureau – NGB was recognized for its excellence in digital communications. It recently won an award for proactively communicating through multiple communication channels to reach the largest possible audience establishing more than 5 million impressions through a dynamic news widget, and communicating its mission and message to 22,000+ blog viewers.
The purpose of this year’s event was to address concrete steps agencies can take to fulfill the objectives of the White House’s Digital Government Strategy:

  • to enable the American people to access high-quality digital government information and services;
  • to insure that government adjusts to a new digital world; and
  • to unlock the power of government data to spur innovation across the nation and improve the quality of services for the American people.
The event’s sponsors provided a Digital Government Strategy checklist as follows: Keep reading →

The future of the federal statistical system in an era of open government data was the subject of the recent Association of Public Data Users Conference (APDU). It gave me the unique opportunity to pose three questions about the ironic state of federal statistics to an august panel of experts.

The panel included: Connie Citro, Director of the Committee on National Statistics, National Academy of Science; Robert Groves, Director, US Census Bureau (until recently when he became the Provost at George Washington University, that allowed him to speak freely as you will see below); and Shirin Ahmed, Assistant Director for Economic Programs, US Census Bureau.


My three questions were:

Q: Why, when the current administration has spoken so much about data, hasn’t the federal statistical community leadership (e.g. OMB Chief Statistician Kathy Wallman) spoken up?

Answer: Because she is only an SES (Senior Executive Service member) and not a political appointee. (Connie Citro)

Q: Why didn’t a data agency like the Census Bureau get the job of hosting and managing Data.gov?

Answer: It fell victim to (former federal CIO) Vivek Kundra’s federal IT program and federal IT and statistics live in two different worlds. (Robert Groves)

Q. Why doesn’t the federal statistical community maintain FedStats.gov when my analyses shows that it still has better quality data and metadata than Data.gov does?

Answer: The current administration and political candidates are not talking about statistics and there needs to be a statafacts web site for this. (Robert Groves)

It is important to understand both the history of federal statistical data and the perception that statisticians have of open government data to understand the different worlds of the federal IT, Data.gov and the federal statistical communities FedStats.gov

I thought Connie Citro expressed it best with the title of her presentation: “The Federal Statistical System – A Crown Jewel, But Its History Makes It Harder to Meet Today’s Challenges.”

Shirin Ahmed articulated one of the key problems: There needs to be a legal basis for statistical agencies to share data and passage of the Data Synchronization Proposal by Congress will help.<

Then, Census Historian Margo Anderson, co-author of Encyclopedia of the US Census asked: Haven’t We Been Here Before? Historical Perspectives on the Federal Statistical System.

Anderson described the evolution from our basic founding principles to know who we are, to how are these data critical for job creation, poverty alleviation, and policy making. She described how the federal statistical system has periodically attempted to integrate administrative and operational records into the statistical system and how open data proposals before us now are different from the past.

Open data includes operational and administrative data used now by public agencies for statistical purposes. It’s changing the world of data users and producers alike.

Statisticians refer to their data as design data (data from survey’s designed to capture data that can answer questions with statistics with confidence levels) and open data as data that just turns up and becomes “big data” because it has no end, which a survey does, like the every 10 year census that lasts several months and then is done until ten years later.

So in answer to the question about bringing these different worlds of data together: Yes we have been there before, but it is definitely different from the past and in the words of Margo Anderson: “it is both brash and exciting.” Keep reading →

After my recent SafetyData.gov review, (“Long On Text, Short On Data Tables”), I was resolved to review the new Energy Data.gov Beta Web Site and check some of the claims presented at the White House’s Energy Datapalooza held earlier this month.

I decided the best place to start was the Energy Datapalooza fact sheet because it did not contain any links to actual energy data. Energy Data.gov says “data and insight are combined to facilitate public discussion and awareness of our Nation’s energy activities.”

So I tried to match the facts to the substance using a knowledge base found elsewhere. Here’s what the fact sheet contained – and what I found:

Administration Announcements

  • New Application Programming Interfaces (APIs): 4 – (My comment: I only found three in the Presidential Innovation Fellows Blog and they were not APIs – see below.)
  • New Data for Entrepreneurs and Innovators: 2 – (My comment: Energy.Data.Gov with more than 900 data sets – not really – see below, and 20 new datasets from DOE – actually only 19 that require closer inspection.)
  • New Events and Challenges: 2 – (My comment: At Apps for Energy there are 9 winners from 56 submissions with no new challenges.)
  • New Green Button Integration – (My question: Where do I find how to do this? See below for the work it requires.)
Private Sector Committments Keep reading →

One of the best presentations at the recent Big Data Innovation Summit in Boston was by LinkedIn Senior Data Scientist Yael Garten. Garten, who leads LinkedIn’s Mobile Data Analytics team, in a presentation entitled Data Infused Product Design & Insights at LinkedIn provided a glimpse of how big data is used by LinkedIn to explore usage patterns, on mobile devices, for instance.

This is a challenge facing the US Government in the new Digital Government Strategy: namely delivering existing web sites and database information — and eventually the types of big data results that the intelligence and scientific communities have — so that mobile devices can access that information from supercomputers.

For those who haven’t kept up with LinkedIn’s progress of late: It’s mission is to “connect the world’s professionals to make them more productive and successful.” Here are a few facts at glance, from Garten’s presentation: Keep reading →

Big data, which has been the hot topic for conferences this year, has also received a good deal of attention on Capitol Hill in recent weeks, most notably with two recent events:


As one who represents a population of data scientists, a group for which the TechAmerica says there is growing demand, I have seen quite a few–and written a number of–articles about recent big data conferences:

For those who contributed to the ACT-IAC discussion with Congressional staff members on Big Data at the Hill – Defining and Understanding Policy Implications, I offer some specific ideas to three suggestions in their report:

What Congress should do to help big data Keep reading →

“The liberation of government datasets is important in itself, but data are truly powerful when used in the development of informative apps.” So proclaimed Todd Park, Brian Forde and Jo Strang in a recent White House Blog, Safety Data Jam connects Tech Innovators with Public Safety Officers.


That safety jam was part of a broader initiative to challenge developers featuring new Data.gov Safety “data sets” that gained fresh exposure this month at the White House’s Safety Datapalooza.

I took a deeper look to see what innovation with government data is possible.

Keep reading →

There has been a lot of activity from the Obama Administration this week in the name of innovation and best practices.

There was a double-post on the White House Blog by Federal CTO Todd Park and the Federal CIO Steve VanRoekel about two new initiatives that would seem to be related, but it is not clear how. The first was a prelude to White House’s Plans To Announce Presidential Innovation Fellows and the second was from VanRoekel touting the progress of the Digital Strategy Progress. The latter featured the use of the term “building blocks.” Keep reading →

Page 1 of 912345...9