Spotfire

I have been trying out the General Services Administration’s new portal for Governmentwide Acquisition Contracts, or GWACs.

There is a lot of useful information here, but the user experience remains uneven and in my experience, there are available tools that could improve the ability to analyze the data.

The new GSA web site says:

A Governmentwide Acquisition Contract (GWAC) is a pre-competed, multiple-award, indefinite delivery, indefinite quantity (IDIQ) contract that agencies can use to buy total IT solutions.

The GWAC program has taken their data to the next level by creating an interactive tool that allows GWAC stakeholders to view and segment GWAC information to make better decisions.

Users have the ability to:

  • Explore GWAC data by contract family, federal agency, and industry partner
  • Build customized reports and download them to your computer
The GWAC Dashboard is compatible with Internet Explorer 8 and 9 using Flash Player 14.4.X. If you are using any other version, you may experience usability issues.

The Users Guide contains the footnote:

The Governmentwide Acquisition Contract (GWAC) Dashboards are intended for informational purposes. The data contained within may not be fully accurate.

The contacts page says:

Request for accessible dashboard content may be directed to herman.lyons@gsa.go.

That may be important to some users, since they, like I experienced, may not be able to download the complete data set. That’s what happened to me, so I contacted the GSA contact, but did not get a response so I downloaded each agency separately and merged them into one spreadsheet 19,168 rows and 18 rows. I also created a data dictionary spreadsheet.

Keep reading →

“If people do not emit or discuss their behavior, it’s hard to find out what they are going to do,” declared Director of National Intelligence Jim Clapper, speaking recently at the huge annual conference of intelligence professionals called Geoint.

The U.S., he made clear, did not have tactical warning of the Benghazi attacks. He also noted that anti-American protests were taking place in 54 countries when the attacks occurred, clearly implying the intelligence community was hearing a lot of noise that day.

While it’s true, it’s hard to predict what people will do, it’s also increasingly true that people worldwide do emit and discuss their behavior through online social media and that can be monitored and analyzed in nearly real-time.

Companies such as intelligence analysis contractor Recorded Future, which is backed by Google, the CIA, and others for this very reason, offer examples of that on a routine basis. Recorded Future, for instance, tracks protests occurring around the world by extracting references to those events and their locations from online media.


Evidence offered at the first ever Recorded Future User Network (RFUN) conference at the Newseum in Washington, DC, and by a recent Washington Post story, clearly suggest that protests are rising worldwide, according to this visualization produced by the Recorded Future.

One of their measurements records “intensity” of references to protests in online media and social media, with color coded indicators signifying heavy discussion of protests, little or no discussion at all. One can also see the list of terms associated with protests in a selected country during the selected week and protests planned for the future.

There are certainly caveats to be kept in mind with these data sets and visualizations. The protest metric is not perfect, and not all protests are comparable. Much has been made of the role of social media in facilitating protest movements in the Middle East and China, but it could also be amplifying their impact.

Dr. Melissa Flagg, a senior Department of Defense manager at the RFUN event noted how analysts are starting to use Recorded Future to guide future DoD science and technology investments by tweeting: “We have to tell the story of the long term to the people who only care about the next three minutes.”


Recorded Future CEO Chris Ahlberg and inventor of Spotfire, says that Recorded Future is the only comprehensive source of past, planned and predicted events on the web and the world’s first Temporal Analytics Engine. His goal is to eventually provide access to everything on the Web in nearly real-time. This of course would be an example of really big data analysis, using the cloud like the National Security Agency has talked about.

Recorded Future is also stressing the importance of data science and data scientists in all of this. Their event featured Drew Conway (@drewconway) speaking about the joys, challenges, and power of data science. Among other insights, Conway showed some results that suggest the peaks in country protests are related to food shortages.

As a data scientists having also worked with Recorded Future on things like Visualization Of The Osama Bin Laden Letters, I could not resist checking and building on their analysis of trends in global protests (see table below.)

It shows, for instance among the top 10 countries for protest intensity, registered in the Recorded Future data set for October 1, 2011, to October 11, 2012, the United States is the most frequently mentioned term – showing up in five of the top 10 instances.

The chart also includes comments worth noting.

Keep reading →

I recently looked at the Veterans eBenefits portal and couldn’t help but conclude that it was overwhelmingly complex and needed a make over. One approach to reducing that complexity is to digitally mine the web site for its business processes and data sets and then integrating them. That would both improve the user experience and save the agency/taxpayer money.


That’s not the only approach, but it is a timely one in light of a recent story on the overgrowth of federal web sites talked about the “State of the Federal Web Report,” released Dec. 16 by a government task force. The report represented the first comprehensive review of federal websites, following the Obama administration conclusion earlier this year that there were simply too many government websites.

The report also highlighted findings from interactions with the public, which led to hundreds of suggestions for ways to improve users’ experience with federal websites. I found there were simply too many web pages with too many options and links at the Veterans and eBenefits Portal for even me to follow, let alone our veterans, their families, and service members. Keep reading →