“If people do not emit or discuss their behavior, it’s hard to find out what they are going to do,” declared Director of National Intelligence Jim Clapper, speaking recently at the huge annual conference of intelligence professionals called Geoint.
The U.S., he made clear, did not have tactical warning of the Benghazi attacks. He also noted that anti-American protests were taking place in 54 countries when the attacks occurred, clearly implying the intelligence community was hearing a lot of noise that day.
While it’s true, it’s hard to predict what people will do, it’s also increasingly true that people worldwide do emit and discuss their behavior through online social media and that can be monitored and analyzed in nearly real-time.
Companies such as intelligence analysis contractor Recorded Future, which is backed by Google, the CIA, and others for this very reason, offer examples of that on a routine basis. Recorded Future, for instance, tracks protests occurring around the world by extracting references to those events and their locations from online media.
Evidence offered at the first ever Recorded Future User Network (RFUN) conference at the Newseum in Washington, DC, and by a recent Washington Post story, clearly suggest that protests are rising worldwide, according to this visualization produced by the Recorded Future.
One of their measurements records “intensity” of references to protests in online media and social media, with color coded indicators signifying heavy discussion of protests, little or no discussion at all. One can also see the list of terms associated with protests in a selected country during the selected week and protests planned for the future.
There are certainly caveats to be kept in mind with these data sets and visualizations. The protest metric is not perfect, and not all protests are comparable. Much has been made of the role of social media in facilitating protest movements in the Middle East and China, but it could also be amplifying their impact.
Dr. Melissa Flagg, a senior Department of Defense manager at the RFUN event noted how analysts are starting to use Recorded Future to guide future DoD science and technology investments by tweeting: “We have to tell the story of the long term to the people who only care about the next three minutes.”
Recorded Future CEO Chris Ahlberg and inventor of Spotfire, says that Recorded Future is the only comprehensive source of past, planned and predicted events on the web and the world’s first Temporal Analytics Engine. His goal is to eventually provide access to everything on the Web in nearly real-time. This of course would be an example of really big data analysis, using the cloud like the National Security Agency has talked about.
Recorded Future is also stressing the importance of data science and data scientists in all of this. Their event featured Drew Conway (@drewconway) speaking about the joys, challenges, and power of data science. Among other insights, Conway showed some results that suggest the peaks in country protests are related to food shortages.
As a data scientists having also worked with Recorded Future on things like Visualization Of The Osama Bin Laden Letters, I could not resist checking and building on their analysis of trends in global protests (see table below.)
It shows, for instance among the top 10 countries for protest intensity, registered in the Recorded Future data set for October 1, 2011, to October 11, 2012, the United States is the most frequently mentioned term – showing up in five of the top 10 instances.
The chart also includes comments worth noting.
| Rank | Country and Number | Most Frequent Term and Number | Comment |
| 1 | Egypt-68,419 | United States-87,569 | Pro-democracy movements. |
| 2 | Syria-55,932 | Bashar Al-Assad-93,963 | Syria’s far fewer “protests” can involve heavy artillery. |
| 3 | China-43,278 | United States-40,887 | China experiences about 500 protests every day, yet its protests tend to be small and largely peaceful. |
| 4 | India-39,006 | United States-20,015 | Related to India’s anti-corruption movement. |
| 5 | United Kingdom-28,074 | London-41,556 | Related to anti-austerity protests |
| 6 | Russia-24,957 | Validimir Putin-43,104 | Related to Vladimir Putin’s contested reelection. |
| 7 | Lybia-24,744 | United States-60,265 | Pro-democracy movements. |
| 8 | Greece-22,246 | European Union-27,527 | Related to anti-austerity protests. |
| 9 | Spain-20,733 | Madrid-12,114 | Worsening economy. |
| 10 | Pakistan-20,516 | United States-33,220 | Anti-drone protests. |
All of this provides a simplistic view of the type of intelligence analysis that tools like Record Future makes can provide.
But Recorded Future continues to make improvements in technology and content richness, which is helping to make relationships between people, places and things more transparent, and potentially more predictable.
Such tools, of course, won’t be enough. There is still a need for organizations to improve the quality of their internal data and analysis work, so “they know what they know,” and an increasing need for data scientists to improve the quality and richness of their analysis of data of the kind being generated every minute via social media.