A group of technology leaders came to the Capitol this week to make the case that the age of “big data” is not only upon us, but also represents a powerful and practical way for federal agencies to create substantially more value for the public – for relatively little incremental investment.

That comes as refreshing news compared to the relentless promises of big savings that accompanied the dawn of cloud computing, which has proven far trickier to implement. The difference is due in large measure to the fact that big data is really more of a phenomenon than a technology.

“Data is one of our most valuable emerging natural resources,” said Steve Mills, senior vice president and group executive at IBM. “The implications of capturing value from big data are limitless for making business, the global economy and our society work better.”

Mills, who oversees 110,000 employees and $40 billion of IBM’s worldwide software and systems business, co-chairs the TechAmerica Foundation‘s Big Data Commission, along with SAP global executive vice president, Steve Lucas.

The commission just released a report to help government officials better understand the potential of big data sets — and provide a practical roadmap to help them grasp the “art of the possible” in federal agencies.

The report, “Demystifying Big Data,” focuses on how ever-cheaper computing power – and yes, cloud computing – are making it economically feasible to analyze and make strategic sense out of vast reservoirs of data at more and more agencies. Big data capabilities are no longer limited to NASA or the Defense or Health and Human Services Departments.

“The cost of capacity has come down so significantly, that solving big data problems is quite affordable,” for most agencies, Mills said in an interview with AOL Government, just before presenting the report’s findings to a crowd of government and industry executives gathered in a Capitol Visitor Center conference room Oct. 3.

Big data is something agencies can begin doing “incrementally on their own. It doesn’t need Congress to step in,” or a new round of IT investment, Mills said.

From hindsight to foresight

More to the point, the emergence of big data capabilities is making it increasingly viable for law enforcement, healthcare, homeland security and other federal offices to analyze data and make critical decisions in real time.

Bill Perlowitz, chief technology officer for Wyle, a science, technology and engineering group for Wyle, and one of the commission’s leaders, described the transformational power that agencies are beginning to grasp as they move from “hindsight, to insight, to foresight” in divining meaning from their data.

Perlowitz said big data analytics will allow agencies to examine four questions:

• What happened – which is currently accomplished through forensics.
• What is happening – which supports real-time decision-making.
• What will happen – through what he called predictive analytics.
• What would happen if we do this thing – which he called prescriptive analytics.

The applications are already accelerating efforts, for instance, in detecting social security, tax or health benefit fraud as it is occurring, instead of after the fact. But the power of big data is expected to continue to ripple throughout business, government and society.

“Imagine a world,” said SAP’s Steve Lucas “where a growing number of elderly and sick people receive better healthcare outcomes at lower costs; where there is reduced congestion on the roads due to real time insight into traffic patterns and events; or greater accuracy in prediction of severe weather events.”

Growing demand for analysts

As with most new technology waves, the rise of big data brings about its own set of challenges.

Harnessing the overwhelming volume, velocity, and variety of data — while also ensuring the data’s veracity — are just a few of the issues agencies and industry will need to tackle.

Perhaps one the most significant challenges, though, said Mills and others on the commission, is not the technology. Rather, it’s the need for analysts who possess the right combination of skills capable of merging the knowledge and experience of a particular field, such as healthcare research, with the technical prowess needed to analyze big data pools.

“We’re right on the cusp of having a shortage of skilled analysts,” capable of working with big data sets, acknowledged Mills. Indeed, by 2018, McKinsey Global Institute estimates the U.S. will face a shortfall of 140,000 to 190,000 qualified big data analysts.

“The acute shortage of analytics professionals and data-savvy managers,” is expected to become a pressing problem for agencies as the volume of data continues to escalate, said Dr. Michael Rappa director of the Institute for Advanced Analytics at North Carolina State University and an academic co-chair on the commission.

While Rappa envisions that shortage can be addressed through creative partnerships between industry, government and universities, the explosion of data already defies comprehension. The U.S. Government in 2009 alone produced 848 petabytes (million gigabytes) of data. That figure is dwarfed by the data being produced in the U.S. health field, which will soon reach zetabyte (million petabytes) in scale.

Another major concern is privacy rules governing personally identifiable information.

Some 40 laws provide various forms of privacy protection in the U.S. alone, the report notes. Even more stringent rules in the European Union have held back applications of big data in marketing and fraud reduction, notes Michael Nelson, a former White House and FCC advisor and a technology policy analyst for Bloomberg Government.

All that could be overwhelming for agencies. But the question of how and where to start is relatively easy to answer, the commission concluded.

“You don’t need to spend a lot of money or take a lot of time. You can take data sets first and try it, and analyze it, and then make decisions about how you pull in more,” said Teresa Carlson vice president of Amazon’s global public sector.

Carlson, who also played a leading role on the commission, said that the ability to buy computing power as a service, on an as-needed basis sign, significantly lowers the risk for agencies to begin testing big data applications.

The key is to identify small number of mission or business requirements and “define and develop underpinning use cases that would create value for both the agency and the public,” the report recommended.

Central to the TechAmerica group’s findings is the fact that the federal government already has much of the data in hand and can focus mostly on training workers to analyze it.

“The good news is that the government doesn’t have to plow money into pure research in order to achieve results, because the commercial, off-the-shelf technology has become so robust,” said Mills.

Among the report’s recommendations is for agencies to name a Chief Data Officer (CDO), citing an example already established by the FCC. The report stated a CDO would “generate and promulgate a governmentwide data vision, to coordinate activities, and to minimize duplication.”

The commission said that like a federal chief information officer, there should be a federal CDO inside the Office of Management and Budget, who would “bring cohesive focus and discipline to leveraging the government’s data assets to drive change, improve performance and increase competitiveness.”