Donna Roy, Executive Director of NIEM, recently sent a ‘State-of-NIEM’ letter to the community thanking them for continuing to help develop a plan for scaling NIEM and addressing challenges facing the program.
The
National Information Exchange Model is an XML-based information exchange framework from the United States. NIEM represents a collaborative partnership of agencies and organizations across all levels of government (federal, state, tribal, and local) and with private industry. The purpose of this partnership is to effectively and efficiently share critical information at key decision points throughout the whole of the justice, public safety, emergency and disaster management, intelligence, and homeland security enterprise. NIEM is designed to develop, disseminate, and support enterprise-wide information exchange standards and processes that will enable jurisdictions to automate information sharing.
In short, NIEM is not a software program, database, network, or computer system, it is a message format. Probably the most easily understood and used product of NIEM is the
data dictionary spreadsheet. It includes all of the element names that are organized hierarchically under core data components (person, property, organization, etc.) with hyperlinks to related elements. This spreadsheet also provides information on the type of data being represented (date, integer, string, etc.) and a precise, context-rich definition of each dictionary component.
Donna Roy‘s memo says, “Adding new domains to the NIEM model is manually intensive and we have a choke point for growth that is restricting our ability to address new demand.”
In simple terms, the NIEM model does not scale to the “big data” the Intelligence Community Loves and needs.
Our world has entered the era of “big data.” Estimates are that 2,500 exabytes of new information will be added in 2012, with complex, unstructured Internet information (not business transaction relational databases) as the primary driver.
This “big data” is characterized by three V’s: Volume, Variety, and Velocity. The merger between structured and unstructured data is fueling the next revolution in data integration and business intelligence as shown in the diagram above.
A real-world example of Velocity is large scale XML processing. Before: 3 gigabytes of XML could take 25 minutes across 6 servers; After: it takes just 50 seconds in Amazon EMR with 16 nodes.
An example of Volume and Variety is NIEM itself, when Donna Roy says:
“If NIEM is to move beyond its current success and realize its full value, implementers of NIEM need to easily leverage, create, and maintain over their lifecycle reusable information exchanges with a significantly higher level of ease and lower cost than our current model and processes provide. I would like to see us put forth a challenge to industry and our standards community partners to propose a NIEM 3.0 that addresses complexity around using, governing, and managing the model.”
I suggest they look to their partners in the Intelligence Community who have a new shared services in the secure cloud architecture and strategy as articulated by their leaders such as NGA Director Letitia Long and CIA CTO Gus Hunt. These shared services transform structured and unstructured data into the underlying facts, entities, relationships, and associated terms to eliminate the need for data analysts to read in order to understand. These shared services address the volume, velocity and variety of today’s big data challenges to provide automated understanding for big data.
The technology for dealing with massive volumes of unstructured data and integrating it with structured data has progressed rapidly since NIEM was started in 2005 and would help Dona Roy and NIEM do as their want to do: “open our thinking to continue to provide new ways to work with NIEM in ways we cannot yet imagine: what is NIEM in a mobile strategy, how does NIEM allow for increased Cyber Security for Cloud Computing, and what is NIEM outside of the current XML use case, are those that come to mind.”