When Wolf Tombe took over as chief technology officer at the Department of Homeland Security’s Customs and Border Protection Agency (CBP), there were more than 3,000 deployed technologies, a major data center had run out of room and power, yet server utilization was only 10 percent, and almost every server in the agency had a different configuration and operating system.

That was 8 years ago.

Slowly, Tombe began the herculean effort of standardizing and modernizing the vast CBP IT infrastructure – an infrastructure that operates around the clock and around the world supporting 35 billion SQL database transactions per day and 10 petabytes (1,000 terabytes) of storage.

Then, two years ago something else happened. As the federal IT world began humming to the tune of cloud computing and the unprecedented efficiencies and cost savings associated with software as a service, Tombe came face-to-face with a new fiscal reality. CBP saw its budget reduced to the tune of about $335 million. Suddenly, standardization and cloud computing were less about choosing technologies and more about choosing success or failure.

“For us, doing more with less is survival mode. It’s the only way we’re going to survive,” said Tombe, who spoke (pictured above) at the government technology symposium sponsored by Red Hat on Nov. 16 in Washington, D.C. “Reducing costs is critical,” he said. And a $335 million reduction in funding over two years “is not for the faint of heart.”

The biggest obstacle is making sure that your applications are cloud ready.”

When Diversity Is A Bad Thing

When you consider what the IT landscape looked like at CBP when Tombe became CTO it is easy to understand why it has taken so long to finally get to the point where the agency is now, as Tombe puts it, “in good shape to pursue cloud” computing.

The agency had more than 3,000 applications deployed across the enterprise. At least one data center had run out of physical space and power capacity, while the servers located in that center were only experiencing a 10 percent to 15 percent utilization rate. The agency also needed a massive maintenance capability to manage updates for those thousands of servers, most of which ran on different operating systems and different configurations.

Application architecture was also “all over the map,” according to Tombe. “We were developing in Java, we were developing in .net, and we were developing in COBOL. If there’s a language out there we were developing in it,” he said. “There were virtually no standards for how those applications were developed.”

All of that diversity led to higher costs.

The Effort to Standardize

By 2007, CBP had developed its first enterprise technical architecture and began the push to standardize. And in the last year Tombe has managed to sunset 1,000 of the 3,000 applications deployed throughout the CBP infrastructure.

Today, CBP’s target technical infrastructure is based on software re-use and open standards.

“Everything is service based,” said Tombe. “I really don’t need five program offices building an authentication capability. One of them will build it for the other four so there will be a single enterprise authentication service. I don’t need five different logging capabilities, and I don’t need five different auditing capabilities.”

The push toward a common infrastructure has also given CBP the ability to divest technologies and move toward bulk purchases and enterprise licenses for the first time. As a result, the agency is experiencing more purchasing and negotiating power.

“If a vendor wants to play in the common infrastructure they have to promote open standards,” said Tombe.

As an example, CBP is currently the world’s largest consumer of IBM Websphere MQ , a middleware product that enables distributed systems to communicate. “It’s a fine product,” said Tombe, “but it’s an expensive product.”

As a result, CBP is now looking at things like the open standards-based version, Advanced Message Queuing Protocol (AMQP). “We’re making those moves off of vendor proprietary [and] more expensive capabilities onto open standards and lower cost capabilities.”

And although Tombe and the rest of the agency may not have realized it when they started this effort, CBP is now perfectly positioned to leverage the new massive federal push to cloud-based computing.

“We’re actually looking at our common infrastructure as a private cloud capability today,” Tombe said. “We are in good shape to pursue cloud and we are pursuing it privately within our DHS data centers today. Although I’m not going to rule out that there will be public cloud opportunities in the near future. The reason I don’t want to take public cloud computing off the table for us is I think there are tremendous cost [savings] opportunities there.”

More Work To Be Done

So far, stress testing of CBP’s standardized cloud infrastructure has revealed it is capable of significantly out-performing the legacy infrastructure.

The most recent test revealed the enterprise Linux infrastructure with an 8-node virtual cluster was capable of delivering a less than one second response time to 18,000 concurrent Web application users.

But standardization and preparing the CBP enterprise for cloud computing options has revealed an obstacle. And it doesn’t, as most people might think, have anything to do with security. For Tombe, the biggest challenge is a slow, cumbersome acquisition process.

“It’s not uncommon for a reasonable acquisition to take anywhere from 18 to 36 months,” said Tombe. “When you’re talking about technology that means by the time I get that thing it’s obsolete,” he said. “As you look at moving toward service models and cloud computing it makes very little sense to have a cloud capability where I can burst up in a matter of hours if it takes me a year to get the acquisition through.”

Although security remains a legitimate concern for most agencies, Tombe’s experience at CBP leads him to believe there are bigger challenges to overcome when moving a large federal agency to the cloud. “I think security is the easy part of cloud adoption,” he said.

“The biggest obstacle is making sure that your applications are cloud ready.”