As Federal Computer Week‘s Alice Lipowicz notes in her ongoing coverage of the troubled relaunch of USAJobs.gov, “It is unusual to see a federal IT story that generates the kind of strong emotions that the USAJobs 3.0 launch has done in the past several weeks.”
Officials at the Office of Personnel Management have come under withering attack for failing to anticipate the initial surge in user demand or the need for greater system and help desk capacity that followed.
By most accounts, federal job seekers, USAJobs.gov partners and government critics have all had good reason to come down hard on OPM for the initial failings of its new job site.
Lost in almost all of the coverage, however, has been the determined behind-the-scenes crises management efforts at OPM to get on top of the fiasco and set things right.
A lot was at stake with USAJobs 3.0. There were many good reasons–and plenty of risks–to migrating 17 million user accounts and more than 6,000 job opening from a system outsourced to Monster to one that promised to advantages in being less proprietary and more flexible. But OPM got caught flat footed when thousands of users couldn’t get onto the new system when it went live Oct. 11.
While certainly not flawless, the degree to which OPM’s senior management stepped in to address the unfolding crises, rather than fixing blame on the system’s designers, deserves at least some credit, beginning with the leadership style taken by OPM Director John Berry (pictured above.)
Berry may not have seen the train wreck coming, but he demonstrated what separates good leaders from mediocre ones by taking total ownership of the problem and then finding good people to help him–bringing in Kathy Dillaman–and by offering the resources needed to help the USAJobs management team find a solution.
Dillaman, a senior policy adviser, is perhaps best known for dramatically reducing the time it took OPM to work through a massive backlog of security clearance investigations for federal agencies a couple years ago.
When she and her team were tasked to fix the problem, OPM was taking an average of 115 days to complete 90% of initial security clearance investigations, according to fiscal 2007 figures. Three years later, that average was down to just 39 days–on 620,000 comparable security clearance investigations.
While that improvement probably wouldn’t earn a resounding endorsement in the private sector, it nonetheless represented some herculean work in back office policies and IT system overhauls, including: the development of a Central Verification System that captured information on security, suitability and credentialing decisions across the government; automated forms and record checks for processing investigations; digital fingerprint capturing; and an upgrade of OPM’s processing system that allowed secure web-based interactions between prospective candidates, employing agencies and OPM.
Fortunately for Berry, Dillaman, who retired from OPM last February after 35 years of government service, agreed to continue serving as a senior policy adviser at OPM.
Berry’s decision to assign someone of her caliber to the USAJobs mess was one of several right steps he and OPM took. The next was the decision to get the unvarnished facts and analyze as quickly and thoroughly as possible what was the at the root of the problems.
And there were many: While it’s rarely a good idea to blame your customers for the problem, the reality was thousands and thousands of users needed help remembering or resetting their passwords to access the new system, resulting in almost 12,000 help desk tickets just for password resets.
But OPM was just as at fault in underestimating the demand on its systems, which spiked into the red zone of peak capacity in seven out of the first 10 days of operation. Not until OPM added 10 extra virtual servers and called on Akamai to balance content delivery did the capacity problems fall back into line. And then there was the geographically-dysfunctional search algorithm that famously mismatched results with queries. It remains hard to fathom that USAJobs engineers didn’t catch or foresee these problems, despite having 18 months to develop and test the system.
It also didn’t help that OPM’s daily progress announcements, which dressed up gains in completed job applications as signs of progress, did little to quell an unruly mob moaning for a return to the old system on USAJobs’ Facebook page.
However, the rapid analysis work and Dillaman’s attention quickly made clear what OPM needed to fix and got USAJobs’ managers focused on getting the right wheels in motion to mitigate the websites problems.
Dillaman’s assessment of the situation, and the data that helped affirm that the problems were getting fixed, are summarized in a slide deck presented to journalists last Thursday (slides available here.)
Those facts gave Berry the confidence to declare, “I believe we have turned the corner. The trends are all moving in the right direction. Obviously, we made some mistakes, and I want to apologize to all the applicants who have been dealing with the bumps in the new system,” Berry said. “There is no question but that we were overwhelmed.”
In publicly saying so, Berry demonstrated a level of humility that though requisite in circumstances such as these, nonetheless reflected the right tone by the government’s leading personnel official.
Berry deserves credit for one other thing: While the staff that launched USAJobs 3.0 no doubt took its share of flaming arrows from all sides, Berry publicly praised his managers and employees for their efforts to make things right.
While OPM’s troubles aren’t over, and the agency will likely be licking self-inflicted wounds for some time to come, there may be some solace in learning from what OPM’s management did deliberately to address the USAJobs 3.0 launch disaster.
Part of that lesson may be found in Dillaman’s slides, which though sanitized, provide a revealing glimpse of what OPM faced and how the site’s capacity and help desk metrics began to change after OPM began tackling the core issues.
What went wrong with USAJobs 3.0 deserves further investigation. But as Apollo 13 taught the nation, even the best-engineered efforts can turn run into trouble. It’s how leaders respond that makes the difference and OPM by most appearances took the high road in demonstrating that.