Every experienced CIO sees IT failures throughout his or her career. With studies reporting that up to68 percent of IT projects fail, managing these situations is an essential part of life in enterprise technology. Although most IT departments try to hide breakdowns and trouble, hoping they will quietly go away, a recent guest on CXO-Talk takes the opposite approach, trumpeting failures with massive transparency.
Dr. John Halamka is Chief Information Officer of Beth Israel Deaconess Medical Center, Chairman of the New England Healthcare Exchange Network (NEHEN), Co-Chair of the HIT Standards Committee, a full Professor at Harvard Medical School, and a practicing Emergency Physician. He also runs a working farm, plays Japanese flute, and is of the brightest and most interesting CIOs in the world. All of which means that when he speaks I pay attention.
Halamka is no stranger to managing an IT meltdown. In 2002, the backbone of Harvard’s teaching hospital system, Beth Israel Deaconess Medical Center, suffered a crippling network outage. For four long days, the hospital system had to operate using manual, paper-based processes without any computer network. Halamka created a detailed technical presentation explaining what happened. As credit to the steps Halamka took following the outage, Network World subsequently recognized Beth Israel Deaconess for network innovation.
An article in the New England Journal of Medicine summarizes the situation:
In the fall of 2002, the Beth Israel Deaconess Medical Center was operating an extended computer network designed to meet the requirements of a much simpler environment. In addition, some network equipment was old and needed to be replaced. The principal point of failure was a software program for directing traffic on the network. The program was overwhelmed by a combination of data volume and network complexity that exceeded the software’s specifications. Once the network became dysfunctional, the diagnostic tools for managing it, which had to be used from within the network, were of no help.
On his blog, Halamka states the facts even more succinctly, while accepting full responsibility himself:
In the years after the 1996 merger of the Beth Israel and Deaconess Hospitals, operating losses caused several years of capital starvation (the network budget in 2002 was $50,000 for the entire $2 billion dollar enterprise). We were not available to invest in infrastructure, so our network components were beyond their useful life.
However, it was not this infrastructure underinvestment that was the root cause of the problem, it was my lack of enterprise network infrastructure knowledge.
During the CXO-Talk conversation, Halamka emphasized openness, accepting responsibility, and transparency during a crisis.
On crisis management: There are two ways you can get through a crisis. You can hide it, in which case when they find out you will end up on page one. Or, you can invite the press to watch the decision-making and the stress and make them understand, at a deep level, the issues. Then you end up either on page 37 or as a Pulitzer Prize winning article that might get published in a major trade journal.
I’ve always chosen the latter, which is, “Oh, my network collapsed. Why? What was the impact on the patients? How did I keep my job?” I invited the press in; I brought in the Boston Globe, brought in CIO Magazine. They watched us through the problem resolution.
When I’ve had issues of security, we disclose immediately to federal and state regulatory agencies, and then I go on the road saying to CIO’s, it’s not that you haven’t been hacked, you just haven’t looked.
You know we’re very, very honest about the mistakes you’ve made and somehow the industry forgives me because of my honesty.
Rahm Emanuel was right, let no urgency go unused. And so if my capital budget was zero and then a disaster occurred, suddenly $5 million appeared and I was able to create the infrastructure necessary to keep patient records safe.
On mistakes: Mistakes are made every day. Mistakes are learning experiences and I don’t actually say, “Who is at fault?” I ask, “What was wrong about our process, our strategy, or structure that enabled that mistake?” I never shoot the messenger. So maybe that attitude makes people understand we’re all imperfect and if we can learn from each other, that’s probably not a fire-able offence.
On IT project deadlines: You should never go live based on a deadline. You go live when the product is ready and the people are ready to use the product. If you go live too early no one will ever forget; if you go live late no one will ever remember. Having done hundreds of go-lives in my life, I know there is no naysayer, no pressure, and no political issue important enough to go live too early.
– – –
CxO-Talk brings together prominent executives, authors, and analysts to discuss leadership, transformation, and innovation. Join me and Vala Afshar every Friday for a new episode of CxO-Talk.
(Cross-posted @ ZDNet | Beyond IT Failure Blog)