|
|
|
|
|
|||||||
![]() |
||||||||||
|
     
|
Ground Zero: Business Recovery During the Worst of Times
By Pat Ferdinandi: survivor, witness, listener, and facilitator.
It was a normal occurrence this particular morning. I look at my watch and notice that the train is again about ten minutes late. A fellow commuter sees my aggravations and comments on how our train is taking longer and longer. We continue to complain as we depart, only to notice that the PATH train to downtown New York had been shut down. Oh well, it will be one of those bad commuting days. We walk to the ferry noticing the lines to board were building. I then turn to my husband, who also works in the city, and suggest that we treat ourselves to breakfast in Hoboken, figuring that the mess would have cleared up by the time we finish. As we walk out of the train terminal and look up, we see that this was NOT a normal commuting disruption. We watch in horror as the second plane ploughed into the World Trade Center - which is where I worked every day for the last 23 years! Cell phones were working sporadically while people tried to call loved ones, coworkers and colleagues. The one land line near Hoboken terminal had two people, with Palm Pilots open, notebooks running, giving instructions on initiating disaster recovery plans for their data center. The days that followed were filled with frantic telephone calls and emails checking on the well-being of anyone you knew that could have potentially been effected by this tragedy. The conversation always started with the miracle of how they narrowly escaped and of the misfortune of others who are still "missing." But then the conversation moved on to the business impact of this disaster. Without mentioning specific names, here are seven stories about companies trying to get their business operational. I describe them here to provide food for thought. No disaster recover plan is perfect, however, now is the time to review your policy and procedures for disaster recovery. Spend money, even in these economically challenging times, to minimize business interruption. Do so by staffing a review and planning effort and by buying extra equipment, duplicate networks, develop DOCUMENTATION. Let's review these seven stories to see how some simple and basic steps may save your business from having significant outages. 1. Communications - Human plus internal and external networks. Once people were evacuated and accounted for, they needed to be notified about what to do next. The email and voice mail systems were not operational. Many telephone lines where also down, preventing direct calling of many employees. Though not part of their disaster recovery plan, this financial company continually updated their web site with information on where and when people were to report to work. Additional instructions were provided on how to set up personal Hotmail and Yahoo email accounts if they did not have an existing personal email address. Another financial company spent over $3 billion on information technology this year. Their biggest cost item was their network. Ironically, the network configuration is one area that is not backed up. Software exists that provides this functionality but it was not installed yet. Fortunately, a key network engineer survived the tragedy and had hand-written notes and hard copy descriptions of the network in his briefcase. This enabled him to bring the network communications into working condition from the backup location. Yet another financial company lost their communication between clients. They did not have an automatic means to bring up communication links from their alternate location. They also did not have documentation providing the information to do it manually. Their clients were contacted one by one and provided that information. Some of the clients, being at there alternate location, had to do some research to find out their new connection information and update their network configurations as well. This process took 2 days before full connectivity was established One media company had insufficient telephone lines to support the existing influx of staff at their new facility. They immediately purchased cellular telephones for everyone. It was a sight to see the administrative assistant sitting at a make-shift desk with cell phones for the executives organized with name cards next to each phone. However, it worked until the telephone company could supply additional telephone lines in the building. 2. Automate - Protecting the valuables. It is amazing how much technology exists but is not utilized to protect key assets. Re-insurance is the business of insurance companies insuring themselves. It protects one company from having to carry the burden of a particular disaster. These are very integrated contracts-within-contracts which contain legally binding hand-written comments. Though the systems carry basic information about the contract, it still requires actual physical review of the contract to understand all of the legal nuances. The list of companies residing in the twin towers included reinsurance firms. This is the busy season for reinsurers as renewals are underway. Since the contracts with hand-written comments were disintegrated, one survivor was in the process of calling all of their clients to obtain copies of these marked-up contracts. What was lost are all notes of his analysis of proposed changes to existing contracts. The next step for this company would be to identify the needs for a system that would allow for scanning of documentation and allow for the addition of electronic "sticky notes" that only select individuals can review. Another client is located near ground zero. He may be allowed back in his building in a couple of weeks. Sitting on his desk are corporate tax returns that were originally due September 15th. All he had to do was the make a copy, package and mail it. Luckily, Federal and State agencies issued an extension. Luckily, he will be back in his office before some of the extensions expire. Luckily, some of the information is available at the backup location. Hopefully, the water and soot damage have not made the documents unusable. For those destroyed or unusable, hopefully the information is available at a backup location to reproduce the returns in time. The lesson here - backups should be taken on a regular basis and kept off premise. 3. Adequate Alternate Facilities - Sufficient alternate sites to fund and run the business. Mayor Giuliani represents the strength of New York. I support his decision to get everyone back to work as soon as possible. It is agreed that some business processes are more crucial than others and they have the highest priority for having mirror facilities. It is also important to have backup plans for all non-priority areas of the business. I spoke with one individual who survived by climbing down 86 stories of 2-WTC. After knowing that all of her employees had survived, she arranged for all of them to meet in a hotel room in midtown. They all knew that their area was not on the critical path. Their computer access would be delayed for a few weeks though they would have access to all information as of 9/7/01. Therefore, their first meeting was to discuss what to do to run their small area until computer access was made available. They realize, that any information that was not entered into the system or on their local PC drive was disintegrated. They next defined a process to facilitate the updating when computer access was provided. Another company had two computer facilities, one mirroring the other. They were located one mile from each other. Unfortunately, smack in the middle was the WTC which meant that both facilities were in the one mile freeze zone and were inaccessible for days. Finally, one building was opened to select individuals after they completed a 3-day hazardous material course. Then police escorted them into the building to collect files, tapes, and equipment. Alternate facilities were available but were not mirror images. One of the business areas went to this alternate site that supported development work. Over 200 support personnel were forced to maintain business operations with six very old terminals. Charges for additional terminals and connections to run production were made with the corporate credit card. The business operation that was originally in this building was displaced and their function such down until production was operational. Development was also on hold until one of the main buildings was reopened. An additional week of development was lost waiting for backups of the development system. Luckily, developers had copies of their work. Some had remote access from home. In Retrospect Through sheer perseverance, knowledge, and commitment, all my clients have helped their companies get their business operations running. They have worked many hours including some 72 hours without sleep. I applaud them. They are heroes to the corporations and to every American because of their work to keep our companies and economy going. We all learned from these experiences. I now advise my clients to spend 15 minutes a day thinking about disaster recovery. By year-end, they will have thought of scenarios to minimize business interruption. Again, no plan is perfect. The goal is to have a plan in place with many different scenarios, revise when necessary and ensure that new technical advances are incorporated - especially back-ups. Not doing so is what terrorists hope for. It has now been four full weeks since the disaster. The downtown area is starting to find the "new normal" routine. I am sitting on the train that is, again, five minutes late. I see for the first time my fellow commuter that I talked with that morning. He leaned over and said, "I will never complain about the train being late again." Where do you start? While working with my clients in developing and refining their disaster recover plan, we started with Y2K plans and updated them by asking the following questions. These questions will initiate discussion and start enhancing your disaster recovery plan. Begin your 15 minutes today.
How can we help? SBDi has been working with their clients to review their current plans. They have initiated task forces to walk through scenarios, including those discussed above. Due to the overwhelming number of requests, SBDi has partnered with several firms that have disaster recovery, crisis management, and/or continuity of business design to ensure that we support our customers with this need. Together we review plans and recommend changes to the design of your infrastructure that have extremely high availability and data integrity requirements supporting the continued functioning of the global economy. Both SBDi and the following three partnered firms have the objective to help you ensure that your business has the minimum negative affect by business outages both to system and business workflow related.
Partnerships include (but are not limited to):
        Top of Page | View Current Tip | Get Tips in Your Email! | Visit Our Blog
|
|||||||||
|
||||||||||