AT&T Network Disaster
AT&T Network Disaster
AT & T is a large telecommunication provider in the United States/ Its Network Disaster Recovery (NDR) team was established in the 1990s. Currently there are about 29 full-time employees the NDR team, but the total expanded emergency management team has a population of about 100 people who deal with emergency management and business continuity. The AT and T NDR is committed to ensure that their customers are conceding even when encountering various events such as catastrophic events and unpredicted events, whereby it ensures the maintenance of reliability of the global network of AT&T. The mission of this team is to ensure recovery of AT&T voice as well as data service network elements in an area that is affected by the disasters. AT&T generally invests about 600 million of its finances in the United States and another $15 million in international incidences. They identify that telecommunication is an integral component to businesses and government customers who experience a disaster, whereby it is required by the affected areas as well as the country at large (Buchnan, 2010).
According to Global Disaster Prepared Center (2014), the AT&T’s Natural Disaster Recovery plan has three main goals which include; to route the telecommunication traffics that are not involved in the event around the affected region. To give access affected by disaster, communication access to the other parts of the world. And to recover the communication services back to normal condition through repair and restoration as fast as possible after the event of distaste.
The AT&T Network Disaster Recovery System
Permanent staffs ate the NDR are divided in accordance with the functions. The work of responding to the disaster recovery plan at the tire is done with a multiple of functional groups. All the functional groups are required to work as a team and collaborate with each other in the process of dealing with the disaster. For instance, the Assets management team will be helped by the Technology operation team, with installation and modification of equipment (Kapucu, 2013). All the groups work within the wider team of operation, once they reach the recovery site with their equipment. The NDR team is also supported by the part-time staff that ensure that everything they do is in extreme bases ranging from network to radio frequency to geography and location of every central office as well as Network location. The operation team works at the incident command, even planning and management, whereby it includes the special operations, finance, logistics, system support and developments and well as training. The technology operations are designed of recovery equipment, construction management, engineering and testing. The strategic planning is a group that anticipates future recovery equipment that requires the AT&T network to evolve and also plans for the future capital expenditure. The Asset management team maintains the extended ranges of warehouse equipment as well as escort equipment in the deployment. Finally, the other team is the Most of World (MOW) which is deployed in incidence involving the rest of the world. The MOW team includes all the other teams and rolled up under same functions since they’re required equipment’s are limited as compared to the once required by the United States (AT&T Natural Disaster Recovery, 2005).
This team responds to any kind of disaster that range from the large GSM switch to internet data center and multiple cell-sites. The team also has part-time members who are in most cases volunteers from the all parts of the United States whereby each one of them is required to be part of the team of Natural Disaster recovery. However, not all volunteers and applicants are accepted since some disaster situation requires particular skills and personality who repeatedly work in the conditions (Jordan & Library of Congress, 2005). The volunteers are assessed whenever there are these exercises and then deployed. Those selected are then trained fully so that they can become full members of the NDR. This enables all the NDR management team as well as the volunteers to be suitable and competent in the performance of the duties.
National Incident management system
The national incident management system is comprehensive systems that work all over the United States, while using systematic approaches to the management of incidents. It has a set of prepared concepts and principles that are applicable in the response process of any incident. These principles are essential and common in the operating systems as well as the interoperability of information management and communication. The system is standardized with resource management procedures so as to ensure that all the resources in the AT&T NDR are applied adequacy and effectively. The system and resources are scalable in that they can be used in the operation of all incident response.
During Disaster response, there are many logical processes that are required to be followed. There are also papers that require to be filed with the Department of Transport which details the beginning to end of the trips, routes and roads, weights of the trailers in the fleets, safety and any cases of exemptions from weigh stations because of the critical nature of the payload (Rapp, 2011) There are also clearance reports that are required for equipment’s and personnel to enter an area that is encountered by disasters depending on the nature of the occurring disaster which is called Access and Credentialing. Once a disaster occurs in a place the overall events are in most cases handled by the various public safety organizations such as the police and fire department, who seclude the place from nay access by others hence. In order to be permitted access to the disaster region, they NDR team is required to present the appropriate level of credentials showing that the team has the ability to perform in the damage assessment as well as engage in recovery and restoration duties.
In the case of international events, similar paperwork is required to be filled by the international deployment team, whereby all the equipment of the MOW team is designed to fly and travel by road. In this case they are also aware of the existence of other advanced clearance procedures such as the appropriate import and export fees as well as the telecommunication equipment customs. The AT&T ensures that they monitor all the regulations within their key markets and also maintain a constant contact with the relevant authorities as well as the trade groups in every country.
The NDR participates in about four full-scale recovery exercises annually since its establishment in 1993. Disasters exercise involve testing the process of the recovery, ranging from the initial call out, team member is trained to the new technologies and turn-ups of technology, testing and action. In this case, the teams also expect to experience variation in the weather due to the geographies and zone of the disaster in the region whereby they might engage in the recovery exercise in areas with extreme hot climate and other extreme cold. They are trained on the various the level of preparedness of such cases (Rapp, 2011). For instance, the team was sent for an exercise in Tampa, Florida, which has a temperature of 35 degrees while in the other one at a salt-lake City in Utah which had snow of 60cm. These teams also work in other parts of the world, hence they must be trained on what to expect in the new regions they are deployed (Arean, 2013).
The NDR team is also required to work in collaboration with the local emergency agencies as outlined in the NIMS guidelines. For instance, when the teams were deployed in special operation exercise (HAZNAT) which were conducted in 2014Septener in Ohio, the operation involved assimilation with more than twenty five state, regional and local emergency response agencies, whereby the AT&T worked as the private partner agency. Disaster sites without gathering the necessary information regarding the areas such as the scale and nature of the disaster (Rapp, 2011). Therefore, the AT&T does engage in a blind exercise that, are they not entering any disaster area without gathering the appropriate information. They recognize that blind exercises are extremely expensive, especially in the large and extreme countries that require extensive travel requirement.
Recovery Exercise process
The first decision the AT&T is required to make before they engage in the recovery process is to identify the exercise. The AT&T NDR has many centrals offices in the United State. They consider the center that is near the disaster region, but not affected by the incident. The next stem they identify the services required in the site by examining the site so as to identify the configurations necessary in the recovery process in that location which are mapped in accordance with the personnel and equipment trailers needed. These services range from the traditional SS7 to IP as well as TV services. In larger incidence they can obtain equipment from about four warehouses and load them in about forty trailers for transportation to the site (ASEA (Conference: 2008-), Kim, & DRBC (Conference), GST (Conference), 2012). Once the required map is completed, the exercise incident command identifies the regulation and logistics required so as to meet the requirement of the US department of transportations. While the trailer is on the road, they require protection, hence they warehouse team also travel in convoy.
In every exercise, it is assumed the they available resources add up to 40 percent of the local staffing while the rest is unavailable that is through trauma, fatality and searching for loved ones. For this reason, an arrangement is made of flying in or driving more NDR staff from other locations in the United States in the recovery site (Jordan & Library of Congress, 2005). The team is also required to secure the site since the required space many involve a large area, as well as access to the backhaul is also critical. The identified optimal recovery location site need to be adjacent to the destroyed site so as to allow easy access to the most fire-optic ring that impose most of the backhaul. However, this is usually a critical process and often impossible. For instance the case of recovery in the Word Trace Centre site in 2011 September the optimum recovery location was positioned across Hudson River in the New Jersey.
This preparation process only reflects the initial stage of getting the personnel and equipment necessary to the exercise in the location. Once the personnel and equipment arrive at the deployed site, other subsequent procedures follow so as to allow the team to successfully complete the exercise. It is also very essential to recognize that the equipment is brought to the recovery site in a manner that is designed to ensure connectivity of every power outlet and also to each other. Is done with the recognition that ate time power access may be challenged while in some exercises it may require a high power voltage. The positioning and leveling of the trailers along the power connection as well as grounding may also take some hours to complete. Once the equipment is powered-up configurations is loaded into every need in accordance with the determined services set in the planning stage. The exercise imitates the real disaster zone where the first priority is established through the service network only when the communication is re-established for use by the local community.
Services in Support of the Emergency management Agencies
Once the NDR team is deployed, it provides emergency communication support for the various emergency management agencies whereby their relationship can be a formal structure within the incident management structure of the agency or many be fluid or ad hoc. For instance, in the formal relationship, the deployment of satellites COLT aims to provide mobile services in the remote wildfire command camp in the United States. The wildfire is usually managed by the Federal teams using the NIMS model (Grath, 2014). The AT&T teas and their services become functional where they report directly to the commanders of the incidence through the appropriate units of communication. They are usually every day check-ins at the start of the day and the end so as to assess the need of the camp. The NDR has also provided other services such as the management of the radios, on-site PBX, repeaters and satellites based internet services.
The AT&T NDR team is also requesting to deploy satellite COLT and ECV so as to temporally provide communication services at the commands and relief centers and the occurrence of natural disaster such as the tornado or hurricane. In case the Federal, state or local emergency recovery agencies are massed in an area and require mobile, VolP communication or Wi-Fi, the NDR team is required to report through the AT&T’s Incident Command System stems to the NCS (National Communication System) representative located in the Department of Homeland Security who received the requests of the services. For instance, in the Hurricane sandy Response, the ECV deployment for relief in the site was located in the Rockaways are in Queens whereby the AT&T NDR provided services for wired internet to the FEMA and NYC mobile command Centre (Grath, 2014).
The social Operations Team, which constitute of 30 people and a team of volunteers with regular assignment such as operations, environmental health and safety and information technology in the AT&T team operations. The special team is fully functional of hazardous materials (HAZAT). The team was established after the 9/11 incident, whereby it is called upon in situations arising that require skilled people to attend to them and repairs the network infrastructure broken as well as equipment’s whenever they are nuclear and chemical spills and other hazardous materials. In regard to the size and the weights of the equipment, there is strict consideration of maintaining physical and fitness of the team members. The special team is required to be part of the NDR is that they can also have access to sites where nuclear, particle and chemical radiations are present and do remediation in the network equipment (Grath, 2014).
The Customers of AT&T enterprises recognize the recovery exercise to present a competitive advantage in the management of business continuity my saving the network of the company in the occurrence of a disaster. Telecommunication is recognized to be a critical part of the business during the occurrence of the disaster and AT&T have demonstrated their expertise in protecting the interest and revenue of the business from the existing customer which lead to the attraction of new ones.
Learning from experience
The AT&T NDR utilized the ICS software applications which help the, in management of inrfomatsiom and documentation of the progress of deployment exercise. Their actives are logged in throughout the event by the command team member, but other entities are done by the AT&T staff member serving as the dedicated scribes. This allows issues arising in the recovery process and equipment to be a logger and tracker in the course of all the shifts (Arean, 2013). The issue is also recovered quickly, but whenever the process requires modification or long-term repair, the occurrence is tracked using the software after the recovery and resolution of the event but the Standard and Documentation team in the NDR. For instance, during the Hartford exercise in 2013, one of the team members who was responsible for the power reported that is the size of the limber brought into the site were inadequate of fabricate the cable bridging and clears the spill berms since the required bridge was supposed to be 3 feet wide. This documentation was altered to reflect the new requirement. This is the type of issue that is discovered during the exercise rather than deployment period.
Methods of recovery
The AT&T NDR has four major recovery methods of physical network disaster recovery. These tactics include the static recovery, mobile recovery, hybrid recovery as well as the Vendor Supported Recovery solution. In the mobile recovery solution, custom trailers are designed, constructed and engineered in such a way that they provide and support network element recovery. The solution provides a technology for replacement in the self-sufficient deployable unit (Global Disaster Prepared Center, 2014).
In the static recovery solution, the assets are dedicated in the AT&T network so that they can enable the network disaster recovery. This solution is used for the elements of the network that appropriately scale of the mobile solution.
Hybrid recovery solution consisted of the utilization of existing mobile asses such as the 53-foot trailer which are equipped with HVAC and power whereby they are constructed to provide support of timely installation of the elements of the network. In this case the equipment required are moved from the areas they exist, such as the AT&T equipment testing laboratory, training facilities, vendor stock or maintenance spares and shipped to the disaster site. This method is an alternative of having to dedicate network elements seem to be installed on the mobile recovery trailer. The strategy is used as the teams constructed a dedicated solution or the requirement of small miscellaneous equipment which are easily shipped (AT&T Natural Disaster Recovery, 2005).
Vendor supported solution involved an agreement with the telecommunication equipment vendor that provide recovery equipment that comes from the existing sticks or “next off the line materials”. The strategy is usually used as the team constructs a dedicated solution or the requirement of small miscellaneous equipment which are easily shipped (AT&T Natural Disaster Recovery, 2005).
The fundamental goal of the NDR organization is, it strives in restoring the functionality of the networked elements, central offices and work centers in the network of the AT&T that were destroyed and rendered useless after the occurrence of a natural or man-made disaster. The goals of AT&T are to route the telecommunication traffics that are not involved in the event around the affected region. To give access affected by disaster, communication access to the other parts of the world. And to recover the communication services back to normal condition through repair and restoration as fast as possible after the event of distaste. In order to perform their function of recovery and managing the network system, the team is required to have sufficient planning for the equipment, personnel and the process to use in the recovery process. This enables those to follow the procedure stated in the NIMS guidelines and also to work in collaboration with other networks in the recovery process.
Arean, O. (2013). Disaster recovery in the cloud. Network Security, 2013(9), 5-7.
ASEA (Conference : 2008- ), Kim, T., & DRBC (Conference), GST (Conference). (2012). Computer applications for software engineering, disaster recovery, and business continuity: International Conferences, ASEA and DRBC 2012, held in conjunction with GST 2012, Jeju Island, Korea, November 28-December 2, 2012. Proceedings. Berlin: Springer.
AT&T Natural Disaster Recovery. (2005, January 25). AT&T Network Continuity Overview. Retrieved June 23, 2018, from AT&T Best Practices: https://www.corp.att.com/ndr/pdf/cpi_5181.pdf
Buchnan, M. (2010, February 8). Inside AT&T’s National Disaster Recovery Batcave: Who AT&T Calls When the Death Star Explodes. Retrieved June 23, 2018, from Gizmodo: https://gizmodo.com/5602414/inside-atts-national-disaster-recovery-bunker-who-att-calls-when-the-death-star-explodes
Dolewski, R. (2008). System i disaster recovery planning. Lewisville, TX: MC Press Online, LP.
Global Disaster Prepared Center . (2014). Disaster Response: Business As Usual: How AT&T deals with Natural Disasters. Retrieved June 23, 2018, from Global Disaster Prepared Center: https://www.preparecenter.org/resources/disaster-response-business-usual-how-att-deals-natural-disasters
Grath, F. (2014). Disaster Response: Business As Ussual-Hiw AT&T deals with Naturak Disasters. Atlanta,GA: GSMA.
Jordan, M., & Library of Congress. (2005). Federal disaster recovery programs: Brief summaries. Washington, DC: Congressional Research Service, LIbrary of Congress.
Kapucu, N. (2013). Collaborative Governance and Disaster Recovery: The National Disaster Recovery Framework (NDRF) in the U.S. Disaster Recovery, 41-59. doi:10.1007/978-4-431-54255-1_3
Rapp, R. R. (2011). Disaster recovery project management: Bringing order from chaos. West Lafayette, IN: Purdue University Press.