[Originally published in the November/December 2020 PSC magazine.]
By Christine Burke Massengale
The Tennessee Advanced Communication Network (TACN) is a statewide P25 communication network serving over 45,000 state, local, federal and non-governmental users. TACN is also the point of contact for the FirstNet Authority in the state of Tennessee, the Office of the Statewide Interoperability Coordinator (SWIC), and the operations management for call processing and dispatching services for the Tennessee Highway Patrol (THP), operating out of four ECCs across the state: Jackson, Nashville, Chattanooga and Knoxville.
CONTINUITY OF OPERATIONS
In late 2018, an All Hazards and Risk Assessment Committee, composed of selected dispatch personnel from all four centers, reviewed continuity of operations plans (COOPs), identified hazards and risks at each center, and highlighted concerns regarding operational or site security issues. Committee recommendations included regular evacuation and/or shelter in place drills. The center managers were tasked with conducting quarterly emergency drills such as personnel recall exercises or center operations fail-overs, where operations would transfer from one center to another (Jackson and Nashville partner, while Chattanooga and Knoxville partner).
In June 2019, TACN participated in a FEMA regional tabletop exercise, Shaken Fury, as part of a series of tabletop, functional and full-scale exercises in response to a 7.7 magnitude earthquake scenario along the New Madrid Fault. This tabletop exercise involved TACN radio techs, ECC personnel and TACN management, as well as many partners across the state and region. It also involved the transfer of operations from the Jackson center to Nashville, testing responsiveness to unexpected “radio or telephone silence” and simulated loss of connectivity between centers.
The radio system provides backhaul linking and dual-path connectivity between the transmitter sites and zone controllers, allowing the system to reroute automatically when a transmitter site is interrupted. This design architecture prevented a statewide outage when the master site was damaged during the tornado outbreak in March 2020.
STORM IMPACT
On the evening of March 2 through the early morning hours of March 3, 2020, a strong system of supercell thunderstorms tracked southeast across the Midwest, forming as many as 10 tornados with damaging hail and wind. As the storms moved
into Tennessee, an EF-3 carved a 60-mile path from Nashville into Smith County and another EF-4 caused severe damage to Putnam County, just west of Cookeville. The widespread damage resulted in hundreds of injuries and 25 deaths.
Among the many businesses and residential structures that were destroyed were critical components of infrastructure for the State of Tennessee’s P25 radio network. A site transmitter and a 140-foot tower at the Tennessee State Prison grounds were toppled. The master site for the mid-TN zone, which consists of servers, switches, zone controllers, generator and more, were housed in a former food warehouse meat locker, a fortified structure, which preserved these key components of the system. Although the microwave tower and cabling were destroyed, the generator and the master site equipment inside the locker were found in working order under a pile of debris and partial collapse of the building itself.
The impact of this loss of connectivity caused the transmitter sites to go into site trunking where each site becomes a stand-alone transmitter, and the radios associated with the site are limited to the coverage area for that site. This also meant most communication from field responders back to the comm center in that coverage area was lost. However, it is important to note the radio system did not go down; the other three zones in the system were fully functional, while the sites in the mid-Tennessee zone were in site trunking. This allowed troopers to communicate with others affiliated with the same site. Approximately 5,100 out of the over 45,000 TACN users were impacted during the recovery period.
In less than 42 hours, a coordinated effort would fully restore connectivity and system functionality. While seemingly nothing short of miraculous, it was the result of both the immediate efforts of well-trained TACN technicians and Motorola engineers coupled with a stroke of luck, which had brought the team of engineers to town for a scheduled upgrade that week, putting the right players in the right place at the right time.
RESPONSE PHASE
As storm warnings went out across the state, the Nashville THP dispatch center began experiencing heavy rain, wind and hail. When a tornado warning was issued, existing protocols for transferring operations to Jackson went into effect, and the employees went to the lower level of the building to shelter in place. The tornado struck the master site shortly before 1 a.m. March 3. As soon as the storm passed, they resumed operations from Jackson (approximately 20 minutes later) and began assessing the issues with the radio system and making necessary notifications to technicians and command staff.
By 2:30 a.m. both Motorola and TACN technicians, accompanied by Tennessee Department of Corrections staff, arrived at the master site to begin assessing the damage. They were able to turn off the generator and power off the equipment in the server room, but it would be late mid-morning before the structure was deemed safe to enter.
By 5 a.m. TACN staff had assembled at the TACN main office network operations center (NOC) to coordinate the response and recovery plans. An additional challenge included the looming threat of the COVID-19 pandemic; social distancing, mask mandates and travel bans were not fully in place yet, but there was added anxiety from bring- ing people from other parts of the region together in one space.
Documentation and incident logging was initiated almost immediately, with a scribe assigned to take notes and time log for an ICS-214. This immediate chronicling of events provided a more accurate timeline of activities, ensuring key details and actions were correctly cataloged for future review. This also eliminated the need to recall and reconstruct events based on memory hours or days later.
The multi-faceted response phase included: immediately establishing communications incident command, including documentation and incident logging; TACN technicians and Motorola engineers work ing to manually remap and configure some of the transmitter sites to route to the Chattanooga master site; THP telecommunicators at all four centers coordinating their efforts to assist troopers in the field with search and rescue efforts underway in several communities while still managing daily operations; TACN staff coordinating with FirstNet to obtain and deploy devices for field responders to assist with communications; and the deployment of incident tactical dispatchers (INTD) to one of the tower sites to provide the link between dispatch and field units while the response district was impacted by site trunking.
RESOURCE DEPLOYMENT
A TACN site-on-wheels (SOW) was deployed to the prison site to assist with radio coverage. However, it was deemed unnecessary as they were communicating adequately via simplex (radio to radio).
Shortly after assembling at the NOC, it was determined that deploying an incident tactical dispatcher (INTD) to operate out of another tower site could fill the coverage gap by providing a point of contact back to the Nashville center. A technician was deployed to outfit the tower site with two radio consolettes, a laptop with internet connectivity and VPN access to CAD, as well as a FirstNet Sonim device. This temporary operation was staffed with INTD trained dispatchers through three operational periods, providing troopers in the area with dispatching services and information flow to and from their ECC.
As the single point of contact for the state’s FirstNet contract, TACN already had a combined 131 FirstNet phones in use with the Tennessee Highway Patrol. Command staff and key personnel were uplifted, a process that provides even higher priority for devices on the network. An additional 180 devices were also requested to assist field responders during the recovery efforts happening in several parts of the state. Procuring, charging, re-programming for public-safety, and field familiarity training proved the biggest obstacles for rapid deployment of these devices. Once in place, they helped field responders remain connected during network congestion and capacity issues.
A decision was made to reconfigure the current radio system design by reassigning five remote transmitter sites to an alternate master site (in this case, from Nashville to Chattanooga) which made geographical sense and would enable them to begin operating system-wide in an accelerated timeframe. This required deploying technicians to each transmitter site for hands-on work, which they completed by 7 p.m. that evening.
REBUILDING A MASTER SITE
The real work of dismantling the master site for relocation would not begin until almost 2 p.m. that afternoon. Structural engineers had to inspect the building and deem it safe for personnel to enter. A van was obtained to ferry out the dismantled equipment through the debris field of the prison grounds where it was reloaded into a box truck for transport to the relocation site, an existing tower near the Nashville dispatch center, which would now become the mid-TN zone master site.
Once the equipment was transported across town, the team off-loaded it at the new site. It was almost 11 p.m. by the time the technicians went back home or to hotels for the night. They would begin installation work at 7 a.m. the morning of the fourth.
The installation site was an un-prepped facility that required electrical wiring for all the components, configuration of all equipment inside the structure as well as the microwave links and inspection of the work. Technicians worked in this small space for almost 12 hours. The only available facility for relief was located at the nearby emergency communications center, so precautions were also taken to isolate the workers from center personnel due to concerns over the pandemic.
Shortly before 7 p.m. on the evening of March 4, technicians confirmed that the master site was functional and wide area trunking was restored to Nashville and middle Tennessee. INTD personnel at the temporary dispatch tower site and the radio technicians and engineers were demobilized. From the moment the technicians began dismantling the master site for relocation to restoration was just under 30 hours, and from the moment the tornado struck until restoration, all told, was 41 hours.
LESSONS LEARNED
Training for continuity of operations must include all personnel, from top to bottom. Knowing what to do is not the same as training how to do it with real exercises. Dispatch personnel were able to quickly transfer operations and ensure their own personal safety during the storm. INTD public safety telecommunicators were prepared for adverse working conditions at the remote tower site. Command staff provided direction and ensured proper documentation and resource allocation, and radio technicians quickly convened and formed a plan for recovering system functionality.
Field personnel need more than just the rudimentary training of finding a talkgroup and pushing the button to talk. Familiarity with radio system functionality should be provided to all system users, especially if or when they encounter issues such as site trunking or a system busy signal. Telecommunicators should receive similar training and be capable of offering simple instructions or troubleshooting/problem identification when issues arise.
FirstNet devices require time to acquire, set up and implement, especially during a time-critical emergency. Even a cache of deployable devices will involve some lag time, as is the case with any piece of equipment that is not already assigned and in use in the field, whether a radio, cell device or other tool. However, the more complex the technology, likely the slower the deployment process.
Establishing communications incident command is one of the critical components to success, ensuring everyone understands the work that is to be done, that efforts are not duplicated, and that personnel do not self-deploy. Without this oversight and mission coordination, individuals can get tunnel vision, tasks may not be properly prioritized, and workflow is not properly routed for assignment.
Christine Burke Massengale, ENP, RPL, Dispatch Coordinator, State of Tennessee Department of Safety and Homeland Security, Tennessee Highway Patrol, Tennessee Advanced Communications Network. She is the Chair of the APCO Editorial Committee.
— This story was written in collaboration with Deputy Director Victor Hernandez, Tennessee Advanced Communication s Network, and Sandy Woody, Lafayette Group, in coordination with the U.S. Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA) Emergency Communications Division.
