FCC Panel Testimony
Federal Communications Commission
Independent Panel Reviewing the Impact of Hurricane Katrina on Communication Network

New Orleans Public Safety Communications: What Went Wrong
by Dominic F. Tusa, Tusa Consulting Services.
March 6, 2006

Good afternoon, I am Dominic Tusa, founder and principal consultant for Tusa Consulting Services, Inc. My firm, as a subcontractor for Moses Engineers, Inc. of New Orleans, was responsible for the design, implementation oversight and acceptance testing for three public safety radio networks directly impacted by the force of Hurricane Katrina: City of New Orleans; St. Tammany, Louisiana; and Harrison County, Mississippi

Much has been previously written by my firm and others about the technical issues involving why the City’s radio network was disabled some hours after Katrina’s passage (see Attachment A for details). Today, I would like to offer a perspective of key issues that should be of relevance to this Panel’s ongoing investigation.

By the morning of August 30th, nearly 85% of New Orleans streets were flooded and impassible. The sudden breakdown of public safety communications at all levels: State, adjacent parishes (Jefferson, St. Bernard and Plaquemines) and then ultimately the City itself, allowed local conditions to descend into a state of chaos and lawlessness. It is important for the Panel to understand the conditions, on the ground, from the perspective of those tasked with maintaining and supporting the City of New Orleans Tier-I radio network’s operation during the period in question. Specifically, much focus should be placed on activities and events that occurred, post-Katrina landfall, during the period of August 29th through November 30th.

It is recommended that the Panel request statements from the following persons:

• Major James P Treadaway, New Orleans Police Department
• Captain Stephen Gordon, New Orleans Police Department
• Mr. Peter Caruso, New Orleans Fire Department
• Mr. Thomas Levy, New Orleans Fire Department
• Mr. John Lyons, Armstrong International Airport
• Ms. Brenda Ireland, New Orleans Emergency Medical Services

These people were on-the-ground, working to restore essential public safety communications during the harshest conditions one could ever imagine. As communications professionals, they fully understood the importance of restoring network functionality in the shortest period of time and were frustrated by competing, and seemingly shortsighted, actions within both State and City emergency management agencies. Those actions unfortunately impeded their abilities in gaining the necessary materials, transportation and logistical support to successfully complete that recovery.

I cite two examples for consideration. Very early during the recovery process, Major Treadaway was requested to coordinate development of a list of communication equipment and resources needed to expedite the Tier-I recovery. Later, it was learned that the State had reconfigured that listing and provided substitute 700MHz equipment that was completely incompatible with the City’s existing 800MHz radio network. The net result was boxes of equipment collected in makeshift storage locations, unused and subsequently returned to the State.

Rising water caused the evacuation of the City’s primary and secondary 911 and radio dispatching facilities. The New Orleans Fire Department, however, sustained very minor flooding to its 701 Rosedale Communications Center and could have been resumed dispatch operations in very short order and, with few building modifications, could have been adapted to support consolidated Police, Fire and EMS dispatch operations. Instead, the City’s Office of Homeland Security embarked on a nomadic movement of public safety dispatch operations using tents, hotel space and trailers.

Each of these relocations caused significant issues with the improvisation and replication of radio dispatching where such capabilities, and the ready means for expansion to support other agencies, already existed at the Rosedale Center. It is imperative that the Panel secure the truthful and unbiased accounts of these six key City personnel to gain an accurate, unvarnished assessment of what transpired during that critical 60-day period.

The City has already provided some information concerning its public safety radio restoration activities through its normal chain-of-command structure. However, as is the case when oral accounts are passed through multiple relay points, subtle but extremely important bits of information become lost in the “noise” of consolidation and summary. Today’s investigation cannot allow any small, but potentially significant, detail to go unnoticed. Case in point are the seemingly small mistakes in interpretation of soil study data by the US Army Corps of Engineers that, in the light of investigation, appear to have resulted in the inappropriate design of critical canal floodwalls.

In my opinion, the City’s Tier-I radio network worked as designed. Its many automatic redundancies, hardened antenna systems and licensed microwave backhaul continued to function during and after the storm’s passage. But no radio network (trunked or conventional; simulcast or single-site; circuit-switched or using the latest Voice over Internet Protocol) can function for relatively long periods without stable electrical power.

The design approach used for New Orleans, St. Tammany and Harrison County considered the effects likely from widespread flooding and 140mph hurricane-force winds. This essential work was not completed in a matter of days or weeks. The design of each of these three shared radio networks was the result of comprehensive, needs-based study of user agency’s requirements coupled with sound radio network planning and investigation of local-area conditions.

In each network’s configuration, equipment shelters and standby power systems were located, and where necessary elevated, to circumvent floodwaters and to assure survivability. The City’s Tier-I primary simulcast transmitter site, for example, was elevated 42-stories and its equipment installed within a concrete-hardened room.

For the most part, this aggressive design approach paid off. The actual damage to vulnerable site and antenna infrastructures was remarkably slight. Harrison County’s 800MHz simulcast network, which was exposed to the full force of Hurricane Katrina’s winds and storm surge, sustained damage so minimal that repairs to one microwave antenna were completed within hours of the storm’s passage.

Continuing this point of discussion, the City’s Tier-I network’s “nerve center”, termed the Control Point, was located within the New Orleans Police Department’s Parking Garage, 7th Level. This critical site was equipped with dual standby generator systems, a 3000-gallon diesel fuel supply and a battery plant capable of sustaining operations for over 14-hours. Remote tower sites were equipped with battery backup systems as well as LPG and/or natural gas-fired generators as this fuel type is least subject to contamination. And, experience has shown where natural gas is the least likely fuel source to be disrupted during hurricanes (as witnessed by my 25-year history designing radio communication networks supportive of the oil and petrochemical industry).

Despite these aggressive precautions, the use of standby generator systems for prolonged periods, coupled with the unprecedented difficulty in site access due to widespread flooding, created its own set of problems. In instances where diesel generators were employed, which for the City included the Tier-I Control Point, failures were so numerous it became necessary to have caretakers on-site to correct them as they occurred. And, since this mission-critical site was operated via generator power for 133 days, the cost for maintenance and the risk of sudden shutdown was high, yet the City’s team of technicians, engineers and maintenance providers held the network together.

Much has been stated about the inability of key repair technicians to gain return access to the City. As contract civilian workers, they too were impacted by the City’s mandatory evacuation order. It is imperative that a new, nationwide identification system be developed to allow unimpeded return access for key radio and telecommunications personnel. Without these critical resources, public safety radio and telecommunications network recovery will continue to be delayed, unreliable and functionally-sporadic.

Much, too, has been said that the communication failures within New Orleans were the result of incompatible radio systems. Such statements are a gross simplification of on-site realities. The four-parish region already had multiple levels of radio network connectivity in place to support daily interoperable communication needs and more was scheduled for implementation. The loss of essential public safety communications within Orleans, Jefferson, St. Bernard and Plaquemines parishes had nothing to do with proprietary radio networks, technology or frequency bands. These various radio networks failed due to a combination of four reasons: prolonged loss of reliable electrical power; insufficient site planning in flood-prone, hurricane-vulnerable areas; over reliance on leased infrastructure connectivity (in lieu of dedicated microwave backhaul) and lack of ongoing user-personnel training.

Industry professionals have already touched on how electrical power and site planning are critical components for network survivability. Complex communication networks, involving multiple transmitter/receiver site locations, require interconnectivity so that the various locations function in concert. There are two principal ways in which this location interconnectivity is accomplished: leased telephone facilities or owned microwave.

Without question, leased telephone facilities offer the most expeditious and least costly approach to site interconnectivity. Yet, these facilities are the most prone to failure as wired or buried cable/fiber networks are subject to damage anywhere along their vulnerable routings to remote tower-site locations. By contrast, microwave connectivity offers superior reliability but requires a large capital investment (typically $250,000 per segment/20-mile linkage). Yet, one characteristic of the New Orleans, St. Tammany and Harrison County public safety radio networks that cannot be overlooked is all three used microwave backhaul for their interconnectivity medium. How the use of licensed microwave contributed to the remarkable reliability of these three public safety radio networks is important and must be considered by this Panel in its ongoing investigations.

Finally, urban public safety radio networks are not static in their design. As user needs change, so do the capabilities and features of complex urban radio networks. The majority of these new functions and features are imbedded within a radio network’s infrastructure, with little change to the actual user equipment in the police officer or fireman’s hand. And, what changes do appear in either the user’s talkgroup structure or portable/mobile radio flash code are so transparent and seemingly minor that few department managers see value in ongoing user training. It is the first cost to be cut in any budget scrub session. I believe a basic lack of training contributed to the confusion in adapting to the degraded radio conditions that existed, post Katrina.

The advantage the Amateur Radio Service brings to our community is its vast network of self-funded, self-equipped and self-trained radio communication experts. Often, messages are relayed, amateur to amateur, without the need for sophisticated high-capacity infrastructures. In normal operations the City’s Tier-I radio network processes over 3.5-million radio transmissions each month. Emergency response Amateur radio communications cannot sustain anywhere near that level of radio traffic, but the very small numbers of communications supported are each of vital interest and limited communications is always preferred to zero communications.

The City’s Tier-I network, just like our partnered Amateur Radio enthusiasts, has the capability to support unit-to-unit transmissions absent of radio infrastructure. But, few City radio users understand neither how to best use this feature nor how to group users into ad-hoc frequency nets similar to trunked talkgroups. Public safety radio users, for the most part, are not radio enthusiasts. The radio, for them, is simply a tool having a specific fixed purpose. If that tool’s ability is suddenly diminished, and there is no familiarity with how it could be used differently, then the tool becomes ineffective and useless.

Federal funding must be made available and earmarked for ongoing training in the proper use of radio communications equipment during both normal and emergency situations. Local governments simply have too many other budget needs to provide the sustained funding necessary for meaningful radio user training. Further, the Department of Homeland Security’s (DHS) Grant Program must be expanded to allow improvements and enhancements to today’s existing radio infrastructures. The City of New Orleans requires repairs and replacements to its Tier-I radio network power generators and battery plant equipment, NOW, in order to be prepared to face the 2006 Hurricane Season. Yet, bureaucratic red tape is holding up these vitally needed repairs and exposes the City and its public safety agencies to unacceptable risks.

In closing, I wish to thank the Independent Panel for this opportunity and look forward to answering any questions you may have in the coming weeks.

Attachment A
Public Safety Communications in the Wake of Hurricane Katrina

Introduction

In the early hours of August 29th, 2005 areas of Southeastern Louisiana and the entire costal area of the State of Mississippi was devastated by the effects of Hurricane Katrina. An unprecedented 90,000 square-mile area was impacted by the storm, forcing the mandatory evacuation and displacement of over 1.5-million people.

Residents and public officials initially believed that the City of New Orleans had, once again, been spared the doomsday scenario of a slow-moving Category-4 storm having an approach path just west of the City. Such a trajectory would have, in itself, caused widespread flooding throughout the southeastern region of Orleans, Jefferson, St. Bernard and Plaquemines Parishes. Instead, and as has happened several times before, as the outer bands of the storm made contact with the Louisiana shoreline, a slight jog to the east occurred...sparing New Orleans a direct hit but placing cities such as Slidell, Waveland, Bay St. Louis, Gulfport, Biloxi and Pascagoula directly in harm’s way.

While New Orleans surely was spared the storm’s peak sustained winds and 32-foot tidal surge, it didn’t escape flooding. Several levees and floodwalls did later collapse, causing the City of New Orleans and surrounding areas to flood to a degree never witnessed in modern times.

The combined effects of Hurricane Katrina...wind, tidal surge and floodwater... played havoc with the area’s commercial and public safety communication networks. Cellular telephone systems first were overwhelmed by call volume and then silenced by the lack of electricity and tower site interconnectivity. The public switched telephone system collapsed due to broken aerial lines, broken buried lines (uprooted trees), loss of electrical services and key switching centers. The terrestrial packet-switched Internet system, while “self-healing” to some extent, cannot overcome the complete disruption of an entire region’s broadcast cable and telephone infrastructure. These adaptive, highly reliable networks failed as well.

While the loss of public communication services results in the partial disruption of command and control capabilities for federal, state and local-area public safety and disaster recovery services, such disruptions are anticipated and risk mitigation/contingency planning is the norm. All such contingency plans, however, are heavily weighed toward the concept that some form of local-area public safety communications would be available. That is, while public safety radio systems located in the direct path of a Category 3 or higher storm would likely sustain some damage, their design configuration would be conservative to the point where a fundamental ability to communicate, at some level, would remain.

Several public safety radio systems did, in fact, perform near-flawlessly during Hurricane Katrina’s landfall, yet many others failed catastrophically. Specifically, the 800MHz public safety radio systems operated by the City of New Orleans, St. Tammany Parish, Louisiana and Harrison County, Mississippi survived the full wind and tidal surge effects of Hurricane Katrina. Other public safety radio systems operating throughout the region...St. Bernard, Plaquemines and Jefferson Parishes...failed in the height of the storm. The New Orleans public safety radio network did, however, go silent from approximately 4PM August 29th until approximately 4:30AM September 2nd as a result of debris damage to a key standby electrical power generator.

In the immediate aftermath of Hurricane Katrina, the City of New Orleans descended into a period of panic, fear and lawlessness. Images of helpless residents trapped on rooftops, bridges and within the Superdome and Morial Convention Center kept our nation riveted to the television set. Widespread looting and even bands of snipers had overtaken the City, with the nation and seemingly the New Orleans Police Department, itself, helpless to regain control of a runaway catastrophe.

Many factors gave rise to these deplorable scenes and conditions. Certainly the unprecedented multi-parish flooding....whereby patrol cars, ambulances and fire trucks were replaced by helicopters and small, slow-moving boats...gave rise to the near surreal atmosphere. However, the region’s loss of reliable public safety radio communications had a devastating impact on the New Orleans Police Department’s ability, as well as for those public safety agencies attached to surrounding areas, to maintain law and order in their respective jurisdictions.

This Testimony concerns only the operability and functionality of the City of New Orleans 800MHz public safety radio system. Here, we will provide information on the radio system’s infrastructure configuration, design aspects that were of critical importance in its survivability during hurricane conditions, what components and portions of the radio system failed as a result of the storm (and their relative impact on system functionality) and what steps were taken within the first several days of the storm’s passage to restore functionality and reliable performance

Infrastructure Configuration

Before we begin a discussion of the communications failure, itself, it is helpful to understand how the City’s radio network is configured and the thought processes used in its design.

Prior to 1991, the City’s Police, Fire and Emergency Medical Service agencies operated aged, but independent, radio systems. In fact, most of these radio systems had configurations and components that dated to the mid or late 1960s. But, a unique aspect of the City’s radio network was that it operated within the same UHF radio spectrum for all three agencies, thereby allowing radio-to-radio interoperability between agencies. When implemented, that was a cutting edge idea as most cities had their public safety operations scattered across incompatible frequency segments such as 40-50MHz, 150-170MHz and 450-470MHz.

Where New Orleans enjoyed today’s hottest buzz word, interoperability, in the 1970’s and 80’s, their rationale for having all three agencies operable within the same spectrum was purely need-driven. This City has successfully hosted, for well over a hundred years, a two-week festival event that other cities would consider to be a planned emergency: Mardi-Gras. The inability to effectively communicate across public safety departments would have resulted in resource inefficiency at best, chaos at worst.

By the late 1980’s, the City’s separate UHF radio systems were becoming very expensive to maintain (as coverage and capacity had to be replicated for each agency) and the lack of available UHF radio channels for expansion had created communication bottlenecks and inefficiencies.

Once the Federal Communications Commission released new 800MHz spectrum earmarked for public safety operations, strong consideration was given toward a new, shared radio network infrastructure. It would depart from the tradition held by many similar or larger-sized cities where instead of each agency having an autonomous radio infrastructure, a single shared architecture, using the latest trunked radio technology would be constructed. By pooling resources, the City could thereby deploy a single radio network having superior coverage, inherent radio-to-radio interoperability, enhanced site/interconnectivity reliability and superior technical performance.

Design studies for the 800MHz radio network commenced in early 1991. The design team was well aware of the City’s vulnerability to hurricane-induced flooding. Mindful of history, its goal was to design a radio infrastructure having multiple levels of coverage overlap, equipment redundancy, independence from leased telephone connectivity and isolation from the commercial electrical services. While the coverage and radio performance requirements would demand location of some infrastructure sites in flood-prone areas, steps would be taken to harden those facilities to withstand then-known conditions.

In 1992, computer models for storm surge predictions had a wide margin for error. Therefore, the design team relied principally on historical data from the two most recent storms that had devastated the New Orleans/Mississippi Gulf Coast: Hurricane Betsy (1965) and Hurricane Camille (1969). Using storm surge data obtained from the Stennis Space Center and other sources, minimum platform heights were established for equipment located at risk settings, thereby minimizing the potential for flood-related damage. Further, the design required that all equipment platforms be constructed of poured concrete and that equipment shelters, generators, fuel tanks and other related items be mechanically strapped to these platforms.

Equipment buildings, tower equipment and antenna systems were designed to withstand 140 mph sustained winds. Microwave antennas were equipped with radomes, to lessen wind loading, and were installed using ruggedized stiff-arm braces to prevent undue flexing in storm conditions. Microwave antenna mounts and towers were designed to resist undue twist thereby preventing path decoupling and subsequent service interruption during high wind periods.

A total of five transmitter and/or receiver sites and 45 RF channels were included in the design, to meet the City’s stringent 97% portable in-vehicle coverage requirement. Functionally, these 45 RF channels are configured into four separate communication systems, integrated within a single network topology.

Working from west to east, a nine-channel 800MHz trunked site is located at the Armstrong International Airport. This site supports local airport emergency response and operations departments and provides radio coverage along Interstate-10 (Jefferson Parish) into the City of New Orleans.

The core of the City’s radio network is an Ericsson (now M/A-COM) analog/digital 24-channel two-site simulcast system which supports the majority of Police, Fire and EMS voice/data radio traffic. Its dominant transmitter/receiver site is at the Energy Centre, a skyscraper located in the Central Business District, within blocks of City Hall. This site was selected for its excellent, wide-area coverage potential, the ability to place the radio equipment within the structure’s hardened interior core and that the standby electric power generator could be installed on the rooftop, thereby protecting it from flooding and low-level airborne debris.

A second simulcast transmit/receive site is located in New Orleans East and shares a tower facility owned by Cox Communications. The equipment contained here is identical to that of the Energy Centre site except that the standby generator is supported by both natural gas and liquefied petroleum gas (LPG) sources. Permission to install the secondary LPG fuel source for the Energy Centre site was denied by the original building owners.

Finally, there is a small 5-channel trunked site located at an FAA-owned termed Irish Bayou, which provides supportive in-building coverage to a set of camps and houses near the Rigolets....the small, but critical water inlet into Lake Borne/Pontchartrain.

These three independent trunked radio systems are operated, in concert, to form a fully integrated, seamless communications network. The facility that provides this integration is termed the Control Point.

The Control Point for the simulcast/multisite network is housed in a hardened facility within the New Orleans Police Headquarters Complex. It is here where simulcast timing, receiver voter-selectors, a 24-channel receiver site, console/network controllers, telephone interconnect, mobile date interfaces and other related equipment is housed.

Each of these three trunked systems: Airport, Simulcast and Irish Bayou were designed to operate either as independent systems or as part of a citywide network, depending upon the mechanics of an individual user’s talkgroup structure. For example, Airport Operations personnel normally have talkgroup assignments that support airport facility use. Airport supervisors, however, may have a need to travel off-campus and into the coverage area of the expansive simulcast system. In that instance, supervisory radios may have certain talkgroups mapped to both systems, thereby providing wide area coverage.

The Simulcast and Irish Bayou infrastructure tower sites, as well as the City’s primary public safety dispatch facilities, are electronically interconnected using licensed microwave systems, independent of wired or fiber connectivity. Since the majority of traffic at the Airport is contained within that facility, the interconnection to the network controller (at Police Headquarters) was made via a low-cost leased T-1 circuit.

There are three principal advantages for using licensed microwave facilities in lieu of less costly leased telephone connectivity. First, microwave systems can be designed to achieve normal up-time reliability at least two orders of magnitude higher than that of commercial services. Secondly, these systems, from a maintenance and traffic loading standpoint, are under the City’s direct control. Finally, public safety microwave systems can be deployed using highly conservative (more expensive) design criteria than that used by public carriers since their designs are not constrained by profit-versus-cost commercial targets.

Each of the City’s public safety radio infrastructure sites, with the exception of the Airport, is equipped with high-capacity battery backup systems. These were sized with differing run times in accordance with site importance and loading. For example, the network’s Control Point battery supply can support full operations for, minimally, 14-hours. Each of the simulcast transmitter sites can support all 24-channels, at full output, for 4-hours. Finally, the microwave network was designed to operate for, minimally, 48-hours using its independent battery supply. Large battery backup systems were not provided at the Airport due to the redundant standby generator configuration that supports the facility.

Each of the remaining infrastructure sites includes at least one standby power generator. These, with the exception of the Airport and NOPD Control Point, are natural-gas fed. Airport and NOPD Control Point generators are diesel-fed. Each gas generator, except for the one located at the Energy Centre, has a standby 500-gallon LPG fuel tank to support operations should the natural gas supply be interrupted. Under normal circumstances, each LPG tank should sustain generator operations for approximately 14-days.

A fourth, yet highly critical, radio network component is the 800MHz Mutual Aid System. The FCC, in its creation of the public safety 800MHz spectrum, reserved five radio channels for mutual aid operations. These five channels are replicated throughout the US, thereby providing a means for emergency response support from agencies having “home” 800MHz radio networks. Since these channels, by regulation, must be operated in the conventional analog FM mode, their use is independent of any vendor-proprietary communications protocol. That being the case, radios sourced from any manufacturer of 800MHz trunked radio equipment can fully interoperate using these five conventional channels.

In the case of New Orleans and its surrounding metropolitan area, these five Mutual Aid channels are physically displaced throughout the region. The purpose of this displacement was to ensure that some mutual aid functionality would be retained should one or more tower sites be disabled or temporarily unavailable for any reason.

The FCC’s regulated purpose of these mutual aid channels was never to sustain the normal operations of a region whose public safety user radio count is in excess of 6,000 radios. The concept of a nationwide mutual aid channel plan was to support only the temporary operations of outside-area responders. Unfortunately the collapse of the region’s many primary trunked/conventional radio systems forced the use of this highly limited five-channel resource in a manner that far exceeded its capabilities.

Communications Readiness, Prior to Katrina

The City of New Orleans procured its 800MHz radio communications network through a competitive Request for Proposal process. At the time of purchase (1992) only two vendors furnished, installed and maintained public safety simulcast/multisite trunked radio networks of this level of sophistication: Motorola, Inc and Ericsson, Inc. (now M/A-COM, Inc). The network was fully completed in late 1995 and included a comprehensive one-year parts, preventative maintenance and emergency-maintenance warranty.

Once the original network warranty expired in 1996, the City executed Ericsson’s after-warranty service option (essentially a continuation of the original network warranty). This level of factory maintenance has been supported continuously and is in effect, today.

In the course of routine maintenance, all network alarms are monitored by M/A-COM’s locally–based service facility. Each infrastructure site is visited once each week. The entire radio infrastructure received a comprehensive test and alignment every six months, whose time of completion is at least four weeks before Mardi-Gras (date varies each year) and June 30th which is typically prior to the region’s peak hurricane season.

Since the radio network was installed in 1995, it has undergone a series of infrastructure enhancements. In late 1999, all dispatch console devices were upgraded to comply with Y2000 concerns. A mobile-data overlay was completed whereby the Police Department conducts license and warrant checks as well as field report writing. Antenna systems were fully replaced in 2003. The radio system infrastructure was upgraded to support the latest IMBE digital voice technology and new, light-weight portable radios were fielded to the Police Department in 2004. Also in 2004, the New Orleans Fire Department began replacement of its original portable radio devices with those capable of full water immersion. Infrastructure battery backup systems were replaced in 2005.

Other enhancements had been considered, to bolster the survivability of the City’s trunked radio network....hardening, if you will. These enhancements included a fully-redundant, building-located simulcast site to replicate the wide-area coverage of the prime Energy Centre location. Also considered was the development of a fully-transportable single-site dual-technology (M/A-COM and Motorola) trunked radio package. Upgrades and enhancements to the City’s original UHF radio network, to allow its seamless integration within the public safety 800MHz simulcast/multisite network were also considered. Each was ultimately rejected by City administrators as being too costly to implement, outside the definition of cost recovery for Homeland Security funding grants or simply ignored. In any case, these enhancements had been studied and deemed technically viable but were impossible to construct due to funding constraints.

While M/A-COM is responsible for the City’s 800MHz radio infrastructure maintenance, inclusive of microwave and dispatch radio subsystems, it was not responsible for maintenance of HVAC and site electrical systems. These were the responsibility of the City and maintenance of these facilities was often coordinated by the City’s Property Management Department.

In May 2005, however, testing of the Control Point tower (located atop the Police Department’s Parking Garage) revealed corrosion had occurred within one of the tower’s legs. The City responded by initiating an emergency procurement of a replacement tower for the Control Point site. This new tower was being custom-manufactured in August with installation commencement scheduled for late-September. Consequently the weakened tower was subjected to Katrina’s hurricane-force winds. It is interesting to note that, although weakened by corrosion and not considered likely to survive a Category-3 storm by neither the City’s Radio Consultant nor M/A-COM, the old tower survived the same hurricane force winds that toppled others in the area.

The City of New Orleans has been diligent in the maintenance and continued modernization of its shared public radio communications network, since its well-documented acceptance test completion in late 1995. Prior to the City’s mandatory evacuation, radio technicians had deployed critical spare parts at all infrastructure site locations and a final set of functionality tests were completed. At the time of Hurricane Katrina’s landfall, the City’s 800MHz radio network was fully operational and in a heightened state of readiness.

Communications Integrity, Post Katrina Landfall

The City’s August 28th mandatory evacuation order applied to all area residents and visitors, including M/A-COM’s radio service engineers and field-service technicians. Correspondingly, the New Orleans Police Department provided special documents to M/A-COM service personnel authorizing their unimpeded return to the City once hurricane conditions had subsided.

On the morning of August 29th, the City’s Radio Consultant, Dominic F. Tusa, had made contact with the New Orleans Fire Communications Dispatch Center at approximately 10AM. Tusa evacuated to Jackson, Mississippi and later to Nashville, Tennessee. He was surprised to have his telephone call to the affected area answered on just the second ring. Communications Supervisor, Pete Caruso, explained that all was well, the City’s 800MHz radio network was fully functional and that radio calls were being placed to units throughout the City. Next, Tusa contacted Major James P. Treadaway of the New Orleans Police Department and received essentially the same report...that the radio network had apparently survived the peak winds and was functional. Treadaway also commented that while the Control Point tower was intact, one microwave antenna, used to link the distant Cox and Irish Bayou sites had blown off-course. The prime simulcast transmitter site, located atop the Energy Centre office building, was fully functional.

From a design standpoint and considering the ferocity of the storm, the network had initially functioned as intended. Although the Cox simulcast site in the extreme New Orleans East area was operating in a stand-alone mode as a result of the misaligned microwave antenna, public safety communications was maintained and field users could perceive no change in network functionality or call processing ability. What was unknown, at the time, was that the Energy Centre generator had been damaged by wind-blown debris, an odd failure due to its location in a seemingly secure 42nd floor rooftop location. The 800MHz trunked radio network was continuing to function, due to the security and large capacity of its battery backup power system.

Tusa again called Caruso and Treadaway, just after noon on the 29th, to both confirm that the radio network was still functional and to report that M/A-COM’s Hurricane Response Center (Lynchburg, Virginia) had been notified of the network’s initial condition...that the system was damaged but operational...and that repair technicians would be returning to the City to commence network restoration. It was then that Tusa learned the disturbing news that neighboring Jefferson Parish’s radio system had, during the storm’s peak, sustained a critical self-supporting tower collapse and the State Police site in nearby Bridge City had flooded, rendering both systems off the air and disrupting vitally needed communications. And, Caruso mentioned that he had just heard over the radio where a small break had occurred in floodwall.

This was the last direct communications Tusa had with either Treadaway or Caruso as cellular and telephone services within the City were totally disrupted. Sporadic communications was restored on August 31st to NOPD’s Major Treadaway and Captain Steve Gordon via text messaging and manual voice relay, using M/A-COM’s Hurricane Response Center in Lynchburg, Virginia as a message clearing point.

Late on the afternoon of August 29th (approximately 5PM), Tusa had viewed a televised news interview of NOPD Captain Marlon Defillo whereby Defillo described that all public safety radio communications in the area were being forced onto a small number of mutual aid channels. This signaled that the Energy Centre site had suddenly, and unexpectedly, shut down. Captain Defillo was obviously shaken by this event as the City’s radio communication network had never been fully silenced in its ten-years of operation. Tusa immediately contacted M/A-COM’s Hurricane Response Center to immediately expedite the return of field service technicians and engineers to the City.

M/A-COM’s restoration efforts were focused on a number of different activities in order to restore critical communication capabilities as soon as possible. Many tasks were being handled concurrently and personnel assets were deployed on an as-needed basis – often changing hour-to-hour as priorities were re-evaluated.

Initial entry into New Orleans was delayed (M/A-COM technicians attempted entry on the Tuesday morning following the storm) when service technicians were turned back at multiple entry points by security personnel and were not afforded the opportunity to show letters of authorization provided by the New Orleans Police Department prior to the storm.

Hours melted into days. Tusa had sent email messages to the State Police and the Louisiana Office of Homeland Security imploring them to allow passage of M/A-COM technicians, however, no reply response was received. It was Wednesday morning, August 31st, during a conference call with M/A-COM’s field service and engineering resources, that frustration levels on all sides hit the boiling point. The repeated and unexplainable delays in gaining entry to the City were maddening as every minute without these vital communication services was needlessly risking the safety of City personnel and interfering with public safety’s command and control structure.

The City’s Radio Consultant finally exclaimed: “If we can’t get our guys in, find someone who has a radio/mechanical background that might already be in!”

Somehow, M/A-COM found that someone: Goff Communications. Goff had technicians already in New Orleans providing site restoration services for cellular companies such as Nextel. (The entry into the City by M/A-COM’s staff personnel was not possible until Friday, September 2nd, four critical days after the storm’s passage.)

On August 31st, The New Orleans Police Department (Captain Steve Gordon), City Consultant Tusa and M/A-COM’s Field Service and Engineering staff, via telephone conference calls, developed a priority task list. Restoration efforts, initially through Goff Communications and later with M/A-COM personnel, commenced.

Initial efforts were to bring the Energy Centre prime simulcast site back on the air. This entailed having technicians stage equipment, climb dark stairways to the 34th floor equipment room and the 42nd floor (roof) of the Energy Centre site. At this point in time, the streets accessible to the Energy Centre were flooded. The entire Central Business District was without electrical power. Entry into the building was difficult and extremely dangerous.

On Thursday September 1st all equipment, boats and personnel were in place. The building finally entered. It was subsequently discovered that the site’s backup power generator had sustained damage to its radiator from flying debris, causing loss of coolant and subsequent over-temperature shutdown. All other radio infrastructure equipment, including the site’s rooftop-mounted antenna systems, had survived without any apparent damage.

Temporary repairs were made to the radiator, the generator brought online. Once standby electrical power was restored and considered to be stable, the infrastructure equipment initiated a final self-restart and placed itself back on-line at approximately 4:30AM the morning of September 2nd. At this point, and to our knowledge, the only public safety trunked radio communication network operable within the Orleans, Jefferson, St. Bernard and Plaquemines areas then was the City of New Orleans 800MHz radio network.

Once communication functionality at the Energy Centre site was restored, a detailed damage assessment of infrastructure site equipment commenced.

In sequence, the following was determined:

Airport site Operational – on air except for a short transition period between loss of commercial electric power and startup of airport generators. Dispatch from radio console equipment and multi-site linkage to the prime simulcast system was impossible due to loss of leased Telco T-1.

Cox simulcast site had not flooded – the site’s generator LPG fuel supply was exhausted. Natural gas service was disrupted. When temporary diesel generator power was provided by FEMA, the site was powered up and an equipment damage assessment begun. Minor damage to one microwave antenna coaxial cable (Cox-to-Irish Bayou path) was observed. Simulcast and microwave equipment fully functional.

NOPD Control Point Operational – None of the radio infrastructure equipment was damaged. The site’s primary generator would not start for several reasons, all fuel-related. Diesel is fed from a 3,000-gallon tank. While the generator is equipped with day-tank to allow initial startup, it was apparently run dry. Fuel could not be pumped from the main 3000-gallon tank to the day tank without electrical power.

A day-long effort began to provide initial day-tank fuel load to be able to utilize the 3,000-gallon tank. When power was finally established, M/A-COM personnel began equipment damage assessment.

The only radio infrastructure equipment damage noted was the Cox-site microwave antenna alignment, as first reported by Major Treadaway. Since the NOPD Headquarters building was completely flooded in the electrical switchgear and first-floor area, it was determined that Control Point power would be supplied by generators for an unknown time period. A search began to identify and install a second generator as backup.

M/A-COM determined that all Telco circuits to the Control Point were out and plans were developed for alternate connectivity methods to restore linkage to the Airport site.

Floodwater at this location made access particularly difficult. When water finally receded to service vehicle-traversable levels this site was manned 24/7 in order to assure functionality and network integrity.

Irish Bayou site – unfortunately discovered that Hurricane Betsy’s and Camille’s flood profile did not accurately match Katrina’s. This raised site’s shelter had flooded to approximately 18 inches, ruining the backup generator, shelter air conditioner, two repeaters stations, and power supplies for the multiplexer equipment and a UPS for the Site Controller. The site’s generator propane tank was ripped away by the storm’s tidal surge and the receive antenna was bent off-axis.

NOPD Dispatch Center lacks Electrical Power – no water damage found to radio dispatch equipment. All dispatch consoles were removed to M/A-COM’s New Orleans Service Center for recertification. Restoration of building electrical service is underway.

NOFD/NOHD Dispatch Center – partial flooding occurred on 1st floor. Damage to the New Orleans Health Department’s radio consoles, telephones and computer-aided dispatch systems was observed. The radio and computer-aided dispatch UPS equipment was likewise water-damaged.

Other issues addressed concurrently with the infrastructure restoration involved the procurement and programming of user equipment shipped to replace water-damaged or lost user radio equipment. This portion of work required a major logistical and planning effort as equipment had to be shipped to either Jackson, Mississippi or Baton Rouge by commercial carriers, then brought into the City by M/A-COM and subsequently delivered to transient distribution points.

Temporary emergency dispatching centers were installed and moved on several occasions as City requirements changed. As secondary backup generators were procured, Automatic Transfer Switches were ordered and installed, thereby providing enhanced levels of reliability.

The City’s need to establish a Consolidated Dispatch Center at the Hyatt Hotel, to replicate functionality lost at flooded or access-limited NOPD and Fire/EMS dispatch sites required engineering labor to investigate microwave backhaul, path analysis, microwave and multiplexer equipment procurement and labor to install both this backhaul, dispatch console and backup radio equipment.

To date, both the NOPD Control Point and Cox Cable sites remain on generator power. Commercial power has been restored to the Airport and Energy Centre prime simulcast site. The use of multiple generators at these infrastructure sites, coupled with massive battery backup systems, has maintained network functionality. Plans are well underway to restore commercial power to the NOPD Control Point and are expected to be completed within the next two weeks. The Cox tower site will remain on generator power for at least the next 30 days.

Observations

The coastal areas of Southeastern Louisiana experienced the worst hurricane impact since Hurricane Betsy (1965) and Hurricane Camille (1969). All public safety radio systems operable within this area sustained damage ranging from minor (St. Tammany Parish Sheriff’s Office) to catastrophic. Within the New Orleans vicinity, the only public safety radio system operable in Katrina’s immediate aftermath was the City’s 800MHz trunked radio network. Unfortunately, it later automatically shut down as a result of a power generator failure and communications within the area immediately descended into chaos. 800MHz radio service was in the process of being restored the night of Thursday, September 1st . Work continued through the night and service was fully restored in the early morning hours of September 2nd, three and a half days after the initial shut down occurred.

No radio communications network can be expected to survive a storm of Katrina’s intensity without sustaining some physical damage. In the case of New Orleans, the damage to the radio infrastructure...radio transmitters, receivers, antenna systems, towers, etc. was remarkably slight. Protective fallback modes and subsystems operated as designed.

Yet, no matter how robust the design and well planned the backup systems, it is imperative that technical support be on-hand, in a timely fashion. The inability of technical resources to gain unimpeded City access resulted in needless chaos and angst. The damaged Energy Centre power generator was easily repaired in a few hours, yet it required three precious days to locate and assemble a useable response team and a fourth day to insert the trained M/A-COM service staff.

Throughout this whole event, the single-most labor-intensive activity has been the fueling and maintenance of standby power generator systems. The transportation of diesel fuel was remarkably difficult both during flooded conditions and even after the flood waters had subsided. Diesel generators, of various manufacture, proved to be woefully unreliable, at all levels. Our experience through this suggests that radio systems should incorporate redundant standby generators at all sites. Further, these should be powered by both LPG and natural gas, where available, thereby providing dual fuel sources. In addition, standby battery backup systems should be provided for all public safety radio networks. The configurations used by New Orleans, St. Tammany Parish and Harrison County, Mississippi maintained network operations for as long as 14-hours, providing ample time to complete emergency generator maintenance and repairs....assuming service technicians are allowed the necessary, unimpeded access.

Much has been said that the communications problems experienced within the New Orleans area was the result of incompatible radio networks. That observation is wrong and misdirected.

The wide spread functionality and reliability problems experienced within the New Orleans area and adjacent Parishes were the result of inadequate design, poor planning and insufficient maintenance.

Interoperability is the term generally used to describe how users, operable on various radio networks, can somehow have the ability to communicate in a time of need. Yet, for interoperability to occur, at any level, one must first have operable radio networks. Shared radio networks, such as those operated by the City of New Orleans and Harrison County, Mississippi have inherent interoperability as their various user agencies occupy the same, common radio infrastructure and use identical-technology radios. (In direct contrast, Police and Fire operations in the City of New York, in response to 9/11, were hampered as those departments operated separate radio systems on incompatible VHF and UHF frequency bands) Collapsed radio towers, broken antennas, flooded equipment facilities, dead backup generators and denied repair-service access have nothing to do with Project-25 technology or the frequency band used.

In the aftermath of Hurricane Katrina, one casually observes far too much emphasis being placed on new frequency bands, modes and technologies. For example, some are calling on the urgent need for deployment of Project 25/700MHz radios within the New Orleans area. The idea being that the deployment of these new radios will somehow provide everyone...FEMA, State, Municipal, US Army, National Guard and others...the ability to communicate via a single radio. Yet, these military and federal agencies operate in radio spectrum assignments that are far removed and incompatible with public safety’s 700/800MHz spectrum and the problem of unit-to-unit interoperability continues.

A critical component for the successful deployment of public safety communications system is careful planning. The three systems that survived Hurricane Katrina...St. Tammany, New Orleans and Harrison County...all required 12 to 18 months of study before design specifications were completed. And, each required two-years to be fully implemented and functionally tested. By contrast, the State of Louisiana had, on September 5th, contracted Motorola, Inc. to design, furnish and install a three-site 700MHz trunked radio system for the New Orleans area in a scant 25 days.

The fact that no 700MHz public safety radio systems exist anywhere in the nearby States of Texas, Arkansas, Louisiana, Mississippi or Florida and that no federal agency is currently operable on 700MHz immediately questions the ability of such a system to provide a measure of effective interoperable communications with anyone. But, is it simply reasonable to expect that a proper and thorough consideration of user needs, coverage requirements and network survivability could be accomplished in only twenty-five days?

Many lessons can be learned from Hurricane Katrina. Clearly, more attention should be placed on those radio infrastructures that survived the event and sustained operations...to learn what ideas worked well and what areas need improvement. The blind pursuit of new technologies, devoid of network survivability planning, can only lead to renewed failures. This 90,000 square-mile disaster has no parallel in our nation’s history. Let’s take the necessary time to study all aspects of these failures, where fact-based decisions can be made that positively impact the operability and survivability of both existing and future public safety radio solutions.