Industry-Submitted Notification

What Happened

The operator received a call-out on the ESD alarm string. On arrival, the site had already shut down. The first indication of a problem was the smell of burnt plastic. Approximately five seconds later, smoke was visible coming out of the MCC building's exhaust fan. The operator did not enter the building, citing smoke exposure, the potential for electrical arcing and other unknown hazards, and activated the emergency response plan.

The drive had failed internally and was smouldering, melting the insulation on its wiring. All three line fuses feeding it had blown, which removed the energy source. The utility was called for an emergency disconnect at the pole. 911 was called and the local fire department, a mutual-aid industrial response team and company personnel attended.

Once disconnected, the building was ventilated, and an electrician opened the panels, isolated the damaged cell from the horizontal bus bar, and inspected the bus bars and insulators before anything was re-energized. The facility was back in service the same day.

Preliminary indications pointed to an internal failure of the drive itself. The Fire Marshal attended several days later and removed the drive for review with the manufacturer. That review remains open at this time.

The Hazard

A variable frequency drive can fail internally and overheat inside a closed cabinet. There is a great deal of plastic in a drive cabinet, and components can glow red hot and smoulder without ever producing an obvious flame. That smouldering fills the building with dense, toxic smoke, and the building's own exhaust fan can distribute it. In this event, soot was drawn through the MCC and deposited on an otherwise undamaged contactor cabinet two cells away.

The hazards are smoke inhalation, heat, and the possibility of arcing or a phase fault on re-energization if bus bar insulators have been heat-damaged.

Why You Will Not See It Coming

• The failure is internal and out of sight. Nothing is visible until smoke leaves the cabinet. There is no external stage of this failure that could have been caught on a walk-through.
• It does not fail the way drives usually fail. The normal failure is a tripped breaker that is reset on arrival. An electrician involved in the response had seen drives fail during a long career, but never on this magnitude. Expecting the usual failure mode is reasonable, and it is what makes this one unfamiliar.
• Current maintenance does not rule it out. An electrical inspector had been on site the week before, preventative maintenance was up to date and housekeeping was in order. Annual switchgear inspections looked for heat, loose connections and blue discoloration on terminations. All are worth doing, but none can see inside a sealed drive.
• The call-out does not tell you what you are driving to. ESD, smoke, battery fire and 40% LEL shared one alarm string. That was a deliberate design decision because the response was to drive either way, but it meant the operator arrived without knowing which hazard was present.
• What you can see understates what is inside. Smoke visible at the exhaust fan is only what the fan is pulling out. After the smoke cleared, infrared readings behind closed panel doors were still around 125°C. The residue left on the building walls was invisible to the eye and came off black on a rag.
• Detection is the only early warning, and it is a single layer. The MCC smoke detector generated the call-out and shut the plant in. Previous company records showed instances where MCC smoke detection had been found not working during testing or facility ESD checks.

Previous Electrical Heat Events

Six events across five areas, plus two separate findings where the MCC smoke detection was not working, were reported by people in the field. The most recent failure was severe in a way none of the earlier ones were, but electrical gear generating heat is a recurring feature of facilities and something the organization should plan for rather than something any individual is expected to catch.

• Responding crews were held back until the site was de-energized. The fire department was permitted an external walk-around only, “look, don't touch,” and stayed on standby until the utility had disconnected.
• The electrical work was done to a zero-energy state. The main breaker and all remaining breakers were shut off before any cabinet was opened; thermal imaging was performed through closed doors; and the bus bars and insulators were inspected for heat damage before re-energizing. Isolating the failed cell from the horizontal bus bar then allowed the rest of the MCC to be brought back safely.

If You Arrive at an MCC That Is Smoking: How to Stay Out of Harm

• Do not enter a smoke-filled electrical building, and do not open a panel that has not been de-energized and verified. Life over equipment. There is nothing inside an MCC worth a lungful of smoke from melting insulation or an arc flash.
• Get the utility moving early. Blown fuses or a tripped breaker may have removed the source, but you cannot confirm that from outside. The disconnect at the pole is what makes the building safe to work in, and drive capacitors can hold a charge after the supply is gone.
• Treat everything behind the doors as hot. After the smoke cleared, panels were still around 125°C internally. Thermal imaging through a closed door tells you what you are about to open.
• If you are close to the equipment when it fails, leave. Evacuate by the route away from the equipment and do not go back for the breaker. If you choose to open a breaker, stand to the side and use your left hand, but leaving is always an acceptable answer.
• Ventilate before re-entry and expect residue. Smoke residue coats surfaces invisibly and the smell lingers. Clean-up is its own task, with its own PPE.

What We Are Doing

The fixes for this belong to the organization that experienced the incident. Coming out of the After-Action Review, actions include:

• Replacing the activated MCC smoke detector. A detector that has been exposed and tripped raises a fair question about whether its sensitivity remains adequate. The correct practice following an activation is also being confirmed with the manufacturer.
• Adding an electrical scenario to the emergency response drill program. Drills had concentrated on process hazards. This event showed that the electrical side brings different questions, including utility disconnect, de-energization, arc flash and higher-voltage gear in some areas.

The review of the drive with the manufacturer remained open at the time of the original alert. If the findings identify anything that should change how the equipment is inspected, protected or replaced, those findings should be considered as part of the corrective actions.

What You Can Do

• Talk about this at your next tailgate, including with contractors who work in and around electrical buildings.
• If you find smoke or a burnt smell at an MCC, stay out and call it in. That is the expected answer, not a cautious one.
• Report the signs of heat you can see. Blue or discoloured terminations, burnt smells, dust build-up on breaker cooling fans, and plugged or oily VFD heat sinks have all been reported before. Those reports are how the hazard can be found while it is still small.
• Confirm MCC smoke and fire detection during facility ESD checks, and report a detector that does not trip. It is the only early warning available for this type of event.
• Be in your PPE from the start of your day. A call-out gives you no time to make that decision. If the way a day is set up makes that impractical, report the concern.
• Look after yourself during a long response. During this event, the operator went from midday to 5:30 p.m. on a hot day without a drink and only noticed once the response was over. Adrenaline can hide that. If you are running an incident, someone should be looking after hydration and other basic needs.
• Keep reporting, and identify what gets in the way. If a check is impractical, if gear is hard to get at, or if something has been concerning you about a piece of equipment, that information is important.

Consistent with Human and Organizational Performance principles, the focus is on the conditions that made this outcome possible: equipment that fails invisibly, a single layer of detection, and drills that had not covered the electrical side, not on the people who encountered it. The maintenance was current, the inspection had been done, and the response was sound; the drive failed anyway. Error is normal; blame teaches us nothing and costs us the reporting we depend on.