A Tale of Wrong Root Cause Analysis in a Refinery
Photo: Refinery Fire Credit: Mike Clark, Billings Gazette
The follodu cornrowing incident is a description of an actual fire event that occurred at a Refinery Vacuum Residual Bottoms Centrifugal Pump twenty years ago. It’s important to document significant casualty events as the knowledge gained will save Industry lives. The pump was involved in a fire which unfortunately took the life of an operator and burned two others. The author was not directly involved in the incident or its investigation but later learned the details of the event. This article is an attempt to explain how Wrong Failure Analysis can lead to incorrect conclusions and unsafe practices or disasters.
The large Crude oil refinery had an expansion several years prior to the fire in the Diesel Hydrotreater Unit which included many Vessels, Columns and rotating equipment. Among the rotating equipment were two identical Vacuum Residual Column Bottoms centrifugal pumps driven by motors and operating at high temperatures. These are between bearing API pumps with Top nozzles. The pumps have a high vapor pressure hydrocarbon flammable liquid at above 400 DegF and are by API standards equipped with Dual Mechanical Shaft Seals, using a cooled return flush to primary wet seal and dry running backup secondary seals. This is a Tandem sealing arrangement. The 450 HP pump was directly motor driven at 3550 RPM and equipped with a 12-inch spacer coupling.
The Column bottom pump’s two mechanical seals on Drive end and Non-drive end had a high failure rate since commissioning with average lifetime not exceeding 6 Months MTBF. The Refinery engineers worked to resolve the high seal failure rate as this was considered a bad actor pump with MTBF far below refinery acceptable mechanical seal lifetime of minimum 2.0 years and above.
The Root Cause Analysis of M-Seal Failures
The pump had been operating for 2.0 years and various mechanical seal failures were analyzed by the rotating equipment team. The primary seals utilized Silicon Carbide against Carbon face combination. The engineers made the following judgement during their failure analysis:
M-Seal Primary Faces Found Scratched = Solid Particles Led to Face Leakage = Stop Solid Particles from Entering Seal Faces = Prevent Solid Particles From Entering the Pump = Install Extreme Filtration
The “Root Cause” analysis was highly simplistic, and led to their conclusion that to remove the failure modes it was necessary to filter out the solids before entering the pump by using a fine mesh strainer. The M-Seal flush plan already included a centrifugal separator otherwise named as Cyclone separator. Now, if the engineers had a practical design knowledge of Mechanical seals they would have realized that a healthy mechanical seal operating with normal face opening, would never allow any solid particle larger than 1 micron to enter the faces because its film thickness is less than 1 micron. Therefore, to truly protect such a seal, the liquid would have to be filtered down to 1 Micron size solid removal, which is basically impossible looking at the flow rate and refinery service involved which is a bottoms residual crude oil pump.
Anyway, the team decided that extreme filtration will be the answer and realized that 1-micron filtration is not even available, so they concluded that installing a pump suction protection filter or screen with fine mesh of about 100-micron mesh [ 100 Micron = 1/10 MM mesh] will greatly reduce mechanical seal failures. The engineers realized that the mesh will probably get blocked and therefore placed differential pressure transmitters across the suction screen to warn of excessive pressure drop which can collapse the filter. The pump was placed into service and as expected, the conical mesh strainer would get blocked within a short time not exceeding one-week operation. Previously the pump was equipped with a standard conical mesh of 0.20 inches’ size, so this would not get blocked for a longer period of at least 12 months operation.
The Incident of Fire and Death
The operators responsible for the pump had no choice but to clean out the pump inlet suction strainer by removing the pump from service including isolating the suction-discharge piping, draining the high temperature Hydrocarbon, and then opening the suction piping to remove the Conical strainer. To perform drainage of this section of suction piping, no permanent drain piping connection was available, so they decided to install a Flexible fiber-reinforced rubber type material hose to perform drainage. There were two operators, one mechanic and one machinery engineer. After installing the hose, the drain valve was opened however the high temperature hydrocarbon at about 450 DegF reacted strongly with the rubber formulation, melting it and leaking pressurized hydrocarbon sprayed into the air, igniting immediately as a large conflagration since it was at self-ignition temperature upon reaction with Oxygen. Sadly, one operator died from burn injuries, one escaped injury, and both mechanic and engineer were exposed to third degree burns that required weeks of Hospitalization however they were able to escape life threatening damage.
If we utilize the James Reason Swiss Cheese Model of Failure, it’s clear that the alignment of several weaknesses allowed this fire incident to occur. Definitely the fact that a Rubber hose was applied is a major error and one hole in the Swiss cheese model, and there should have been a carbon steel drain pipe installed, but the second cheese hole alignment came from installing an extremely fine mesh suction screen that had to be opened weekly to remove blockages, thus requiring 52 shutdown-cleaning events per year which is completely impractical and unsafe for a hazardous Hydrocarbon liquid at self-ignition temperatures. This is unsafe practice even if carbon steel drain piping was utilized. The Committee decided that a permanent closed-drain piping system was required but did not recognize the major error caused by “Wrong failure Analysis”.
New Root Cause for The Mechanical Seal Failures is Discovered
The Author was by chance assigned to a special two-month safety and reliability mission to oversee rotating equipment at the same refinery facility about one year later. During a normal walk-around inspection one day, the author noticed something unusual about the pump operation and this was something that few would realize its implications as it is hard to notice. The same vacuum residual Bottoms pump had been repaired for a year as it was not significantly damaged by the fire event and was operating with acceptable bearing housing vibration of 0.12 inch/s RMS radially which is fine for such size and service pump. However, the shaft coupling while mostly covered by the coupling guard, showed clear axial displacement or shuttling, roughly about 0.030 inches or 0.75 mm axial play during operation.
Being experienced with at least 8 years’ experience in Mechanical Seals at the time, It’s important to recognize that such excessive axial movement will cause mechanical seal faces to open during operation, especially on higher speed pumps above 1800 RPM. The typical machinery engineer visualizes the mechanical seal springs as shown in drawings, but cannot see the physics of operation.
The assumption is that springs maintain the faces closed at all times but this is not the operating physics at all. Both Misalignment of the seal faces to shaft centerline, or strong axial movement or axial vibration will cause inertial reaction forces that force the faces to open momentarily and any hard solid particles in the fluid will now enter the opening and damage the faces. And this is how the faces actually got damaged, as they would open beyond the normal limit of 1 micron, and allow 5 to 10 micron particles to enter and cause damage. As per recollection, the seal flush was equipped with Cyclone separation but this is not enough to insure reliable face operation as the Cyclones will allow particles up to 10 microns to reach the primary seal cavity.
The high axial shaft movement of pump was reported to the Area Foreman and engineering but upon next repair, the site engineers had not taken precautions to resolve this finding. So, after inquiring with the plant foreman I visited the pump during site repair and noted that the mechanics were assembling the pump thrust bearing [Hydrodynamic sleeve radial bearings and ball thrust] while the mechanical seals were installed and tightened. I stopped the repair process and asked him to loosen the seal sleeves, and now measure the thrust axial float. The finding was about 0.030 inches axial play when it should have been 0.003 inches only.
The mechanics had assumed that the thrust bearing was manufactured with proper clearances and there was then no need for shimming to reduce axial movement. By this we uncovered the largest root cause of the two pumps high seal failure rate: Excessive axial movement or axial shaft vibration of pump and thrust bearing due to highly excessive thrust play from both User error and pump manufacturer error.
It’s clear that this defective pump assembly allowed the primary seal faces to separate during rotation and gradually fail due to solid particle contamination of the faces causing wear and with several months’ operation, seal leakage failures. The pump was then shimmed at thrust bearing per our exact guidelines and once started, the axial shuttling was not to be seen anymore.
Conclusion
Due to accurate Root Cause Analysis, the Mechanical seals on this pump immediately rose in reliability achieving minimum 1.5 years’ operation between failures as a direct result of this change. We also advised that the existing SiC versus Carbon faces combination in this service is wrong and SiC-SiC faces should be fitted giving both a reliability and safety advantage. The site engineers could not comprehend the reasons for this selection and maintained the same faces. The SiC-SiC face combination would have easily doubled that MTBF to 3.0 years due to strength of the faces which resist Carbon faces typical failure modes. Lesson Learned: Engineers can make operating plants safer for workers or introduce dangerous solutions; it’s your choice
Posts
0 Comments
please do not any spam link in the comment box