All things old are new againby Wayne Eleazer
|
| Two different vehicles, two different major manufacturers, one NASA launched and the other by the Air Force, over 12 years apart and both launched from the same base, but they both failed due to the same type of very simple problem. |
On August 12, 1998 a Titan 4A Centaur lifted off from the Cape, carrying a classified payload. At T+40 seconds the booster pitched over suddenly, the resulting loads causing the vehicle to break up, activating its destruct system. The cause proved to be once again chafed wire, which momentarily killed power to the Titan’s guidance system, resulting in it resetting its clock to T-0 in much the same way that a power failure causes a TV set go to channel 2 when it comes back on. When the guidance system initiated the preprogrammed pitch maneuver that occurs just after liftoff, the loads imparted to the vehicle caused it to break up.
Two different vehicles, two different major manufacturers, one NASA launched and the other by the Air Force, over 12 years apart and both launched from the same base, but they both failed due to the same type of very simple problem. What makes this repeat performance even more tragic is that following the loss of the Delta in 1986 the space launch industry underwent something of an epiphany in regards to booster wiring. Special teams inspected wiring of existing boosters, and generally found much to criticize. Tests were done that discovered all sorts of previously unknown information on the design of wiring and the effects of chafing on types of insulation. Procedures and special tools were developed to prevent damage to vehicle wiring during pre-launch preps. And despite all this, 12 years later it happened again.
This is nothing new. Failure to absorb the lessons learned from launch failures is a recurring nightmare in the space launch industry. While we usually do a terrific job at figuring out what went wrong, transferring that knowledge to other cases has proved to be more problematic (See “Actually, we need more successful failures”, The Space Review, July 17, 2006). Lately, though the lessons learned problem has taken on an added dimension and a new urgency.
After one of the earliest Atlas missile test launches the engineers puzzled over the remains of a failed rocket. It looked as though some of the tubing in the engine section had melted. They could see no reason for this to have happened and ended up consulting the people who conducted the ground run engine hotfires at Rocketdyne. Asking if the test engineers had any problems with engine tubing during the test runs, they were stunned at the answer. “No, no problems at all, at least not since we switched to stainless steel tubing on the test engines. That aluminum stuff you use on the rockets won’t hack it on the test stands. It melts.”
As it turned out, the radiant heating from the rocket engine plume was not just a problem on the test stands. It heated the engine section in the rocket in flight, too. The answer was to provide curtains to seal off the engine section reflecting the heat. And just in case, they switched to stainless steel tubing, too. Other booster programs took note and factored that kind of failure mode into their designs. For the Titan 2 they even encased the electrical cables in stainless steel tubing.
Around fifty years later, the first Falcon 1 booster lifted off, and soon failed. The problem was that a tubing connection had failed, an aluminum tubing connection. The failure was not due to radiant heating, but in response SpaceX switched to stainless steel, just like Convair had a half a century before.
In the late 1950’s an Air Force representative asked a question at a meeting for the Jupiter missile program. “Have you given any thought to the problem of propellant slosh?” The reply from the program manager could not have been more definitive, “Ve don’t haf slosh!” And on the very next launch, the second one for the program, the vehicle tumbled out of control due to propellant slosh.
Almost exactly 51 years later the second Falcon 1 went out of control due to propellant slosh. Vey had slosh, too.
| Basically, no one wrote down all this stuff. It’s not taught anywhere. The knowledge is dying with the people who discovered it. |
In the mid ’60s the Air Force and Army were testing the new Nike Zeus anti-missile system. Atlas missiles would launch re-entry vehicles from Vandenberg AFB to Kwajelein Atoll, where the interceptor missiles would try to knock them out. One day an Atlas was fired from Vandenberg and everything seemed to be going just great. The Atlas kicked the re-entry vehicle off the front end and the interceptor radars at Kwaj eagerly sought their target. Then the Atlas surprised everyone when it passed the warhead and hit the atmosphere first. Propellant captured in the missile’s regeneratively-cooled sustainer engine had boiled out of the nozzle and gave the Atlas some thrust after the engine shut down. It wasn’t much thrust, but it was more than the warhead had, which was none. Hitting the atmosphere ahead of the re-entry vehicle, the Atlas’ stainless steel balloon tank broke up. The resultant cloud of tank skin pieces prevented the radars from acquiring the warhead until it was too late for the Nike Zeus to intercept it. Although unplanned, the importance of this test was not lost on DoD’s senior leadership. Nike Zeus would be too easy to defeat; the program was cancelled.
Around that same time period a Thor LV-2D missile was launched from Johnston Atoll in the Pacific. The payload was a ballistic experimental package, and the Thor repeated the earlier Atlas performance for the same reason—and did it one better, managing to not just catch up to the payload but actually run over it as well, destroying it.
About 40 years later, SpaceX’s third launch of its Falcon 1 space booster lifted off and the vehicle repeated the Atlas and Thor experience in those same tropical Pacific skies. The residual thrust from the Falcon’s new regeneratively-cooled engine slammed the first stage into the vehicle’s second stage shortly after separation, resulting in a mission failure.
On the other hand, SpaceX did learn some lessons that the larger companies had failed to, later lessons that were easier to recall.
Back in the 1970’s Convair engineers gave some thought to possible failure modes on the Atlas booster and concluded that one of the more scary possibilities was a simple hydraulic fluid leak. Excessive loss of hydraulic fluid could starve the vehicle of its flight control ability. The answer was pretty simple, as it turned out. A diaphragm was installed that enabled the booster’s RP-1 fuel to replace the lost fluid in the event of a leak. Someone even asked what would happen if the reverse happened and some hydraulic fluid leaked into the fuel. So Rocketdyne ended up running an engine with 100% hydraulic fluid as the fuel and found that it still worked pretty good (and if that sounds like going to an extreme for doing a test, you need to understand that such “extremes” represented lessons learned in their own right). The mod went into the new production Atlas Centaur vehicles.
Then, 20 years later McDonnell Douglas built the Delta 3 booster, based on the Delta 2, but with larger solid strap-on motors. Those new motors required additional control authority and hydraulic systems were used to swivel the nozzles on the strap-on motors – but the Atlas fuel interface mod had never been incorporated on the Delta 3. On the Delta 3’s first flight the Delta 2-based guidance system had a little trouble handling the new vehicle and used up the solid motors’ hydraulic fluid before the solids burned out. The vehicle spread itself all over the sky.
For the Falcon 1, SpaceX decided to dispense with separate hydraulic fluid entirely and use the vehicle’s fuel for the hydraulic system. No doubt the Delta 3 experience was fresh in their minds.
“Lessons Lost” is an old story, and one that both very large and quite small companies suffer from, but it is one that has taken on added and urgent dimensions in the 21st Century. Basically, no one wrote down all this stuff. It’s not taught anywhere. The knowledge is dying with the people who discovered it.
No one wrote down all this stuff. And even if they did, they did not disseminate it. The traditional way of handling the lessons learned problem was what was known as “corporate memory.” “Graybeards” who had been around enough to have either experienced failures first hand or had learned directly from the people who had done so were expected to raise their hands and say “Not so fast, sonny. We tried that and it didn’t work back in ’57.”
Very early in its space efforts the US Air Force created an entire company, the Aerospace Corp, designed around the corporate memory concept, with a key part being not only long-term specialized expertise but also an independent viewpoint. Aerospace Corp. engineers did not get fired for bringing up unpleasant facts dating from 1957; they were more likely to get fired if they did not.
| Most importantly, what is being lost is not just specific knowledge about what tubing should be made of or how to prevent wires from chafing, but an entire body of knowledge, a philosophical approach to engineering, a unique attitude. |
NASA rejected the idea of such an independent group of experts, deciding the agency could create its own experts and nurture its own corporate memory. Following the loss of the Shuttle Challenger in 1986, the idea was raised again, and rejected again. After the loss of the Shuttle Columbia in 2003 the idea came up again, and was accepted by NASA, or sort of, anyway.
The corporate memory problem is getting worse, as the Falcon flight experience indicates. By being more innovative, we may be gaining advantages in cost and capability, but we are also reaching far back, past the current experience base, and recreating older failures.
The other part of the problem is that corporate memory is weak and getting weaker, even in the experienced organizations. Congressional directives forced manpower cuts on NASA, the Air Force, and even the Aerospace Corp. itself during the 1990’s. Internal initiatives in the organizations made the problem even worse. Corporate mergers had much the same effect on private industry in the same time period.
Most importantly, what is being lost is not just specific knowledge about what tubing should be made of or how to prevent wires from chafing, but an entire body of knowledge, a philosophical approach to engineering, a unique attitude. All those failures and fixes collectively added up to a certain kind of philosophy that was unique to the space business. Astronauts had to have the Right Stuff to get the job done but the engineers that launched them had the Right Way.
Recently I discovered that NASA was hiring people with experience in fields such as nuclear power plants and offshore oil drilling in order to acquire people with the right kind of attitude. This is remarkable, because 40 years ago it would have the space program experts teaching other fields about the Right Way, and not vice versa.
We should have every expectation that the experience loss will get worse. Today, 60% of aerospace workers are over age 50 and almost 26% of them are eligible for retirement this very year. Aerospace is no longer viewed as the glamorous and rewarding career that it once was; few youngsters are choosing it. And even the ones who do come to work in the space launch field don’t have any available “book learning” on the mistakes of the past.
No one wrote down all this stuff. And soon it will be too late to do so.