An image of a portion of the Zvezda module of the ISS, with the thruster units circled. (credit: NASA/J. Oberg)

The real significance of the ISS thruster test failure

by James Oberg
Monday, May 8, 2006

Last month’s failure of a test of a pair of rocket engines on the International Space Station has taught a whole series of unexpected lessons and has answered questions that the station’s operators hadn’t even intended to ask. Yet since nothing actually happened, a senior NASA spokesman said it was a “non-story”. But that’s nowhere near the truth.

The incident and repercussions of it further underscore that operating a space facility as complex and poorly documented as the ISS is an irremediably non-deterministic process. That is, anything can happen, at any time—and blindside everybody involved.

In this case, a significant station operation appears to have been planned without adequate understanding of how changes to the station’s external configuration might impact the process. When something went wrong, the station was out of range of Russia’s inadequate tracking network, and garbled reports circulated within ground control sites, the news media, and even prestige trade press publications. Then, two weeks later, something else went wrong with the station’s maneuver capability, and this made the original test much more critical, and in much more urgent need of repetition.

The original plan developed at the Moscow Mission Control Center (the “TsUP” in the Russian acronym) was to test-fire the engines “just to see if they would work” after being idle for almost six years. The pair of engines are installed at the back end of the Service Module, named Zvezda (or “Star”), and were used to push the 20-ton habitation module from its original low parking orbit up to an altitude where the rest of the embryonic station could fly over and dock to it. That occurred in July 2000, and opened the way for the arrival of the first long-term crew. The station has been occupied continuously since then.

But the engines, each slightly smaller than the reaction control system thrusters that control the orientation of the US space shuttle, were no longer needed after the docking. When the orbit of the complex had to be changed (usually, but not always, by raising it), thrust was provided by a visiting docked vehicle, either a Russian Soyuz or Progress, or a US shuttle.

However, sometimes a docked vehicle may not be in the proper position, or may not have enough propellant, when an orbital change is urgent. The most common need for such a quick move (several times per year, on average) is when computers on Earth are processing radar tracking data and realize that another space object may soon pass uncomfortably close to (that is, to within unavoidable uncertainties, threaten to collide with) the station. A small shove from any available rocket engine, if performed long enough in advance, changes the path of the station enough to miss the impending orbital intersection.

But then, on May 5, the engine test failure suddenly became much more serious. In an internal-use-only “on-orbit status report”, a NASA headquarters official described a new malfunction that directly threatens that debris avoidance capability. “Both Progress ships are currently exhibiting problems with… the electronic interface for [Service Module] computer control of Progress propulsion,” the report notes on the bottom of the last page. A “special commission” is investigating “the resulting inability of configuring Progress thrusters”.

The engines can still be turned on by direct radio commands from Russian tracking sites, the report adds, but it does not address the issue that, due to the absence of any overseas or space-based relay stations, the station is completely out of range for half of every day, and then in range only briefly every hour and a half during the “good” half.

The odds of a debris avoidance maneuver being needed at all, and being needed during the out-of-range periods, is low. Yet the completely unexpected concatenation of problems that directly relate to a major flight safety theme—collision with another space object—is most definitely not a “non-story”.

The original burn

The original test burn had been scheduled to occur at 19:49 GMT on Tuesday, April 18 (the crew’s day is set to GMT, so this was in the middle of their evening mealtime), as the station was passing southwest to northeast across Iran, in range of Russian tracking sites in central Asia. Since the Soyuz TMA-7 had undocked from the aft port ten days earlier and a new supply ship launch was still five days away, the aft end of the station was clear—a fairly rare configuration.

The burn was to last ten seconds and increase the station’s speed by about 0.35 meters/sec. That impulse would, in turn, throw the opposite end of the orbit about 1,400 meters higher, raising the average altitude by 700 meters.

I had tuned in to the “streaming audio” from a news media hosting site, just to hear what might be heard. I got an earful. “We had an emergency message” was the first, brief call-down, in Russian and translated by an interpreter in Moscow. A few more snippets of downlink indicated that engine number two had failed to ignite but they did not know how long—if at all—engine number one may have fired.

For the next twenty minutes, there was only silence on the downlink. Moscow radioed up a few queries, “ISS, Moscow”—the space age equivalent of “come in, please”—but the crew did not respond.

Finally, Pavel Vinogradov came back on the air, just as the station was passing out over the Pacific and out of range of Russian ground sites. Voice coverage was continuous due to piggybacking on NASA’s relay satellites, but in Moscow they had no telemetry from the malfunctioning Russian hardware. Instead, the Russian capcom asked Vinogradov to read him the error messages that had appeared on his laptop, the way most ISS systems are commanded and monitored.

Vinogradov laughed—there just were too many messages. He explained how the problem had initially manifested itself. “The worse thing was it happened when I activated the TV camera in the [service module midsection],” that was watching a station solar array to see how it flexed back and forth during the thrusting. “I thought, wow, look at what happened when I pushed this little button.”

It had all been a coincidence: just a few seconds before the scheduled ignition, the autopilot detected an error and aborted ignition of both engines. The messages told of the detection of “lack of readiness”, then of the burn cancellation, then of the absence of the orbital correction. Finally there were several messages announcing that pressurization and propellant valves were being closed, back into their passive posture. In Moscow, this was the only information they had, at first, to troubleshoot the problem.

Vinogradov was thanked, and told to relax and sleep well. They would talk more in the morning. “Thank you,” he radioed down, “and best wishes”.

In the following hours, two initial misinterpretations began circulating in Moscow and Houston. First was the report that the burn was aborted because “a propellant isolation valve did not open” (in the words of a NASA spokesman). That was quickly retracted, but another misinterpretation gained traction—that the crew had seen something was wrong and had manually commanded the burn be aborted (that would be the way Aviation Week reported it, based on reading a news wire story).

The true cause becomes clear

That misinterpretation can be traced directly to a spokesman for Moscow TsUP, Valeriy Lyndin, when he was also giving the correct technical explanation for the cause of the abort. “Nozzle covers have to open before the engines are fired,” he told newsmen the following morning, “but telemetry showed that one of the covers did not open fully so we decided to delay the orbit firing.” He further explained: “It was precisely the cosmonauts that spotted the glitch in preparing for the maneuver.”

But there was no tell-tale telemetry reading at the TsUP: they had no data since the ISS was out of range. The reading was detected onboard the ISS and the autopilot, as it had been instructed, stopped at this point and began “backing out” of the burn configuration. Contrary to Lyndin’s description of people making a no-go decision, all the people involved were left in ignorant confusion while the well-programmed autopilot did its job perfectly and then presented the humans with a HAL-like fait accompli.

A week after the abort, the Russian space team had figured out why the cover hadn’t opened fully: it had bumped into something. On April 27, the TsUP released a statement attributing the fault—the door had been 15 degrees short of fully opening—to physical interference from an antenna. The hardware had been installed by spacewalking crewmen for use in guiding the European Automated Transfer Vehicle (ATV) to its planned docking at the back end of the Service Module.

That spacewalk, by Gennadiy Padalka and Mike Fincke, had occurred on September 3, 2004. Computer animation of the work at the SM’s back end, released by NASA prior to the EVA, showed one of the crewmen working very close to one of the engine covers.

The proximity of the additional antenna to the engine cover had never been invisible. Two years earlier, while Russian spacewalk experts prepared a training session in their “hydro-laboratory” at Star City’s cosmonaut training center, the test equipment clearly showed very small, if any, clearance. An American engineer observing the test recalls remarking about the dubious clearance, but he was assured by his Russian colleague that this wouldn’t be a problem. “We’re not going to use this engine again anyway”, is how he recalled the Russian waving off the question.

Inadequate ground preparation on the part of the TsUP had been the cause of another more disastrous interference error, according to Guy Pignolet, a French space official working on a joint project. In February 1999, Pignolet observed a Mir experiment from the Russian control center. In the experiment, a thin-film aluminum space mirror, called Znamya, was to have been deployed as part of a program to illuminate regions of the Earth at night with reflected sunlight. As the rotating dispenser unfurled what was supposed to become an aluminum disk 25 meters in diameter, another console in the control center suddenly issued a command to deploy a boom-mounted antenna. The boom extended directly into the space where the disk was rotating. As a result, the aluminum wrapped itself around the boom and tore itself into shreds. The operator claimed that nobody had told him there would be any physical contact.

After the failure, senior space engineer Vladimir Syromyatnikov, the developer of the mirror, remarked bitterly to a TASS reporter that “Our style of life is responsible—such a complex experiment demands more time, more specialists.” When asked why the command to deploy the antenna had not been canceled, he answered, “Because we didn't think of it.”

Salvaging interim capabilities and long-term recovery

Following the cancellation of the burn, TsUP officials were in no hurry to reschedule it. The station’s orbit was plenty high enough, and the upcoming launch of a Progress cargo ship proceeded without incident.

The question of whether the engines will work remains unanswered. Previous space station modules in the Salyut program, and Mir, were equipped with such engines. The main engines of Salyut-7, launched in April 1982, were last used four years later during a joint maneuver with the just-launched Mir, but by now that experience was twenty years in the past. Mir’s back end was soon blocked by the addition of a new science module, and its pair of engines was never fired again after its first year in space.

The SM engines, with a rated thrust of 3,090 newtons, are canted fifteen degrees outwards from parallel to the module’s long axis. Furthermore, they have a small range of inboard/outboard swivel, about plus/minus 5 degrees, to fine-tune the thrust vector close to the station’s center of mass.

This thrust centerline cant, in theory, allows either engine to be fired, alone or as a pair, with or without a docked vehicle at the SM’s back end. There would be a lot of plume impingement on the docked vehicle, and perhaps contamination of sensors from fuel residue, but it should hold together.

So at the very least, engine number one, whose cover opened fully, is available for testing and verification. Engine number two could probably also be used once the logic sequence of checking for a fully-opened cover is deleted. So the next time the aft end of the SM is open, now planned for mid-September, another test is possible.

Meanwhile, the docked vehicles remain the station’s mainline orbital adjustment engines. Although the vehicle docked to the SM’s back end thrusts most efficiently through the station’s center of mass, even vehicles docked at the Russian segment’s two side ports can be used to impart thrust, if needed. A test firing last February showed this was feasible, if less efficient.

So even the current computer commanding problem with firing the engines of docked vehicles from the station’s on-board computers is just another complication, not a crippling catastrophe-in-the-making. The overlapping complexity of the station’s own engines and its visiting “propulsion providers” provides a wide range of options, just as it has—this time—provided another example of a system far too complicated to ever really predict with any certitude.

Things will keep catching station operators by surprise. As long as their bag of tricks stays just one layer deeper than the demands placed upon it, things should work out. That’s the unrequested—but unavoidable—lesson of this highly informative “non-event”.

James Oberg (www.jamesoberg.com) is a 22-year veteran of NASA mission control. He is now a writer and consultant in Houston.

Home