A failure of foresight and oversightby Jeff Foust
|
 An image from NTSB documents released last week showing debris from the accident arranged in a hangar for investigation. (credit: NTSB) |
Anatomy of an accident
The fourth powered flight of SpaceShipTwo, designated PF04 by Scaled, was to be the most ambitious flight yet of the suborbital spaceplane. Scaled planned to fire the rocket motor on the vehicle for 38 seconds, twice the duration of previous tests. That would accelerate the vehicle to Mach 2 and a peak altitude of about 41,100 meters (135,000 feet). It would also be the first flight test of a new solid motor, using a nylon-based fuel rather than the rubber-based one of previous tests.
| A town hall meeting, Scaled officials said, was held at the request of Virgin Galactic CEO George Whitesides, who asked Scaled officials after the loads issue, “What else are we missing?” |
The test was also months late. “The goals for PF04 morphed a couple of times driven by dependency on the rocket motor,” Scaled stated in a submission to the NTSB, one of dozens of documents about the investigation released by the board the day of its public hearing on the investigation. Planning for the flight began immediately after the previous powered test flight, PF03, in January 2014, on the assumption it would be a longer test of the existing motor. “However, due to continued issues with that path, work was ceased on that effort by June,” the company said.
In late May of 2014, Virgin Galactic announced that it was switching to the nylon-based, or polyamide, fuel, requiring other vehicle modifications. (Some of the specific details of those modifications, including the contents of extra tanks installed in the wing of the vehicle, are redacted in the publicly-released documents.) “This modification cycle along with associated rocket motor development and qualification testing paced the program from May through PF04 in October of 2014,” the Scaled document states.
Scaled had planned to conduct the PF04 flight on October 23, but postponed that after deciding it needed more time to analyze new data on loads the vehicle would experience. In a November 6 interview with NTSB investigators, Matt Stinemetze, the program manager at Scaled for “Tier 1B” (SpaceShipTwo and WhiteKnightTwo), said he felt pressure from Virgin Galactic to carry out the flight, who he felt expressed “some frustration” that the loads issue wasn’t previously identified, but delayed it because he felt the vehicle was not ready. “He had 100% support from his management to stop the flight if they were not ready,” the NTSB summary of the interview stated.
As PF04 slipped from October 23 to October 31, Scaled and Virgin Galactic used the time to hold a “town hall” meeting. That meeting, Scaled officials said, was held at the request of Virgin Galactic CEO George Whitesides, who asked Scaled officials after the loads issue, “What else are we missing?” That meeting, which lasted four hours, was considered “productive” by Scaled, but did not result in any issues that would further postpone the PF04 flight.
One issue that did come up, recalled Scaled vice president Cory Bird, was the feather locks on SpaceShipTwo. “They talked about the feather locks ‘quite a bit,’” Bird recalled, after Whitesides asked about that system.
The locks are a critical part of SpaceShipTwo’s feathering system, which raises the twin tail booms to a 60-degree angle to provide increased drag and stability during reentry. The locks are intended to keep the booms in place when the tails are in the lowered position, since aerodynamic forces, particularly in the transonic region around Mach 1, might raise them. Scaled engineers recognized that a feather deployment during that phase of flight would be “catastrophic.” It was important, then, to keep the feather locked during the early phase of the powered part of the flight, as SpaceShipTwo accelerated past Mach 1.
Another aspect of vehicle safety, though, required the feathers to be unlocked while the engine was operating. The failure of the feather to rise when needed during reentry—because, perhaps, the locks failed to disengage—would also be catastrophic to SpaceShipTwo. Thus, the pilots had to unlock the feather once beyond the transonic region, so that if the locks failed to disengage they could abort the mission at a speed low enough to allow a safe landing without the feather.
Scaled engineers believed that the speed at which the feather could be unlocked was as low as Mach 1.2, but added some margin to that, instructing the co-pilot to unlock the feather once SpaceShipTwo reached Mach 1.4. At Mach 1.5, a warning would appear if the feather was still locked, and pilots were instructed to shut off the motor and abort the mission if the feather remained locked at Mach 1.8. PF04, with a planned top speed of Mach 2, was the first test flight where the feather had to be unlocked during powered flight: on the previous two test flights, the feather was unlocked at around engine burnout, and was never used at all on the first powered flight.
| Alsbury’s job during the powered portion of the flight was to call out when the vehicle reached Mach 0.8, read out the angle of the stabilizers, and, at Mach 1.4, unlock the feather. |
While the feather locks were discussed at the town hall meeting, it did not appear to be an issue for the flight. The loads analysis was completed, and nothing else appeared to prevent the October 31 flight. Those involved with the test flight, including surviving pilot Peter Siebold, didn’t report anything out of the way with the preparations for the flight, or the events leading up to SpaceShipTwo’s release from WhiteKnightTwo at 10:07 am Pacific Daylight Time in the skies north of the Mojave Air and Space Port. It was not trouble-free: takeoff was delayed to allow SpaceShipTwo’s nitrous oxide propellant to reach the right temperature, as well as concern about winds; and a flight computer on SpaceShipTwo spontaneously rebooted while still attached to WhiteKnightTwo. Nothing, though, that seemed out of the ordinary for a test flight.
Prior to the flight, co-pilot Michael Alsbury memorized a set of procedures he had to carry out during the powered portion of the flight. Given the fast pace of events, Scaled concluded using a conventional checklist would not be efficient, nor a “challenge-response” system where one pilot verbally called out a command and the other confirmed it. Alsbury’s job during the powered portion of the flight was to call out when the vehicle reached Mach 0.8, as a warning that they were approaching the sound barrier and the “transonic bobble,” a set of oscillations the vehicle experienced as it passed through the sound barrier; read out the angle of the stabilizers; and, at Mach 1.4, unlock the feather.
At 10:07:26.91 PDT, Alsbury called out, “Point eight,” a reference to the speed SpaceShipTwo had reached, several seconds after engine ignition. Tests by NTSB in the simulator after the accident indicated it would take about 15 seconds for SpaceShipTwo, pitching up to gain altitude, to reach Mach 1.4, when Alsbury was to unlock the feather.
Yet, less than half a second after calling out the vehicle’s speed, the NTSB transcript of the cockpit audio and video recorders shows Alsbury’s left hand had moved to the feather unlock handle. The handle, which looked something like a throttle, was designed so an accidental motion would not unlock the feather: the pilot had to shift the lever to the right to move it out of “detent,” then pull it down. At 10:07:28.39, Alsbury said, “Unlocking.” Six-tenths of a second later, the video showed the handle in the unlocked position.
Less than four seconds later, the recording ends. Neither Alsbury nor Siebold made note of the unlocked feather, and the recording heard nothing more from them beyond each of them saying “pitch up” a little more than a second before the recording ends.
Images and video, though, captured the fate of SpaceShipTwo, breaking apart under aerodynamic forces created when the feather extended as the vehicle was still accelerating at transonic speeds. The debris scattered across the desert floor, with a couple pieces landing about 50 kilometers to the northeast, near a high school in the town of Ridgecrest and on a golf course at the Naval Air Weapons Station China Lake. Siebold, miraculously, survived the accident, thrown clear of the disintegrating vehicle and parachuting to a landing despite broken bones and other injuries.
 An image from NTSB documents from a photographer on the ground showing SpaceShipTwo beginning to break apart. (credit: NTSB) |
Human error, flawed system
So why, then, did Alsbury unlock the feather much earlier than planned? A precise explanation for his actions eludes investigators: Alsbury died in the accident, and Siebold told investigators he was unaware that the feather had been unlocked. Both pilots were in good physical and emotional health at the time of the accident, and had trained extensively in simulators for the flight.
| “I think the question we’re all trying to answer here is, ‘Why did the co-pilot unlock the feather early?’” asked Sumwalt. |
NTSB investigators, though, think that Alsbury was subject to “stressors” during the flight that could have led to a lapse in judgment. “Stressors were present during the boost phase of flight that likely contributed to the co-pilot unlocking the feather prior to 1.4 Mach,” said Katherine Wilson of the NTSB at the public hearing. That including the memorization of the tasks required to be carried out during the boost phase of the flight.
“Because of the importance of unlocking the feather before 1.8 Mach, the co-pilot might have been anxious to unlock the feather to avoid aborting the flight,” she said.
Another factor, she added, was the environment of the flight, including the g-loads and vibrations. That environment could not be replicated in the simulator, and Alsbury had last flown SpaceShipTwo on its first powered test flight, 18 months earlier. “The lack of recent experience with powered flight vibration and loads could have increased the co-pilot’s stress and thus his workload during a critical phase of flight,” she said.
“I think the question we’re all trying to answer here is, ‘Why did the co-pilot unlock the feather early?’” asked Robert Sumwalt, one of the four current board members of the NTSB (a fifth seat is vacant.) “I think that’s a question that people have been pounding their heads trying to figure out for nine months now.”
While investigators could only assume why Alsbury might unlock the feather early, they were much clearer on a more fundamental issue: Scaled Composites, in its development of SpaceShipTwo, never considered the possibility that a pilot might make that mistake, even as it did various analyses of other potential failure modes of the vehicle. “No mitigations were considered to prevent the flight crew from unlocking the feather locks early,” noted NTSB investigator Mike Hauf at the hearing.
Current and former Scaled officials, in NTSB interviews after the accident, confirmed they didn’t think a pilot would ever unlock the feather early. “A pilot-induced early unlocking of the feather system was not considered as a ‘what if’ that he was aware of,” the summary of an interview with former Scaled chief aerodynamicist Jim Tighe stated.
“They never imagined that the feather system would be unlocked too soon,” Bird recalled in his interview. “Unlocking the feather system too soon was not discussed and the issue of when to unlock was not reiterated in that [town hall] meeting.”
Scaled, NTSB concluded, had essentially overlooked the possibility that its pilots might make a mistake, creating a situation where one such error could lead to the loss of the vehicle.
“The fact is, if you put all of your eggs into the basket of a human to do it correctly—and I don’t mean this flippantly, because I’ve made plenty of mistakes—but humans will screw up anything if you give them enough opportunity,” Sumwalt said. “A mistake is oftentimes a symptom of a flawed system.”
“The assumption was that these highly-trained test pilots would not make mistakes in those areas,” NTSB Chairman Christopher Hart said in an interview after the public hearing. “But, truth be told, humans are humans, and even the best-trained human on their best day can still make mistakes. That’s one of the areas they [Scaled] did not adequately cover and led to this accident.”
That was reflected in the statement of probable cause that the NTSB approved at the hearing. Investigators originally proposed a statement that focused on the co-pilot’s premature unlocking of the feather, with Scaled’s failure to consider single-point human failures like that in its planning as a contributing cause. Hart, though, wanted to emphasize Scaled’s “failure to protect against the possibility” of human error.
After a half-hour recess where NTSB board members and staff worked on that statement, the board unanimously approved one that mentioned both:
The National Transportation Safety Board determines that the probable cause of this accident was Scaled Composites’ failure to consider and protect against the possibility that a single human error could result in a catastrophic hazard to the SpaceShipTwo vehicle. This failure set the stage for the copilot’s premature unlocking of the feather system as a result of time pressure and vibration and loads that he had not recently experienced, which led to uncommanded feather extension and the subsequent aerodynamic overload and in-flight breakup of the vehicle.
 The feather lock handle in the SpaceShipTwo cockpit, which required a pilot to move it to the right and then down to unlock the feather. (credit: NTSB) |
Perceived political pressure
SpaceShipTwo was operating at the time under an experimental permit issued by the FAA’s Office of Commercial Space Transportation (AST) in May 2012, and subsequently renewed in 2013 and 2014. That raised another key question: why didn’t AST identify that potential failure mode and request Scaled to correct it?
| “The assumption was that these highly-trained test pilots would not make mistakes in those areas,” Hart said. “But, truth be told, humans are humans, and even the best-trained human on their best day can still make mistakes.” |
AST had, in fact, issued a waiver to some aspects of Scaled’s permit in 2013. “Scaled did not meet these [permit] requirements because it did not identify human or software error as causing hazards,” it stated in the waiver, published in the Federal Register in July 2013. “It did not identify these errors as causing hazards on the grounds that the mitigations it had in place would prevent the hazards from occurring.”
The waiver didn’t describe any specific hazards Scaled overlooked beyond general human or software error, but concluded that several factors mitigated any risks it caused. Those factors included Scaled’s training program, an incremental approach to flight testing, use of chase planes and having two pilots on the vehicle, and the remoteness of the test area in California’s Mojave Desert.
Scaled executives told the NTSB that they were surprised by the waiver, since they has not specifically requested one nor were sure why it was issued. One AST employee interviewed by the NTSB said that FAA’s general counsel considered an application that did not meet requirements “an implicit application for a waiver,” something AST staff members did not agree with.
Others at AST, or working for the office, expressed frustration not just with the process of granting the waiver but also getting information from Scaled. Questions they wanted to pose to Scaled as part of the permit review process had to go through AST management. “If a question was not relevant to public safety, it would be redlined,” an employee told NTSB. “A redlined item meant that the item was deleted; that happened frequently.”
The emphasis on public safety is an aspect of AST’s current regulatory restrictions: its process for both launch licenses and experimental permits focuses on the safety of the uninvolved public and not those on the vehicles themselves, other than their roles as safety systems for the vehicle. AST had been filtering questions, the NTSB found, to limit the burden on industry those issues directly related to the licensing and regulatory process, after some in industry complained.
| “Scaled and the FAA had thought that all the mitigations had been included in the fault trees, and he found that not to be true,” Hardy told the NTSB. |
Terry Hardy, a former AST employee who was working for them as a consultant, told the NTSB he chose to stop working with the office shortly after the accident. “When asked why he stopped after the accident, he said after 3.5 years he did not feel his recommendations or the work he did was improving the safety process,” the NTSB stated in a summary of its January interview with Hardy. “He felt that after offering recommendations to the FAA he was ‘spinning my wheels’ until the FAA made significant changes to the way they approached system safety and their evaluations. He let the FAA AST managers know this.”
Hardy said he had concerns with Scaled’s approach to failure analysis, including its use of quantitative analysis on vehicles with little or no flight experience, an approach that allowed the company to skip documentation of how it mitigated the issue if the calculated risk was sufficiently low. “If that quantitative number met those criteria, they were done,” he told the NTSB. “The idea of using quantitative analysis on an early vehicle that had never flown could be used as a tool, but should never be used alone.”
Hardy said he talked with AST staff about his concerns that Scaled’s approach failed to account for all possible failures of SpaceShipTwo. “Scaled and the FAA had thought that all the mitigations had been included in the fault trees, and he found that not to be true,” he told the NTSB. He assumed AST staff passed on his concerns to management, but acknowledged he didn’t know what the FAA did with them.
One comment that attracted considerable attention was when Sumwalt quoted from the draft, as-yet-unreleased, report, citing “a lot of pressure, political pressure” to issue experimental permits. “When I read that, that worries me,” he said. “What do we mean by pressure, political pressure? What is this?”
NTSB investigators didn’t clarify where that political pressure was coming from, nor did Hart in an interview after the hearing. “Our interpretation of those statements we obtained from staff were pressure of two types,” he said. “One is to meet deadlines, because they have very aggressive deadlines that they are proud to say they have never missed, so that’s one. Number two is figuring out where the line is drawn between protecting the public and mission assurance.”
Hardy, in his NTSB interview, did mention political pressure, but he was referring to events a decade earlier. “The FAA safety engineers were not allowed to talk directly to the applicants, and this was based on political pressures to reduce the burden on applicants. That was what they felt in 2004,” he recalled, an outcome of a “lessons learned” discussion after SpaceShipOne won the Ansari X PRIZE.
“What he felt now was that culture of not wanting to over-burden the applicant still remained,” NTSB stated in its summary of the Hardy interview. “As a result, there was a screening of questions and a limitation on direct communication with the applicant.”
Success depends on safety
While NTSB has yet to issue the full report on the accident investigation, the board did unanimously approve ten recommendations that stemmed from their investigation. It directed eight of the recommendations at the FAA to improve their reviews of experimental permit applications and safety inspections after those permits had been issued.
Some were specific to the contributing causes of the accident. Those included improving application review processes to ensure companies identify all “single flight crew tasks that, if performed incorrectly or at the wrong time, could result in a catastrophic hazard” and develop ways to mitigate those hazards. NTSB also recommended AST develop better communications processes with applicants and “better define the line between the information needed to ensure public safety and the information pertaining more broadly to ensuring mission success.”
| “When I read that, that worries me,” Sumwalt said. “What do we mean by pressure, political pressure? What is this?” |
Some, though, were more generic. One recommendation called on AST to work with the Commercial Spaceflight Federation (CSF) to develop “human factors guidance” for companies developing crewed spacecraft, a recommendation rooted in the NTSB’s funding that there was little formal human factors expertise at either Scaled or NTSB. Another recommendation called for the development of a “lessons learned database” by AST where companies would voluntarily submit and share information about mishaps. Such a database has been something AST, and some—but not all—in industry have proposed for years.
Two of the recommendations were for the CSF. Besides working with the FAA on human factors guidance, it recommended it advise its members to develop emergency response procedures with local authorities in the event of an accident. That stemmed from the fact that while Scaled has such procedures in place for the SpaceShipTwo accident, there were issues that delayed to dispatch of a helicopter to the site where Siebold landed.
CSF, in a statement, quickly accepted the NTSB’s recommendations. “CSF welcomes the NTSB’s report, and we pledge our support to promptly carrying out the recommendations given to us by the Board,” CSF president Eric Stallmer said in a statement that also called for increased resources for AST. The FAA, meanwhile, is still studying its recommendations, and plans a formal response within 90 days.
The NTSB offered no recommendations to either Scaled Composites or Virgin Galactic. Scaled’s role in the program is now diminished: Virgin’s The Spaceship Company is building the second SpaceShipTwo vehicle, which will presumably fly under a permit or launch license that Virgin Galactic applies for. (Virgin had applied for a launch license for the first SpaceShipOne, but had asked the FAA to “toll,” or suspend, evaluation of that application until Scaled completed its planned tests under its permit, something that Virgin expected prior to the accident to be complete by January 2015.)
“Safety has always been a critical component of Scaled’s culture and, as the NTSB noted today, our pilots were experienced and well-trained,” Scaled said in a statement issued after the NTSB hearing. “As part of our constant and continuing efforts to enhance our processes, we have already made changes in the wake of the accident to further enhance safety. We will continue to look for additional ways to do so.”
| “The success of commercial space travel depends on the safety of commercial space travel, at the level of every operator and every crew,” Hart said. |
Virgin, in its statement and its submission to the NTSB investigation, emphasized that the accident was not caused by a flaw in the vehicle’s fundamental design, including its propulsion system. The company has already implemented changes to the vehicle and its operations, including a system to prevent a pilot from prematurely unlocking the feather during the boost phase of flight, and a challenge-response communications system between the two pilots—an approach that Scaled previous dismissed as not feasible given the pace of flight operations.
“We remain as humbled as ever by the difficulty of our work and the challenges of space,” Virgin Galactic’s Whitesides said in a statement. “We are encouraged by the progress to date with our second spaceship, and we look to the future with hope and determination.”
Hart, in his closing comments at the public hearing, also sounded hopeful about the future. “Today, the vision of commercial space travel is close to fulfillment,” he said. “Hundreds of people whose only qualification for space flight is their ability to purchase a ticket await the opportunity to go into space on commercial space launches.”
But, he added, safety must not be forgotten in this rush to space. “The success of commercial space travel depends on the safety of commercial space travel, at the level of every operator and every crew,” he said. “Operators can and do compete on many levels, whether in commercial aviation or in commercial space transportation. But when it comes to safety, they must cooperate and collaborate, with each other and with the FAA.”