Roswell that ends well, part 2by Dwayne A. Day
|
| I realize that while skepticism might be easily acquired, research skills are not. |
Of course, lots of people who maintain an interest in paranormal studies beyond their youth manage to come up with an explanation that satisfies them—it is real, it is legit, but it is also suppressed, they conclude. There may be a Department of UFOs in the government, but it is super-double top secret. There is a conspiracy to suppress it. The X-Files had the best twist on this: it existed, but it was located in the basement of the FBI and run by a crackpot and nobody else at the Bureau took it seriously, even though it was all true…
I never really bought the conspiracy angle once I became aware of it. It just didn’t seem to make sense. After all, if there were space aliens flitting about, why would the government keep them secret? If they were a threat, wouldn’t the politicians want to play that up, just like they did with the Soviet threat during the Cold War? Aliens would be a great justification for a bigger space program. Hell, aliens would be a great justification for raising taxes.
Anyway, I think that I was always a skeptical kid. But skepticism alone is insufficient. You need tools to turn skepticism (which is nearly equal parts emotion and logic) into something more useful—analysis.
It took me awhile to acquire the research skills to accompany that skepticism. And reflecting back on it, I realize that while skepticism might be easily acquired, research skills are not. I’m not sure where and exactly how I acquired them. I don’t remember ever taking a class in college or graduate school where I was specifically taught how to chase down information, cite sources, evaluate them, and weigh their credibility. But I had a lot of classes that helped to impart those skills. I had a number of good professors in college who taught classes where we learned how to check the claims made by book authors, check their bios, look at their footnotes, question their statements. I remember professor John Mueller once chastising me for missing a ridiculous assertion in William Shawcross’ book Sideshow that the Khmer Rouge had been driven insane by American bombing. I learned to read more closely after that.
The most vivid bit of instruction I ever got on this was not related to my formal education at all. In the early 1990s I went to a lecture by William Leary about his research on the CIA airlines, including Air America. Leary carefully explained how he had sought to track down the most inflammatory rumor about Air America, that it had been involved in drug smuggling during and after the Vietnam War. That claim had originally appeared in a 1979 book by Christopher Robbins. Later it appeared in the 1987 movie Lethal Weapon and much more prominently in the 1990 movie Air America, starring Mel Gibson. As such, it became enshrined as a pop culture myth.
Leary explained how he had tracked down the origin of the claim in Robbins’ book and determined that it came from a source who was, well, unreliable (I seem to remember Leary saying that the guy’s nickname was “Weird Harold.”). Leary then explained how he had talked to multiple Air America pilots and tried to gather corroborating or contradictory evidence. The claim that the CIA had run drugs during the war, Leary concluded, could not be supported. But it was his explanation of how he had methodically checked the historical evidence and evaluated its credibility that was a lesson in itself.
This brings me, albeit by the scenic route, to Annie Jacobsen’s book Area 51, which I wrote about in this space two weeks ago. (See “Roswell that ends well”, The Space Review, May 31, 2011) Jacobsen got a burst of publicity that probably tickled her publisher and may have sold some books (I don’t know, is there such a thing as bad publicity in the book publishing world?). But after the initial round, the publicity is turning sour. Nightline did an expose that made Jacobsen look, well, dimwitted. My own review in the San Francisco Chronicle is not positive, and I know of at least another negative one in the works. And then there’s this review by Richard Rhodes (author of The Making of the Atomic Bomb) that just appeared in the Washington Post:
“Annie Jacobsen, a Los Angeles-based independent journalist, does an adequate if error-ridden job of reporting on these black-budget projects and several others besides, using the classic investigative method of interviewing dozens and dozens of worker bees from engineers to security guards and piecing their stories together. Then, like a test pilot who pushes her plane too far, she crashes and burns on the grisly tales of an unnamed single source, supposedly an Area 51 engineer and Manhattan Project veteran who leads her on a wild goose chase of honking absurdity straight down the UFO vapor trail into the very heart of conspiratorial darkness. There Jacobsen is told that Auschwitz butcher Dr. Josef Mengele, the German aircraft-designing brothers Walter and Reimar Horten and Soviet dictator Josef Stalin conspired back in the late 1940s to scare America silly with a Nazi-Soviet flying saucer crowded with wobbly 13-year-olds with large, surgically altered heads. Except that the thing crashed. In a barren corner of New Mexico. Really.”
Read Rhodes’ entire review because it’s a whole lot of fun.
Rhodes referred to her book as “adequate, if error-ridden.” It is indeed error-ridden, but adequacy is a relative term. It’s not an adequate book if you actually care about accuracy. And you should care about accuracy.
| It is actually difficult for somebody who is not a subject-matter expert in nuclear weapons development or classified aircraft programs to see the problems at first read. |
One aspect of the coverage of Jacobsen’s book that I find disturbing is how many people have reviewed the book, torn it apart because of the nutty Roswell claim, yet still referred to it as “well-researched.” It isn’t. Not by a long shot. But many of these reviews have implied that if the publisher simply eliminates that final chapter from the inevitable paperback edition, then Area 51 will be a fine book filled with lots of new and detailed information on American atomic weapons and classified aircraft. That is not the case at all, and simply deleting the Roswell goofiness would still leave a poorly-researched, and error-filled, book.
It is actually difficult for somebody who is not a subject-matter expert in nuclear weapons development or classified aircraft programs to see the problems at first read. What they are likely to see are Jacobsen’s 80 pages of notes and think that she spent a lot of time in libraries and archives. And they are likely to read about her many interviews and think that this means she did a lot of work. Both may be true, but both still fail to make this a well-researched book. And here’s where the book would be really useful as a teaching lesson for some student just entering college who needs to learn the skills to accompany their skepticism.
The first lesson that this book teaches us is that the presence of a huge number of footnotes does not in itself indicate “good” research. Quantity of sources is not the same as quality of sources, or an indication that the author has used the right sources. What a critical reader needs to do is examine those sources closely and determine their quality and credibility. Jacobsen has taken most of her hits over this book because her most outrageous claim—the Roswell story—is based upon a single unnamed source. But many of her other claims in the book are similarly based upon a single source, and not always the best one. She also has a tendency to misread the sources that she does use.
But the source material cited in the book and their quality are not the only troubling aspect, or opportunity for a teachable moment. Area 51 highlights another important issue. The second lesson of this book is that it is not simply what sources an author does use in a work, but what sources they don’t use, either deliberately, or inadvertently.
Two sources that Jacobsen did not use, but should have, are the two U.S. government historical studies of the Roswell incident. One, from 1994, is called The Roswell Report. The other was called Roswell: Case Closed. Both can be downloaded from the Internet. They provide the official US government position on what occurred at Roswell in 1947, which Jacobsen, as a supposed “investigative journalist,” should have at least acknowledged.
The omissions don’t stop there. Much of Area 51 is focused on classified aircraft development, and yet Jacobsen has missed many important sources on this material. For example, she cited Chris Pocock’s 1989 book on the U-2 spyplane, but completely missed his far more comprehensive 2005 book on the U-2, which remains the most authoritative source on the subject. For information on the SR-71 she cited a 1993 book by Paul Crickmore, but not his 2000 or 2010 updates, nor the half-dozen other books Crickmore has written since 1993. There’s also no mention of Peter Merlin’s history of US government statements about Area 51 and Groom Lake and their facilities and activities conducted there.
| The space field has not been immune to problems like those exemplified in Jacobsen’s book. |
In fact, these examples of missing sources provide a good mini-lesson for budding researchers: use the latest sources, and check to see if the authors you cite have updated their research. Had Ms. Jacobsen simply entered Pocock’s or Crickmore’s names into the search box on Amazon.com, she would have discovered that they have written more recent works than the ones she used. Merlin’s article was a little harder to find—it requires a couple of mouse clicks.
There are other lessons that this book can teach. For instance, failing to cite a few relevant sources does not indicate a fatally-flawed work. But failing to cite many relevant sources, particularly the most important ones in the field, is a strong indication both of sloppy research, and a lack of understanding of the field. Similarly, a few errors in a book are acceptable (because nobody is perfect). But many errors indicate that something more serious than common human frailties is at work.
Under normal circumstances, a reader should give an author the benefit of the doubt, at least at first. But in this case there’s clearly a pattern. Rhodes concluded his review by calling Jacobsen journalistically incompetent. He was referring to her failure to properly research the absurd Roswell claim. But it’s not like the book is a paragon of great research and well-supported arguments right up to the point where it smashes into a wall—it is badly researched and badly sourced throughout.
The space field has not been immune to problems like those exemplified in Jacobsen’s book. Tom Frieling noted that Rocket Men, published in 2009, received positive reviews in numerous publications, but eventually started receiving many negative reviews that pointed out serious factual errors—errors that were not even corrected for the paperback edition. (See: “Don’t know much about history: setting the record straight on Rocket Men”, The Space Review, May 24, 2010.)
Another excellent example is Gerard DeGroot’s 2006 book Dark Side of the Moon (see “Review: Dark Side of the Moon”, The Space Review, December 11, 2006). When DeGroot’s book was published, some reviewers criticized it for a whopper of an error: the claim that Apollo 9 flew around the Moon rather than staying in Earth orbit for a test of the Lunar Module. Although a major error that called into question the author’s understanding of the program he was writing about, it did not undercut his primary thesis. But that ultimately proved irrelevant, because later reviewers demonstrated that DeGroot committed plenty of other errors that did undermine his thesis. Reviewers took the book apart for myriad errors, shallow source material, and a tendency to use any incident, no matter how unrelated to the subject matter, to attack the Apollo program.
| Why do publishers produce error-filled books? And why do they fail to correct the errors when they issue trade paperback editions? Of course, the simple answer is money. |
All of this was driven home to me in the past few years when I was considering writing an academic paper on NASA’s decision in the summer of 1968 to send Apollo 8 around the Moon. I know from experience that this story is recounted in great detail in Charles Murray and Catherine Bly-Cox’s excellent 1989 book Apollo: The Race to the Moon. So I consulted that book, carefully looking over their account and digging into their footnotes. Their research is so good, their analysis so sound, that I reluctantly concluded that I cannot really improve upon it. That’s the difference between authors who know how to do research and properly use their sources, and those who don’t—over two decades later, the Murray and Cox book is still solid.
Why do publishers produce error-filled books? And why do they fail to correct the errors when they issue trade paperback editions? Of course, the simple answer is money: they don’t want to spend money on fact-checkers. But there’s another factor at play. Publishers still generally take plagiarism seriously, but that probably has more to do with legal liability than any concern about ethics. Factually inaccurate books probably don’t cost publishers money, particularly if the errors are only pointed out after the books have made their initial sales. Thus, it is up to the reader to be aware.
The average reader is not going to have the specialized knowledge to take apart every book. But the great thing about acquiring the critical analysis and research skills to accompany a (hopefully) skeptical attitude is that these skills are useful even when you’re not putting a lot of effort into it, no matter what the subject. Thus, the next time you wander into Area 51, or the Bermuda Triangle, you’ll have the tools to navigate your way back out.