overview

Advanced

'...The peer review process is broken...' - '...don't have the "situational awareness"...'

Posted by archive 
'...The peer review process is broken and that systemic failure wastes billions of dollars and threatens the prosperity of all mankind.'

<blockquote>'The final product of most scientific research is publication. The number, quality and positioning of publications (the prestige of the journals in which your work appears) are critically important to one’s career as a scientist. Your promotion through the ranks of academia depends on these criteria, but also your chances of funding do as well. The most brilliant concept has virtually no chance of being funded if not accompanied by a track record of publications along that line of research. The same people who review funding proposals also review articles submitted for journal publication. This ‘peer review’ process, which works identically in both funding and publication, has created a perfectly circular chain of reasoning. If your ‘brilliant concept’ interferes with the current prevailing wisdom, you cannot join in the circular chain.


The same circular reasoning also applies to experimental data. I once considered a career in particle physics and was working in such a lab. I sat down one day with the scientist in charge of data analysis. The experiment would acquire 1000 hours of data at a national lab and all of the results would be fed into the computer at the end. This happened to be a search for a new particle, and the theory suggested that a total of six of these would be produced in the 1000 hours. I naively asked how long analysis would take and was told ‘about a year’. I was shocked. “Why so long?” I asked. I was told that the processing would be adjusted and the entire dataset reanalyzed if the number was quite different than 6. That was my last day in particle physics.

As a working scientist, I warn my students about this trap. Yes, some measurements in a dataset do need to be discarded, or modified, if you know they are erroneous. But as the stakes increase (money and reputation), it becomes too easy to discard real data that does not fit in with one’s hypothesis. Most of my students have read the Harry Potter series of books, so I use the example of the Mirror of Erised. The inscription over the mirror reads, “Erised stra oyt ube cafru oyt on wohsi”. I Show Not Your Face, but Your Heart’s Desire.

The scientists in Climategate did not start out corrupt, but they held too tightly to their hypothesis (man is destroying the planet). Once they became the millionaire rock-stars of the scientific world, they spent their days in front of the mirror of Erised.

One would certainly like to believe that this is an aberration in science, but it is not. The peer review process is broken and that systemic failure wastes billions of dollars and threatens the prosperity of all mankind.'
- Bob57, Dec 20, 2009 (Source)</blockquote>


'...Younger guys often don't have the "situational awareness" to know that making the client happy should not extend past strict ethical bounds.'

<blockquote>'There's a growing parallel problem in my world: practically fraudulent project proposals. The renewable energy "industry" is full of nefarious individuals that purport to develop projects (wind, solar, landfill gas, etc.). Too often, conceptional engineering is barely better than what would come from 30 minutes of back-of-the-napkin brainstorming. The financial basis for raising investment money is frequently "justified" on excessively "half-glass-full" accounts of capital cost, ludicrous engineering & construction schedules and unattainable revenue production projections.


Increasingly, professional engineers are getting caught-up in the frenzy to "make it work". Some old guys like me know that it's past time to draw-the-line. Younger guys often don't have the "situational awareness" to know that making the client happy should not extend past strict ethical bounds.

There is tremendous pressure to "go-along-get-along" and given the federal commitment to "renewables", the bias to "press on" overwhelms any technical or commercial objection. Frankly, a lot of good folks have succumbed to what NASA used to call "go fever"....which almost always ends bad.

Risk assessment in any endeavor necessarily rests on a stack of interlocking estimates that must be reality based...otherwise the overall assessment will be hopelessly flawed (and the flaws always mask higher and sometimes fatal risks).

The concept of "ethics" needs to be restored, especially amongst science and engineering professionals whose work forms the basis of herculean commitments in both the public and private sectors.'
- Grant from Comfort, Dec 20, 2009 (Source)</blockquote>



<center>'...others are incompetent and are securing a life-long job simply by being agreeable...'
- 4USA, Dec 20, 2009 (Source)</center>