Is Most Published Research Wrong? p Hacking? (1 Viewer)

[Galbreath34]

I'd say the setup of the video and/or thread title is akin to: "Are all Christian gay pedophiles? Priest scandals".

Are there problems with corporate financed research? Sure. Are these studies widely accepted and used? No. Does peer review really debunk and handle the vast majority of problem science? Yes. Does it miss some? Sure, but not enough to say something ludicrous like "most published research is wrong". Are some soft sciences more suspect to problems? Absolutely. Still, to be honest the "most" claim is even more tilted and biased a claim than that all Christians molest same sex children.

 

[Galbreath34]

Peer review isn't what it used to be. There was an article in a psychology journal that was decrying the lack of unaffiliated peers to review studies. It seems that most peers are beholden to pharma (in psychology) making it difficult to get unbiased information as a consumer.

Psychology as a field is really still suffering too from having treated WEIRD as normal far as studies go for most of its life as a science, which is probably a bigger deal than any peer review issues. Again, you get into specific fields and claims of problems become easier to make than for all of science.

 

[mt15]

My thought reading through the thread is that there's a big difference between different scientific fields. For example, in physics or virology my impression is that the peer review process is very rigorous. I think psychology, sociology probably not so much.

My brother does virology research for a major university, I get the impression it's still pretty "pure" as science goes. My son teaches physics at a small community college, and while he doesn't do research now, when he was in grad school he did, and he still reads a bit in the journals. My impression is that physics research at the university level would still be rigorously peer reviewed as well.

You can't really analyze something like "science" or even "scientific studies" and make broad sweeping generalizations like that. The research is so specialized and varied as to make those types of broad statements meaningless.

 

[guidomerkinsrules]

My thought reading through the thread is that there's a big difference between different scientific fields. For example, in physics or virology my impression is that the peer review process is very rigorous. I think psychology, sociology probably not so much.
...
You can't really analyze something like "science" or even "scientific studies" and make broad sweeping generalizations like that. The research is so specialized and varied as to make those types of broad statements meaningless.

this is a good point - we get enamored with the scientific method assuming that it's a reductive process getting to some essential answer
psychology and sociology don't fit in those and shouldn't be made to
it's tempting b/c we seem to like to make the complex simple

i see it as parallel to the notion of trying to run schools like businesses - on the surface an attractive idea for it's production/product simplicity, but the educational and business models should be kept far far apart

there's a difference between the experimental and the experiential and that difference should be recognized and accepted

 

[Saint_Ward]

My thought reading through the thread is that there's a big difference between different scientific fields. For example, in physics or virology my impression is that the peer review process is very rigorous. I think psychology, sociology probably not so much.

My brother does virology research for a major university, I get the impression it's still pretty "pure" as science goes. My son teaches physics at a small community college, and while he doesn't do research now, when he was in grad school he did, and he still reads a bit in the journals. My impression is that physics research at the university level would still be rigorously peer reviewed as well.

You can't really analyze something like "science" or even "scientific studies" and make broad sweeping generalizations like that. The research is so specialized and varied as to make those types of broad statements meaningless.

Being in Materials Science, I agree here. I think the hard sciences.. Chemistry, Physics, Engineering are pretty well defined and studies actually show something.

I think in Medicine, weight loss, psychology, it's more of a mixed bag. I think some medical studies are very rigorous, but they still with uncontrolled variables of humans. We're all a bit different. Humans are bad controls. We're not the same. So, there is always going to be some issues with those type of studies.

 

[mt15]

i see it as parallel to the notion of trying to run schools like businesses - on the surface an attractive idea for it's production/product simplicity, but the educational and business models should be kept far far apart

As an aside, my son really can't stand this idea. It seems to be pervasive right now in his little corner of post-secondary education. I hear about it all the time.

It reminds me a bit of "best practices" in the medical field. Sure, there are treatment protocols that can be defined as best practices, but they will not be "best" for every single patient. Sometimes lately we seem to be trying to simplify complex issues too much. It's fine to publicize a "best practice" but it's not fine to take away the ability of professionals to make exceptions when the specific situation demands it.

 

[guidomerkinsrules]

As an aside, my son really can't stand this idea. It seems to be pervasive right now in his little corner of post-secondary education. I hear about it all the time.

It reminds me a bit of "best practices" in the medical field. Sure, there are treatment protocols that can be defined as best practices, but they will not be "best" for every single patient. Sometimes lately we seem to be trying to simplify complex issues too much. It's fine to publicize a "best practice" but it's not fine to take away the ability of professionals to make exceptions when the specific situation demands it.

I wonder if all of these are issues with administration/management ?

 

[sinner mike]

In psychology the big problem is that states of being are hard to measure if not impossible. The field of psychology has worked to be seen as a rigorous in its investigations of human being and to some extent have been successful. But on the other hand psychology has engaged in some serious snake oil research and have tried to validate it. Because it is not a hard science like physics the researchers can manipulate the methods and procedures to attain a predetermined outcome. This can be seen in antidepressant research, and research into evidence based practices. Evidence based sounds good but is bunk according to meta-analysis.

 

[Optimus Prime]

<iframe width="560" height="315" src="https://www.youtube.com/embed/0Rnq1NpHdmw" frameborder="0" allowfullscreen></iframe>

 

[Saint_Ward]

I'll have to watch the JO video later.

However, I'd imagine the issue is research that is based off of Data Dredging. i.e. looking at multiple studies and data mining until you see some sort of statistical relationship. Psychology is famous for combining studies, medical too.

I think those studies are more garbage. Variables weren't controlled the same, methods were different, it's a bad use of statistics.

If the research was done yourself, I think that's different.


https://en.wikipedia.org/wiki/Data_dredging

 

[Galbreath34]

Also haven't rewatched JO video, but saw it some time last Summer. IIRC most of what he was on about was the popular misuse, misquoting and miscommunication of science for clicks or to make points. Like taking the early study and saying eating donuts cures cancer when that wasn't what the study said at all.

In the spirit of the video

 

[tomwaits]

Peer review isn't what it used to be. There was an article in a psychology journal that was decrying the lack of unaffiliated peers to review studies. It seems that most peers are beholden to pharma (in psychology) making it difficult to get unbiased information as a consumer.

The Economist had a similar story that there was no money in peer review, so it has been lacking. When it is done, they very often cannot replicate results.