Can we trust peer-reviewed papers?

me-shell (54) · 7 years ago (edited)

Very interesting video. As a budding young researcher myself I know the importance of understanding a quality journal.

Impact factor (IF)

As mentioned in the video I go off the Impact factor (IF). But one thing to note is that the range of IF that is acceptable varies greatly between disciplines. For example Maths type Journals typically have lower IF than those of say some Engineering fields because there is just less research that fits into that specific field.

How peer review works

When I finish a paper and submit it to a Journal which I have chosen based on this criteria. It will then go to the editor. The editor will probably have a quick read but nothing in depth at this stage. Instead he will have a list of researchers who have already proven themselves in their respective areas. Usually 2-3 'reviewers' will be involved (its not just any random). They will get the paper have a thorougher read through and decide if it is quality or not. They can then choose to accept, accept with changes or decline. All of which comes with comments. If the paper is declined it is usually because either it doesn't fit the journal or it is not good. The paper is passed back to the writer who can make changes and submit again for another review process. This can take months to publish a single paper.

Just because you pay doesn't mean its a bad journal

Many journals make you pay to publish a picture in colour. This is because they have to print it and send it to people and colour just costs more. This incentivises images that can stand in black and white. Also a lot of new journals which are usually for new fields of research, which are good, struggle to get enough funds from sales alone. So again paying doesn't always equal bad quality.

Always check content.

Even if I find a journal article that is from a reliable source I never trust it and neither do other researchers. If I need to reference an article I would usually spend a good chunk of time going down the rabbit hole of its references. Does it reference good quality work. Does the findings match up and so forth. Finally on top of this common errors in papers usually boil down to bad stats so that always needs to be checked

Open access

By this I mean anyone can view the papers (don't confuse this with not peer reviewed). This one is a balancing game. Many funding agency's require you to publish in open source so they can keep up to date with your work. It makes for open communication with the world. However, any research institute and library I have ever been in contact with usually have a subscription to all the reliable journals. This makes it easy. I enter in my details and instant available content. Open access is starting to become more reliable you just need to still do all the checks above.

Any questions just ask!

$17.09

18 votes

lemouth (72) · 7 years ago

Today, open access is very very important IMO. It is actually one of the most important point.

All my articles are now published only in open access journals and are available for free from the arxiv (that is spread towards more and more fields). One very interesting platform that I have not yet taken the time to publish with is SciPost. I don't know if you have ever heard about it.

And to answer @lukestokes question, the answer is probably field-dependent and journal-dependent.

PS: I don't have the time to watch the video now as I am at work, but I will do it later.

$0.38

lukestokes (75) · 7 years ago

It's a good video, and yes, "it depends" is usually a safe answer.

$0.00

kyle.anderson (60) · 7 years ago

Scipost is really cool, although I don't read up on as much physics as I used to.

$0.00

justtryme90 (71) · 7 years ago

One thing to remember about things like arxiv is they can be published there with out any peer review. Which means material put there can, be crap. It doesn't mean that it is, but peer review is still, at this point, essential for the reinforcement of research quality.

Open access publishing is essential IMO for the quick effective dissemination of scientific knowledge to all edges of the earth. However these journals must be held to the very highest level of editorial scrutiny and a robust peer review process is essential for this. Should the peer review responses also be made available for all to see upon publication of a paper? I believe it should.

$0.00

lemouth (72) · 7 years ago

There is actually one protection on the arxiv as you must be endorsed to be able to post. This does not prevent from crap, but it at least imposes new users to have one reasonable article to start with.

Should the peer review responses also be made available for all to see upon publication of a paper? I believe it should.

Me too. Which is what I like with SciPost (which I should try one day :p ).

$0.10

lukestokes (75) · 7 years ago

Great comment, thank you!

But one thing to note is that the range of IF that is acceptable varies greatly between disciplines.

Yeah, I think that was mentioned in the video. Good thing to keep in mind.

That's also a good point about payment. I think there has to be mutually aligned incentives on all sides for market regulation to work. Otherwise, we're playing with other people's money, resources, or reputation. Checking the content is key, but many (including myself) barely read the abstract, let alone the full paper, let alone the references. It's a time-consuming process which takes a lot of expertise to do well.

Open access definitely sounds great, but what regulates it? That, to me, is the tricky part. Reviews can be gamed, references can be faked... it's a tricky game.

Thanks again for sharing your insight and experience. Followed.

$0.00

me-shell (54) · 7 years ago

Thanks! I forget sometimes that I actually know stuff haha. You're right he did mention it. I just thought it needed more emphasis. I would consider submitting in a journal with impact factor of 1.6 and other times a 3.8 because they are completely different fields. In this case just looking at the number is irrelevant. This is why I don't stray too far out of my research field (Bio-Mechanical).

I admit the first 3 times I read a paper I barely read past the abstract because you have to sort through first is it related to what you are doing. In my field though if you miss-quote a paper you get hounded so if I use it I better well understand it.

Open access, By this I mean anyone can view the papers (don't confuse this with not peer reviewed).

Open access is great just as long as the journal is still peer reviewed. Wikipedia has a good introduction to what open access means. All the quality open access journals must be peer reviewed in some way.

Glad you enjoyed my comment. :) I found it hard to read some of the others and not want to bang my head against a brick wall. :/

$1.10

lukestokes (75) · 7 years ago

bang my head against a brick wall. :/

I know. Unfortunately there are quite a few "free thinkers" on Steemit who frustrate me with their thinking. They love all things conspiracy theory related, and they make connections to everything. I'm sure my own thinking frustrates many as well when I talk about government as a monopoly on the use of force in a geographic region and how we might be able to thrive without a Hobbesian Leviathan. :)

We all have our stories we tell ourselves which make sense until a better one comes along. I remember how silly and ignorant I was in the past and I'm sure I'll chuckle in the future at myself today. At least, I hope so, because that means I've hopefully learned something along the way.

$0.00

brendio (55) · 7 years ago

Great summary of the important factors in publishing. I think the problems occur when shortcuts are taken with reasoning rather than examining the evidence for themselves. For example, it's published and it fits my beliefs, therefore it must be true. It's peer-reviewed, it must be true. It's open-access and paid to be published, it must be questionable. It's in a quality journal, it must be legit.

I think there is no avoiding taking some of these shortcuts because of limitation of time or expertise to examine further evidence, which is where some appeals to authority can save time, so long as checks are in place.

Publication in a quality, high impact journal can be seen as a stamp of quality. The journal has staked its reputation of the quality of what is published.

$0.00

kyle.anderson (60) · 7 years ago

http://sci-hub.bz

Who cares about open access :P

$0.00

jameskanka (44) · 7 years ago

I was made aware of this earlier because, similar to one of the examples in the video, I heard of fake papers getting published — though not so blatant. Related, I also heard on a podcast, Scientific Guide to the Universe, the story of a dog being on the board of seven editorial boards (http://www.atlasobscura.com/articles/olivia-doll-predatory-journals).

Humor has a great way of revealing the flaws of a system.

Pay-to-play publications is one way that science is undermined in public conscious but there are more subtle flaws in scientific research right now also. Steven Novella, a leader in the skeptical movement, has helped me greatly in my critical thinking skills. One problem he highlights about bad research now is p-hacking. Most don't even know they are doing it as it's a broadly accepted approach to finding significance. From my limited understanding it is the act of expanding on sample size until the results get the significance they need, instead of identifying the right sample size ahead of time. I highly recommend his blog and podcast. Here is an article about p-hacking and the reproducibility problem: http://theness.com/neurologicablog/index.php/science-we-have-a-reproducibility-problem/

$10.02

4 votes

lukestokes (75) · 7 years ago

Thanks James. I've heard of this p-hacking concept before, but haven't looked into it much. Thanks!

$0.00

lys (52) · 7 years ago

Very interesting video. I wrote an article regarding some of these issues. I agree largely with the video. But the video seems to suggest that the problem is with the open access and online journals only and that the only ones to be relied on are the paid high impact factor journals.
The problem is general, the whole journal industry, paid or open access or online, has serious issues with reproducibility and peer review.
Many of the paid journals, which supposedly have strong peer review processes, contain results that are not very reliable.
I am a PhD candidate chemist and I have access to the university library with many "serious" journals. I have found many articles which turn out to be valueless. In my specific field we have learned to trust only publications from 2-3 specific laboratories.
NEITHER "serious" journals nor open access and "free" journals are free of these problems.
Paying for an article is absolutely no guarantee anymore for the quality of the article.

Some universities are rebelling against the paid journal industry and have created Open Access
repositories, before being able to publish there the university reviews and evaluates the publication. So it is not true either that Open Access means it is not peer reviewed.

Here is my perspective on the problems as a scientist around producing papers to get published. I wrote about Open Access sources a while ago:
https://steemit.com/science/@lys/scientific-articles-what-you-have-published-is-no-longer-yours

Here is an excellent article by another writer on Steemit
https://steemit.com/science/@kyriacos/science-under-attack

$8.66

8 votes

lukestokes (75) · 7 years ago

The problem is general, the whole journal industry, paid or open access or online, has serious issues with reproducibility and peer review.

Agreed, but thankfully the scientific process it self exposes these flaws. If we can come up with a better method, let's do it and spread it.

Your first article starts to touch on copyright stuff which just gets me all frustrated at the ridiculousness of government. I'll read the others as well. Thank you!

$0.00

kyle.anderson (60) · 7 years ago

Many of the paid journals, which supposedly have strong peer review processes, contain results that are not very reliable.

This is why everything has to be on a per-article basis.

Going per-group/lab is also good as laboratories form and cement existing trust networks.

Loved your article - following!

Open access is great but not fully executed.

Http://sci-hub.bz for the win.

$0.00

brendio (55) · 7 years ago

I agree that the video has a bit of an anti-open access bias. I think there are also crappy closed access journals too. Impact factors are not infallible and can be "gamed", but they provide a pretty quick clue about a journal's standing within its field, whether open access or not.

A better understanding of what peer review is and isn't would help the community when scientific results are taken from their specialised fields and publicised in the wider community.

I think the biggest problem in all of this is academic laziness and failure to apply critical thinking.

$0.00

kyle.anderson (60) · 7 years ago

Journals have been a great gatekeeper for science for hundreds of years.

While you can blab on and on about impact factors and the overall quality of the journal, I have always treated each article independently. While some journals have rigorous standards, there can always be exceptions.

Citations are a really important part of the equation to keep in mind. There is no substitute for following up on an article and doing your research.

Maybe most importantly: Open Access
Science, unless happening behind closed laboratory doors, should always be open and shared. From pre-prints to final publications, access should not be restricted. If you plan on publishing, your audience is the world. Not the patrons of some corrupt journal.

This is why I support projects like http://sci-hub.bz

Enter a DOI and boom you have a PDF. No login required.

I am lucky that I have access to a university library, this is supposed to be free for all.

$5.75

6 votes

lukestokes (75) · 7 years ago

Thanks Kyle. Great comment. I agree, information wants to be free and to spread! :)

$0.00

justtryme90 (71) · 7 years ago

I have watched the video and I agree with (in general) the sentiment of 100% of points the narrator makes. Your title is a misrepresentation of the point of the video however. It's point was not that there is something wrong with the peer review process, on the contrary, it's point was that the traditional editorial and peer review scrutiny of classical science journals is the best mechanism we have to ensure trustworthy data. Which is true. It is not the only mechanism, and it is not without it's faults and issues however.

The video also goes on at length about open access publishing, at this I draw some issues because there are some open access publications which undergo as much editorial scrutiny as the closed access subscription journals, some of which are run by the same editorial offices as companion journals (nature, science, cell, and a plethora of other high quality editorial offices all have open access journal branches, and the data is as trustworthy from these journals as the parent journal, however the topics just lack the "impact" aka sexyness of work put out in the subscription services).

In short, yes, you can trust peer reviewed journals, so long as they are run by a good editorial office, and get a lot of citations from the scientific community. However not all open access journal are run this way, some really are pay for play, while claiming to be "peer reviewed". It is for this reason that checking a journals impact factor can be useful. Is it a true criterion of what work is better? No. Is it a way to provide evidence for what might be bullshit (in the case of a publication in a very very low impact journal, aka one that researchers never cite)? Yeah it is.

$0.31

lukestokes (75) · 7 years ago

Your title is a misrepresentation of the point of the video however.

Did you notice I used the exact same title as the video? When I share content from elsewhere, I think that's the best approach so those familiar with the content may already know what I'm sharing.

But yeah, as others have commented, he was a bit hard on the pay per play (as he should be, I think) but maybe could have spent some time on how important open access to information is, if we can figure out a good motivational system to keep the quality high.

$0.41

justtryme90 (71) · 7 years ago

Did you notice I used the exact same title as the video?

Then his title is a clickbait misrepresentation of the point he is making in his own video. ;)

he was a bit hard on the pay per play (as he should be, I think)

Definitely, and he should be.

but maybe could have spent some time on how important open access to information is, if we can figure out a good motivational system to keep the quality high.

As I said above, there are plenty of examples of open access publishers with very high editorial standards. The issue is not with open access, it is with fraudulent journals.

$0.31

teamsteem (74) · 7 years ago

Cool! I'll definitely give this its chance.

$0.03

nesian101 (41) · 7 years ago

Good to see another Steemian who watches his videos.

I forgot about him due to his extended hiatus but was still subscribed.

$0.02

lukestokes (75) · 7 years ago

He's one of the few YouTubers I watch right away when they publish. Another (because he's so dang funny) is exurb1a.

$0.00

patricksanlin (63) · 7 years ago

Good video and thanks for sharing.
Please follow me @patricksanlin and upvote. Thanks

$0.00

lukestokes (75) · 7 years ago

Advice: Don't ask to be followed or upvoted without first delivering value. Most people will just move on and ignore you. Those who don't probably won't provide you much value either.

$0.00

patricksanlin (63) · 7 years ago

thanks for advice.

$0.00

builderofcastles (64) · 7 years ago

I really really hate this video.

He states, that the paper wasn't wrongly published because it was about intelligent design, it was that it was full of holes.
Well, if that was the case, then why was the professors entire life destroyed?
Why was he black-balled?
Further, he goes on to state that darwin is criticized, by ... grrrr, all the references are in the form, well darwinism could have happened this way of that way. But no one ever questioned darwinism itself

Having dealt with cancer research and who gets published in peer reviews...
If you come up with a new treatment, or a new classification, your golden.
If you find, discuss or apply a cure, you are black-balled, thrown out, your name destroyed and often shot and killed.
Since we are supposedly looking for a cure for cancer, than the ENTIRE peer reviewed cancer papers are rubbish.

And don't even get started on Monsanto buying the peer reviewed papers to publish their own papers and discredit any opposing views.

Now, to stick a fork in this topic.
In two generations, our current science books will be looked at as works of fiction. The works of Einstein will be ridiculed in jokes. Dark matter will be equated with believing in magical invisible dragons that live in your garage. The Hadron collider will be a museum of stupid things scientists did.

$0.00

kyle.anderson (60) · 7 years ago

grrrr, all the references are in the form, well darwinism could have happened this way of that way. But no one ever questioned darwinism itself

Yeah not really true. Have any evidence against evolution?

In two generations, our current science books will be looked at as works of fiction. The works of Einstein will be ridiculed in jokes. Dark matter will be equated with believing in magical invisible dragons that live in your garage. The Hadron collider will be a museum of stupid things scientists did.

See this is where you took a reasonable critique of the journal industry and ran the trike off the road. We know the works of Einstein are accurate to essentially as much percision as we can fathom. He will not be laughed at, certainly not by anyone with any respect. Nobody is laughing at Newton even though he was totally wrong.

Really need to clarify your position if you really believe the things it seems.

$5.08

4 votes

lukestokes (75) · 7 years ago

ran the trike off the road.

HAhah... this may be my new favorite way to describe jumping to conclusions and creating stories which our brains enjoy.

$0.06

kyle.anderson (60) · 7 years ago

$0.00

builderofcastles (64) · 7 years ago (edited)

Sorry @kyle.anderson I meant what I said, and there is a huge amount of evidence backing it up.
But, none of it is peer reviewed... well, because it goes against everything the peer review / ivory towers believe in.

Einstein's theories of relativity are about 96% accurate in determining the orbit of Mercury.
There is a lady, published recently, her works on equal area in equal time, and her theories match the orbit of Mercury 99.9999%

Nobody is laughing at Newton even though he was totally wrong.

No one is laughing, but everyone either, doesn't know about Tesla, or says he was a shyster. Although our modern world is completely dependent on his inventions.

So, which do you believe? Tesla or Einstein?

When I say that science books are wrong, I mean they are going completely in the wrong direction.
As in, you can never understand electricity as long as you think it depends on an electron.
As in, you can't understand how light travels if you discard the luminiferous aether.

Really need to clarify your position if you really believe the things it seems.

I understand where you are coming from. I loved science. I have more college units then you do. In more subjects than you do. And then I started finding out where I was lied to in these college courses/ science books. I started finding that there was little evidence actually supporting "settled science" and found a huge amount of evidence debunking it. And then more evidence discarding it. And even more evidence showing that there is a conspiracy trying to keep mankind from actually knowing the world around them. (look into who owns the peer reviewed papers. It like the MSM, but worse. A handful of people get to decide who gets to be seen.)

So, when faced with this opposing information, I was skeptical, then confused, then angry.
I imagine you will be too.

$0.00

kyle.anderson (60) · 7 years ago

... sigh.

Sorry @kyle.anderson I meant what I said, and there is a huge amount of evidence backing it up.
But, none of it is peer reviewed... well, because it goes against everything the peer review / ivory towers believe in.

I don't care where it is located, show me whatever evidence you got.

So, which do you believe? Tesla or Einstein?

I did not know they were mutually exclusive. Also Tesla was a loonie.

As in, you can never understand electricity as long as you think it depends on an electron.
As in, you can't understand how light travels if you discard the luminiferous aether.

So um how do you propose describing electromagnetism? I don't think you understand how light works or what an electron even describes. No such thing as the luminiferous ether. Quantum feild theory and Relativity do remarkable jobs describing the physical universe - down to essentially the level of measurement. If you or anyone else found a better way - everyone is all ears.

I have more college units then you do. In more subjects than you do.

Lmao.. how do you know?
You clearly haven't taken enough mathematics and physics.

Go ahead, debunk the math.

You are right that some of the journals have their fair share of issues - the good thing: science does not care about the journals. It cares about evidence and reproducibility.

$4.14

4 votes

builderofcastles (64) · 7 years ago

I will get around to debunking the math, but it is hard.
You have been so programmed, and then had you mind sealed shut from the truth.
So, writing a piece that worms its way in through that sealed box is proving rather difficult.

But, some hints.
Why do we teach calculus immediately after algebra? As opposed to something really useful, such as discreet mathematics.
Why do we believe that you can always add one to a number? It goes against set theory.
Why can't we divide by zero?

If Tesla is correct, Einstein is wrong. But, you have to find and then read Tesla's works. And they have been buried and obscured. And further, to understand them, you basically have to unlearn all you learned in science class.

But hey, our entire modern world is based off of Tesla's work.
Einstein, on the other hand, did nothing for our modern world. Even the atomic bomb; there is evidence that he really didn't help.

And, maybe I should have said, statistically speaking I have more semester units then you do. That is factually true.

$0.00

kyle.anderson (60) · 7 years ago

.........

Well Tesla is wrong if that is what you are wondering. Like I said, he went pretty loonie in his older years. Wireless power xD.

Why do we teach calculus immediately after algebra? As opposed to something really useful, such as discreet mathematics.

... so calculus is not useful, got it. Can you name 3 things that involve change of any kind?

My mind is open, if you have anything to share to support your points I would love to be linked.

Sure most people my age don't have too many units - I give you that one.

If you have something logical to say please do, otherwise I am done wasting time with someone like you.

$0.00

builderofcastles (64) · 7 years ago

I am one of the few people that I know that knows what calculus is for and uses it. For everyone else, especially engineers, calculus is a waste of time. The formulas have already been worked out.

However, my statement is extremely meta.
If you aren't taught calculus, then the entire math is in danger of collapsing upon itself. Calculus adds a self reinforcing loop to the structure. So that any person who is decent at math doesn't immediately find its edges and start to ask the teacher embarrassing questions.

$0.00

kyle.anderson (60) · 7 years ago

I don't even know what to tell you.

Calculus is very important for any science feild.

$0.00

lukestokes (75) · 7 years ago

Looks interesting. How would one going about proving its existence without the scientific method?

You seem to be following a similar pattern I see in many who love conspiracy theories: change the subject. You bring up Tesla and Mercury and luminiferous aether instead of answering the very clear question that was asked about evolution.

Do you know Kyle personally to know his educational background? If not, you're making unsubstantiated claims right here for all to see. You very well may have more institutional education and you may have found some very important facts about reality the world can benefit from. From my perspective, the way you go about explaining yourself does not inspire critical thinkings to follow down your path.

$3.92

builderofcastles (64) · 7 years ago

I made the claim based on statistics. There indeed, is a small chance that I am wrong.

Yes, they are brought up by conspiracy theorist, because there is so much there that has been hidden and obscured.

Its like 9-11 truthers talking about free fall speeds. If you know anything about physics/engineering, then the statement "buildings fell free fall speeds thus explosives were used" is a factual statement. However, those for the "official" story will say, there those crazy conspiracy nuts go again talking about free fall speeds again.

In the same vein, I will bring up Tesla and Mercury because they are very big disproofs of Einstein.

My view on michelson-morley (how modern science doesn't believe in the aether)
https://steemit.com/science/@builderofcastles/the-michelson-morley-experiment-the-error-in-the-errors

My view on the scientific method
https://steemit.com/science/@builderofcastles/the-scientific-method-is-crap-it-is-a-mental-prison-for-scientists

$0.00

anacristinasilva (67) · 7 years ago

😲

$0.00

deanlogic (61) · 7 years ago

Sorry, I can't watch some videos where I am at. But, I noticed this in a comment below.

They can then choose to accept, accept with changes or decline. All of which comes with comments. If the paper is declined it is usually because either it doesn't fit the journal or it is not good.

If someone proposes an alternate theory that is not accepted by the journal, then you won't be published. Sometimes it turns in to a Catch 22. You can't be published unless you are part of the group. You can't be part of the group unless you are published.

This is an issue found when Ben Davidson tried to get a NASA scientist who refuted his research to review his research. He told him to get his alternate theory on the Sun and the connection to Earth quakes published and peer reviewed, before would look at it.

$0.00

lukestokes (75) · 7 years ago

Sometimes, but the example you gave makes perfect sense to me. Why would a NASA scientist waste their time with every crank (not saying Ben is a crank, I know nothing about his claims) they refuted? It would be like asking a car repairman to work on your airplane engine. Sure, it's related, but wouldn't it be better to submit to a journal who's livelihood and profession directly deal with reviewing scientific claims? That's what they do. That's the role of the editor.

$0.10

deanlogic (61) · 7 years ago

True.

In this particular case, the NASA scientist called out Ben and then refused to review his work.

$0.00

lukestokes (75) · 7 years ago

Does that matter? The burden of proof is on those who are willing to go through scientific peer review. Sounds to me Ben was given good advice. If he thinks his work is above reproach, it should be easy to get published.

$0.10

deanlogic (61) · 7 years ago

He tried to get it published. They told him he had to be peer reviewed first.

In the end he had to get peer reviewed outside of the established community and published that way. Not 6 months later, the established scientific community is releasing similar papers.

$0.00