If I could recreate the way research results are quality checked and revealed to the world, I would probably change almost all of what is currently done. I think the isolated scientific paper is a product of the 20th century, being imposed on the 21st purely because of inertia. A better solution would be to give a "living paper" to each general research project an individual researcher has. This living paper can then be updated as results change/improve. In such a system I would probably have ~5 living papers so far in my career, instead of ~20 old-style papers. Or, even better, would be a large wiki edited, annotated, moderated and discussed by the science community as knowledge is gained.
Even if you to wish to keep "the paper" as how science is presented, I think that the journal system, while invaluable in the 20th century, also exists in the 21st century only due to inertia. Pre-print servers like the arXiv are already taking care of the distribution of the papers, and the peer review, which is responsible for the quality check side of things, can (and might?) be organised collectively by the community on top of that. But why should we stick with peer review anyway? Could there be a better way?
Firstly, let me stress, peer review is definitely an incredibly effective way to progress knowledge accurately and rapidly. The best ideas are the ones that withstand scrutiny. The better an idea is, the more scrutiny it can withstand. Therefore, holding every idea up to as much scrutiny as possible is the best way to proceed. However, by peer review I simply mean criticism and discussion by the rest of the scientific community. I think the way peer review is currently done, at least what people normally mean by "peer review" is very nearly worthless (and when you factor in the time required to review and respond to review, as well as the money spent facilitating it I'd be tempted to claim that it has a negative impact on research overall). The real peer review is what happens in informal discussions: via emails, at conferences, over coffee, in the corridor, on facebook, in other papers, etc. The main benefit the current method of peer review has is simply that the threat of peer review forces people to work harder to write good papers. If you removed that threat, without replacing it with something else, then over time people would get lazy and paper quality would degrade, probably quite a lot.
But that would only happen if the 20th century form of peer review was removed without replacing it with something from the 21st century. I wrote above that the real form of peer review happens through conversations at conferences, in emails, etc. The rapid access to papers that we get now makes this possible. In the early-mid 20th century, because the (expensive) telephone was the only way to rapidly communicate with anyone outside your own institute, word of mouth would spread slowly. Therefore some a priori tick was needed, that confirmed the quality of a paper, before it was distributed; hence peer review. But now communication can and does happen much more rapidly. Today, if a paper in your field is good, people talk about it. This gets discussed in emails amongst collaborators, which then disperses into departmental journal clubs and the information about the quality of the paper is disseminated like that. It's worth emphasising that, at least in high energy physics and cosmology, this often happens long before the paper is technically "published" via the slow, conventional peer-review.
However, this information probably still doesn't disseminate as widely or as quickly as might be ideal, given the tools of the web today. What would be ideal is to find a way for the discussions that do happen to be immediately visible somewhere. For example, what if, instead of having an anonymous reviewer write a review that only the paper's authors and journal editor ever sees, there was instead a facility for public review (either anonymous or not), visible at the same site where the paper exists, where the authors' replies are also visible, and where other interested people can add their views? The threat of peer review would still be there. If a paper was not written with care, people could add this in a review. This review would remain unless or until the paper was revised. Moreover, negative reviews that would hold up a paper could also be publicly seen. Then, if a reviewer makes unfair criticisms, or misunderstands a paper, the authors could makes this clear and the readers can judge who is correct. Or, even better, the readers can add to the discussion and perhaps enlighten both the authors and the reviewer (with words that all other readers can see)!
One way to achieve this would be to add comments/annotations to the arXiv. For various reasons, the people at arXiv are reluctant to do this. I can empathise with this. ArXiv is probably one of the best things to have happened to the high energy and astrophysics communities (who use it the most) because it gives access to any paper as soon as submitted, without charging the reader, or delaying the access in any way. I am happy that they want to focus on being able to continue to provide this service well.
But this doesn't mean that the service isn't desired. And, now, in fact, something very near to this does exist. It is called Hypothesis and it is a web annotation tool. Essentially, it is a browser plugin that allows you to read and write annotations anywhere on the web. So, if you have the plugin installed and write an annotation on a paper at the arXiv, then I can read it (or vice versa - note that you do need the plugin installed to see annotations). It seems to work very well.
Unfortunately, I don't think in its current form Hypothesis could replace peer review, even if the inertia problem could be overcome. For such a system to work would require a critical mass of people using it before it becomes effective. At present, an annotation is either visible to just the author, or to anyone. If annotations could be restricted to sub-groups then people will be more inclined to write annotations. Then, the particular annotations that the group (e.g. a research group at a university) finds most useful can be made more publicly available, if desired. Also, the ability to be notified (e.g. via email) whenever annotations are written on specific webpages, or websites would be needed. At present I can only see the option to be notified when someone replies to my own annotation. This means that if an annotation is written on a paper at the arXiv, then nobody else knows until someone specifically chooses to look at that paper, meaning most annotations will lie unread for a long time; a time unbounded from above.
Once such features are in place I think it would provide a good working model for a 21st century peer review system. Unfortunately the inertia behind the 20th century system is so large that I don't hold out a huge amount of hope that change will occur (people will do what the funding requires and so long as some major funding sources judge based on "published papers" people will submit to journals). Such a change might therefore require top-down policy change by funding agencies themselves.
In any case, some (20th century) scientists don't even think we would benefit from doing away with 20th century peer review! So opposition is from more than just inertia. Still, you should install the plugin and have a play with its features.
What are your thoughts? Is the current peer review system sub-optimal, would an annotation system be better? If not, why not?