I was recently in one of those meetings where academics decide what papers to include in a conference

this usually triggers thoughts of: how can we do this better?

my usual thought is: have people review more papers, but instead of writing reviews, just rank order them.. I feel like this would take less time/effort, and do a better job of deciding which papers to include

but I was talking to someone who suggested that the problem with peer review is peer review itself..

which I thought about..

the purpose of peer review, I think, is to determine who to give academic jobs to,
since the fruits of academia are typically not marketable in themselves,

so we're giving money to people to do "worthless" stuff,
but so many people claim to be good at doing "worthless" stuff,
that we need to decide which of them are right..

so I usually think — peer review may be bad, but how else would we decide which researchers are good?

but a couple recent thoughts:

1. the actual fruits of research — the best stuff — the stuff that actually makes an impact — is super super rare. like needles in a giant haystack.

but even a good researcher — even an excellent researcher — may never find a true needle — ever

but we still want the people with the best hope of finding a needle, so we require them to submit papers to conferences and journals that describe stuff that's needle-like.. the thought is that if people find lots of needle-like stuff, then the odds of them finding a true needle is higher.

but, the process of finding needle-like stuff takes away from the pursuit of finding needles, and ultimately selects for people who are good at finding needle-like stuff. The true needle seekers can't compete, because they spend too much time ignoring needle-like stuff in their pursuit of finding a true needle..

2. what, then, is the right way to finance research? one possibility — perhaps researchers can earn their keep doing something that is marketable, and then have no restrictions or expectations on their research time (which is their own time, using their own money). But don't we want researchers to be able to focus on their research full time? probably. But that's not what we have right now anyway. We have researchers focussing on finding needle-like things, to "earn" their keep by having their peers approve it.