Well, the thing is, though-- impact factor DOES mean something. What it means is that a discovery is as "important" as other scientists working in the area determine it to be. That is, a BIG-name in a field can publish something in a HUGE name journal, all right, but that doesn't make it trustworthy or of great significance. A really great study published by a person without any ties to a larger institution, in a virtual backwater of a journal, though... THAT paper might go on to earn lots of citations and become one of the "rocks" of any intro in the niche. KWIM?
Sort of the way that Pumphrey's paper has become. That can happen regardless of how well KNOWN a researcher happens to be at the time of publication.
The problem with evaluations of "expertise" and "trustworthiness" in selecting reviewers is that scientists are people-- and therefore, they automatically grant name-recognition with more trustworthiness than it really warrants, and they also have FRIENDS who are colleagues. Besides, it's already supposed to work this way-- and doesn't. Also-- what warrants more trust-- an article on vaccine safety penned by Jenny McCarthy? Or a physician from Omaha who happens to also have a PhD in immunology? I'd go with the latter pretty much every time, in spite of name recognition; that's a credentialing bias, I suppose, but I stand by it. Credentials DO indicate a certain basic level of expertise-- period-- or medical boards wouldn't require them in order to practice medicine. KWIM?
I've often thought that people who first step into a relative vacuum and are willing to be minor celebrities are granted a sort of magical level of credibility and authority as experts-- sometimes WAY more than is actually justifiable. So no matter how little they actually know, they can still be "the" authority over a long period of time-- just because they were FIRST. FAAN has had some of that going for it over the years.
The problem with reputation systems is that scientists are people first and vulcans second-- and reputation system validation invites publishing and grantsmanship to become popularity contests. NOT good. NOT good at all.