This week I've attended the "Gaming Metrics" conference at UC Davis. From the conference description: "Have we moved from 'publish or perish' to 'impact or perish'? If so, are metrics of evaluation now creating new incentives for misconduct? Are metrics also helping the evolution of forms of misconduct in specific and innovative directions? And, crucially, can we reliably draw a clear separation between gaming the metrics game and engaging in misconduct?"
Not surprisingly, we've only scratched the surface of these enormous questions. This topic ultimately hinges around the desire to advance in the academy and the incentives that drive behavior in doing so. Resisting gaming will be hard, given the human tendency to game any system that attempts to quantify progress and achievements. Nonetheless we have a continuing imperative to ensure that the results reported by researchers are as beyond reproach as possible.
Two of the speakers here, John Bohannon and Jeffrey Beall, believe they are doing this. In 2013 Bohannon wrote the infamous "sting" that exposed the lack of peer review at fly-by-night open access journals. Meanwhile Beall maintains his list of "predatory" open access publishers as another way to shed light on this same phenomenon. By their lights these are efforts to shine light on falsity, which is an unalloyed good.
The truth is more complex. Synecdoche is the literary device in which a part comes to stand for the whole (ie "Oakland" for the Oakland A's, or "No 10" for 10 Downing Street.) Bohannon and Beall are perpetuating an unfounded synecdoche, in which the actions of fringe characters in open access publishing come to stand for the entire idea.
The reason that fly-by-night open access operations have arisen is because author processing charges represent an irresistible source of cash to disreputable publishers, who are profiting on the intense need of researchers to publish. In other words -- the scholarly publication pressures that long pre-date the birth of open access are now being used as a means of discrediting open access.
This is rearguard retrenchment, which is why Bohannon's sting faced such fierce resistance from open access advocates.
Bohannon noted that when he wrote the sting he had no conception of the passions surrounding open access -- this feels very hard to believe, but there is no way to prove otherwise. (As a librarian, Beall presumably does know the passions surrounding open access and has consciously chosen to stake out a heterodox position. I am generally in favor of against-the-grain thinking, but in this case Beall's mission has assumed a worrisome evangelical fervor in its own right.) Even if we take Bohannon at his word, his article appeared in Science magazine. As a venerable subscription based publisher, Science had every interest in perpetuating the unfounded synecdoche that fly-by-night efforts represent the entirety of open access publishing.
This leads to another point -- there is no such thing as a neutral story. Or even a neutral comment posted anonymously. The online journal club PubPeer allows for anonymous comments about the validity of published studies. Anonymity has not led to a lack of civility -- the PubPeer team does an admirable job of insisting upon civility in the comment sections. After assuring this foundation, PubPeer allows the comments to rise or fall on their own merits. There is no attempt to understand the ideological or political motivations behind the comments, and obviously such a task would be Herculean. That granted, I was troubled by PubPeer founder Brandon Stell's professed view that understanding such motivations would not be a useful data point. We all bring our political commitments and personal predilections to anything we touch. Even if are not trying to "game" discourse, we can't help it.
What's to be done, then? Acknowledge that any numerically based ranking system will be gamed. Acknowledge that citation mills will continue, particularly if we maintain the antiquated idea that journal-based citation is the primary means of demonstrating impact. And advocate for the maximum of openness -- open publication, open notebooks, open data -- as the best way on offer to mitigate these ills. Gaming will run rampant whatever we do, but at least it should be out in the open.