Last week I spent three days attending workshops devoted to defining article-level metrics/altmetrics. There was a lively Twitter stream, at #alm13 and #nisoalmi. The goal behind these efforts is two-fold: to diversify the range of metrics we use to evaluate traditional scholarship; and (more importantly in my view) to broaden the range of materials that "count" as legitimate scholarship, beyond papers and monographs. Data sets, Web-based visualizations, and theater performances are all worthy of equal standing within the scholarly fold.
As promised, here is my report. It begins with my understanding of the current state of play; thoughts about the relationship of blog posts and tweets to more traditional scholarship; a plea to broaden our conception of what we consider to be scholarly; some practical ideas for moving forward; and, in closing, a reminder of the (eternal) need for humility.
1. The State of Play
This was the second annual conference on this topic sponsored by PLoS, who have been leaders on the article-level metrics front for years. The meeting brought together all the players. In fact, Pete Binfield of PeerJ noted in his opening on Thursday that he sees the same faces at meetings every few months, and that the more established field of bibliometrics was missing from the altmetrics conversation. Not so this year--several PhD students presented their ongoing research, in which they are linking altmetric data (Twitter counts, downloads, etc) with more traditional citation measures as a means of gauging impact. So perhaps the more established and more avant garde evaluation appraoches are merging.
One sign of maturity is the involvement of NISO, the National Information Standards Organization. Among other things, NISO is responsible for ISBNs and ISSNs...which, respectively, facilitate the efficient sale of books and periodicals. Now NISO has an interest in formalizing--perhaps--additional approaches to scholarly evaluation that go beyond the impact factor method of counting citations. That "perhaps" is key...NISO may elect to issue best practices rather than formal standards, and is in the first phase of a 3 year project that will determine its path forward.
Some of my colleagues at the meeting felt that it was too early for NISO to be involved, and that a standards-setting body would only rigidify what should be an organic development process. I'll reserve my judgment until seeing the ultimate product of the initiative, and if that product is a set of prescriptive standards I too will be concerned. Today I feel it is a positive sign that NISO is getting involved. This shows that the concept of alternative metrics, which was firmly on the fringe just a few years ago, has gained a measure of stability and longevity. The PhD research is another sign that we've entered a new phase.
2. Blog Posts and Tweets
Amy Brand, Assistant Provost for Faculty Appointments and Information at Harvard, spoke about Harvard's effort to institute a more holistic and comprehensive faculty review process. Rather than something mechanistic and formulaic, Harvard strives for a multifaceted review process that makes allowance for more diverse expressions of scholarship. Brand described this process in a commentary in eLife early this year.
During her remarks Brand discussed the importance of distinguishing between "expert" reviewers and others...ie, all tweets and blog posts are not created equal. At first I bristled against this, thinking it was just another way to reify the traditional hierarchy between academic discourse and everything else. But then I realized that this will ever be so--there has always been a divider between academic and popular culture. The question is how to properly value products like blog posts and tweets--we should not willfully equate them with more sustained scholarly efforts, but we should also not dismiss them as mere piffle. I'll return to this in the "practical suggestions" part of this post.
3. A Bigger Tent
Before turning to those suggestions, a plea for scholars to pitch a bigger tent. Just as gay marriage does not threaten my straight marriage, a software developer who treats code as their "paper" does not threaten those who wish to publish in the traditional manner. Theater professors put on a show, they don't write papers that test measurable hypotheses. (I take this example from NISO Executive Director Todd Carpenter, who mentioned his wife's predicament as a theater professor in a system that does not have a ready means of valuing her work.) The world is capacious beyond all imagining, and the Web has infinute space. Nonetheless we have boiled down its acceptable scholarly artifacts to a mere handful of easily digested document types. Not only does this hurt theater professors; as Juan Pablo Alperin reminded us, the current scholarly publishing system favors North American and European scholars at the expense of everyone else.
We can do better. We should pitch a bigger tent, allowing "alt-products" into the fold. This was the message of Impact Story co-founder Heather Piwowar, who penned an excellent commentary earlier this year about the need to have altmetrics for alt-products.
4. Practical Suggestions
Beyond pitching a bigger tent, which is ultimatey an effort that requires cultural change more than a technological tweak, what are some things we can do now? I thought of two ideas, one of them inspired by Amy Brand and another an extension of that.
a. Contributor role taxonomy: Brand, as well as Todd Carpenter, noted that many papers are multi-authored. On a paper with many authors there is no meaningful information about the role of each of these authors--at least not in a standardized fashion that is readily understood. We can develop a taxonomy of easily understood contributor roles, which interacts with the ORCID system currently being developed to disambiguate author names.
b. Participant role taxonomy: Building on that idea, and looping back to the challenge of properly valuing blog posts and tweets, we can also build a participant role taxonomy that is similar to the contributor role taxomony. Yes, the random tweet or post should not get too much credit. But someone who uses these methods to contribute to a conversation in a sustained way should be recognized. They can use ORCID IDs too. Those really do only take 30 seconds to create, as I just discovered.
5. Humility
There were many ideas generated at the conference, and some participants went to a data challenge yesterday devoted to creating practical tools. I have great confidence that this conversation will bear fruit, resulting in a better means of creating, evaluating, and valuing scholarship.
That said, though, humility is a virtue. We will only ever understand so much, and we tend to over-value how much the tools we're familiar with actually explain. Todd Carpenter pointed out that a citation is in itself a static object; it gives us no information about why the researcher cited that particular work. Is it a positive citation? Negative? Just fodder for the lit review that wasn't actually read? Nobody can tell from the reference list. To really understand those questions we'd need to interact with the work and speak to all the actors involved, which of course is impractical on a large scale.
Any creation that is meant for widespread distribution flattens a 3 dimensional experience into a 2 dimensional form. This blog post is my best attempt to render 3 days worth of my experience, but it is inevitably partial and incomplete. And that would be the case for any other attendee. Let's be bold but modest, recognizing that the real triumph lies in framing the questions more than finding the answers.
Recent Comments