USNews Scholarship Impact Issues

In spring 2017, I briefly discussed the problem with scholarship impact factor in law as a response to a recommendation by a law professor to create a rankings methodology based on Google Scholar citation.

Well it seems that USNews may have read that article because USNews is now asking each law school for the names and other details of its fall 2018 full-time tenured and tenure-track faculty. USNews plans to link the names of each individual law school's faculty to citations and publications that were published in the previous five years and are available in HeinOnline.

Using this data, HeinOnline will compile faculty scholarly impact indicators for each law school. This will include such measures as mean citations per faculty member, median citations per faculty member, and total number of publications.

Those measures will then be provided to USNews for use in eventually creating a comprehensive scholarly impact ranking.

Thanks to all of the folks who have opined on this issue. After reading and listening to many varying opinions on topic, here is a list of aggregated pros and cons*:

Pros:
  • USNews Quality Assessment scores may become more objective than the opaque peer assessment scores. 
  • Assist regional law schools that do not currently have a great peer assessment score. 
  • Incentive for faculty to publish. 
  • HeinOnline is the most robust collection of law reviews. 
Cons: 
  • Authors of books/book chapters are not represented in HeinOnline. 
  • Authors of articles in journals other than law reviews are not in HeinOnline. 
  • Issues with journals that are embargoed for 3-5 years in HeinOnline (ex. Journal of Empirical Legal Studies). 
    • Update on 2/25/19: A rep from HOL has indicated that embargoes will not affect citation metrics (see comment below). 
  • HeinOnline could potentially skew results by including minor pieces (forwards, introductions).
    • Update on 2/25/19: A rep from HOL has indicated that these minor pieces are not included the citation analysis (see comment below). 
  • Citation counts are reductionist and bias:
    • Current research being done that shows 80% of the publications in top 10 law reviews for 2018 are written by authors whose alma mater is one of those schools.
    • Embeds gender, class, and race issues. 
    • Strategic self-citing or not citing "rivals" could be used to up impact measure.
  • Could result in a change in publication topics, selected formats, and selected publication. 
  • High-profile emphasis on faculty scholarship could be detriment to focus on teaching. 
  • Citation counts will be a detriment to institutions with a high number of more junior faculty members because their scholarship hasn't had as much time to produce citations. 
  • USNews is using the bright line of tenure/tenure-track regardless of tenure classification or scholarly requirements.
    • How will it affect tenure lines that do not have as much of an emphasis on publishing?
    • Update on 2/25/19: USNews will still ask schools to list all tenure-stream faculty, but will also ask for their primary role to be identified (e.g., "doctrinal" or "clinical" or "legal research and writing"). USNews has not yet decided what to do with this information. 
Recommendations regarding authority control: 
  • Authors must claim/merge HeinOnline author profiles
  • Author profiles must be properly attributed with the correct metadata. 
    • Multiple forms of author names – issues with typos, misspellings, and errors in author names that make count inaccurate should be remedied. 
  • Authors should use the same email address across all platforms, create a Google Scholar profile, and an ORCiD (personalized ID number that is used for all scholarship) to ensure proper scholarship metrics. 
  • Explore the use of PlumX to aggregate scholarly profiles and altmetrics. 
Ultimately, the success of this project depends on law schools providing the data.

As has been said by others, it seems prudent to only provide the requested data on the condition that USNews creates an internal, unpublished sample report demonstrating its methodology.

This will allow the various stakeholders to provide much needed input.


*Individual ideas not attributed for sake of confidentiality

Comments

  1. Hi Jamie, I just wanted to confirm a couple of items from your post. I regards to embargoed content in HeinOnline, while the full-text is not available, the journals are still indexed and are used in our citation analytics, so embargoed content would count in the metrics.

    Also, with regards to forwards, introductions, table of contents, indexes, etc.., these types of sections are not included in our citation analysis. Only articles, notes, comments, reviews, etc..

    If you have any questions, just let me know.

    Thanks,
    Shane

    smarmion@wshein.com

    ReplyDelete
    Replies
    1. Thanks for the additional information, Shane. Is there a place where the methodology has been spelled out in greater detail?

      Delete
  2. Hi Jamie, it doesn't appear we offer these details in the current literature we offer on the tool but I will take this back to our team to update our HeinOnline knowledgebase with this information.

    Shane

    ReplyDelete
    Replies
    1. Thank you, Shane! We started working with Hein today to merge author profiles, and the company has been absolutely wonderful.

      Delete

Post a Comment

Popular posts from this blog

Proposed Change to ABA Standard 601: Written Assessment of Law Library Effectiveness

Creatively Harvesting Bluebook Data

The Information Business; The People Business