Peer reviewing revisited: Assessing research with interlinked semantic comments

Cristina Iulia Bucur, Tobias Kuhn, Davide Ceolin

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Scientific publishing seems to be at a turning point. Its paradigm has stayed basically the same for 300 years but is now challenged by the increasing volume of articles that makes it very hard for scientists to stay up to date in their respective fields. In fact, many have pointed out serious flaws of current scientific publishing practices, including the lack of accuracy and efficiency of the reviewing process. To address some of these problems, we apply here the general principles of the Web and the Semantic Web to scientific publishing, focusing on the reviewing process. We want to determine if a fine-grained model of the scientific publishing workflow can help us make the reviewing processes better organized and more accurate, by ensuring that review comments are created with formal links and semantics from the start. Our contributions include a novel model called Linkflows that allows for such detailed and semantically rich representations of reviews and the reviewing processes. We evaluate our approach on a manually curated dataset from several recent Computer Science journals and conferences that come with open peer reviews. We gathered ground-truth data by contacting the original reviewers and asking them to categorize their own review comments according to our model. Comparing this ground truth to answers provided by model experts, peers, and automated techniques confirms that our approach of formally capturing the reviewers' intentions from the start prevents substantial discrepancies compared to when this information is later extracted from the plain-text comments. In general, our analysis shows that our model is well understood and easy to apply, and it revealed the semantic properties of such review comments.

Original languageEnglish
Title of host publicationK-CAP 2019
Subtitle of host publicationProceedings of the 10th International Conference on Knowledge Capture
PublisherAssociation for Computing Machinery, Inc
Pages179-187
Number of pages9
ISBN (Electronic)9781450370080
DOIs
Publication statusPublished - 23 Sep 2019
Event10th International Conference on Knowledge Capture, K-CAP 2019 - Marina Del Rey, United States
Duration: 19 Nov 201921 Nov 2019

Conference

Conference10th International Conference on Knowledge Capture, K-CAP 2019
CountryUnited States
CityMarina Del Rey
Period19/11/1921/11/19

Keywords

  • Linked data
  • Peer reviewing
  • Scientific publishing
  • Semantic web

Fingerprint Dive into the research topics of 'Peer reviewing revisited: Assessing research with interlinked semantic comments'. Together they form a unique fingerprint.

  • Cite this

    Bucur, C. I., Kuhn, T., & Ceolin, D. (2019). Peer reviewing revisited: Assessing research with interlinked semantic comments. In K-CAP 2019: Proceedings of the 10th International Conference on Knowledge Capture (pp. 179-187). Association for Computing Machinery, Inc. https://doi.org/10.1145/3360901.3364434