Towards Semantometrics: A New Semantic Similarity Based Measure for Assessing a Research Publication's Contribution
Petr Knoth, Drahomira Herrmannova
Full text: http://dx.doi.org/10.1045/november2014-knoth
We propose Semantometrics, a new class of metrics for evaluating research. As opposed to existing Bibliometrics, Webometrics, Altmetrics, etc., Semantometrics are not based on measuring the number of interactions in the scholarly communication network, but build on the premise that full-text is needed to assess the value of a publication. This paper presents the first Semantometric measure, which estimates the research contribution. We measure semantic similarity of publications connected in a citation network and use a simple formula to assess their contribution. We carry out a pilot study in which we test our approach on a small dataset and discuss the challenges in carrying out the analysis on existing citation datasets. The results suggest that semantic similarity measures can be utilised to provide meaningful information about the contribution of research papers that is not captured by traditional impact measures based purely on citations.
Ratings & reviews
This is a very good ideaRăzvan Valentin FlorianRăzvan ValentinFlorian
I think that semantic similarity-based measures similar to the prototype proposed here have a great potential for improving scientometrics.
However, the exact formula probably needs improvement (and this is expected for a first iteration of implementing this concept). The proposed formula of the contribution is proportional to the spread of cited publications and inversely proportional to the spread of citing publications. This reflects the stated hypothesis "that if a paper uses ideas from a narrow field, but has an impact on a very large field, it is a sign of the paper's contribution". However, when creating new concepts, a publication may cite papers from various fields (e.g., multiple fields in cognitive science, such as psychology, machine learning, cybernetics) but its application may be in a well defined field such as computational neuroscience. In this case the publication would cite from a large field and be cited in a relatively narrower field, but this, by itself, should not decrease the relevance of the publication.
I suggest that when assessing semantometric measures, and comparing them to bibliometric measures, they should use as a reference peer-provided ratings (Florian, 2015), because such measures should validated against what it is that they purport to measure and predict, which is expert evaluation by peers (Harnad, 2008).