Easily managing institutional lists of publications

Posted by on June 5, 2014

Scientific publications typically are the final result of the work pursued in basic research. Despite that research is an important mission for many universities and the core purpose of research institutes, many institutions are not able to showcase on their websites complete, up-to-date lists of publications authored by their scientists. Lists of publications are not typically available on the webpages of departments, neither. Individual scientists and laboratories typically have such lists on their webpages, but in most cases they are maintained manually, requiring tedious work for keeping them up-to-date. This is why a significant percentage of laboratory lists of publications are outdated. In a sample of labs that have recently received grants amounting to more than one million dollars from the US National Science Foundation, about 38% had outdated or no lists of publications, and among those that had lists that seemed to be up-to-date, about 73% appeared to be manually maintained.

Maintaining such publications lists is indeed quite cumbersome if appropriate software is not used. For each publication, the person maintaining the list must ensure that all information is in place, properly formatted; this means either tediously applying formatting and HTML tags, or filling with copy/paste lots of fields in some database that is later processed by a script for generating the formatted text and the links. If a scientist gets a new publication, some process should be implemented to ensure that the person maintaining the webpage is notified, gets all the details of the publication from the authors, and updates the webpage in a timely manner.

Many research institutions subscribe to databases such as Web of Science or Scopus, where the publications belonging to a particular institution can be searched. However, this is not a reliable way for an institution to collect the list of publications authored by its scientists, because: searching by the name of the institution yields incomplete results, as there typically are many variants of the name in these databases (up to hundreds of variants for institutions with thousands of publications); the institutional profiles in the databases are automatically generated and also have errors, such as including publications not authored by scientists within the institution; and the databases are limited in coverage and do not include all publications authored by the scientists. The providers of these two databases, Thomson Reuters and Scopus, also sell specialized software for institutions to collect validated information about their publication lists, but their systems are quite complex and expensive. They typically require several months for being set up and less than 1% of worldwide universities currently afford them. Being so complex, they are not suitable for smaller units such as laboratories, individual departments or small research institutes.

This is why we have developed Epistemio Outcomes as a service that allows units of all sizes (from laboratories to universities and even national research systems) to easily collect and manage the lists of publications authored by their scientists, while being able to be set up in as little as 24 hours, and also affordable to a wide range of institutions, including ones in emerging countries.

Epistemio Outcomes is based upon our database that includes more than 56 million publications, i.e. of about the same size as Web of Science or Scopus. This means that most publications to be included by someone on their list are already in our database, and no tedious manual typing or copying/pasting is necessary when adding most new publications. This is a major advantage over solutions developed in-house by universities or institutes, which typically lack such a database. Maintaining such a database requires significant effort and computational resources, much larger than what is possible or reasonable for a university or institute to allocate. But such a database is required if we want a system that does not waste the time of scientists and administrators by requiring them to fill manually the details of publications. This is why adopting a commercial solution based upon a database, such as Outcomes, is an optimal way for an institution to manage its publications.

However, it is not technically feasible today to associate publications with institutions and their sub-units entirely automatically, so the stakeholders must confirm that the found publications really belong to the scientists within the units using the service. We expect that, in most cases, scientists themselves will confirm their publications. With our service, they are motivated to do so because by doing it they also update their list of publications for their own needs, such having it available for being exported to a CV or for updating their personal web page, if they use our embedding feature. Confirming publications can also be delegated to administrative personnel.

When setting up Epistemio Outcomes, the administrator can define the organizational structure of the institution (e.g., departments, labs, etc.) and add other administrators for the various sub-units, as needed. The administrators can invite scientists belonging to the institution to log on to Epistemio. Scientists log in to Epistemio and confirm the publications found by our intelligent search. Administrators may also add or edit publications, if the scientists are too busy to do it themselves. Publications are automatically aggregated upwards in the organizational hierarchy, with automated deduplication. The aggregated lists of publications, for all units within the institution, are available for being embedded on the web pages of those units (code must be added on web pages just once, and thereafter the embedding will ensure that the lists stay up-to-date as scientists add new publications). The lists can also be exported at anytime, e.g. for including them in annual reports.

When a scientist writes a new publication, the scientist may go to Epistemio, click on a checkbox to confirm the newly found publication, and the list of publications of all units to which the scientist belongs (research group, laboratory, department, institute, university), as well as the individual list of the scientist, will be immediately updated. This would also immediately and automatically update all web pages these units on which the Epistemio-managed publication lists have been embedded, with no extra involvement of any webmaster.

The service is suitable for units of all sizes, from single labs to national research systems. Even large collaborative projects, such as the European Horizon 2020 ones, could use this system for easily managing the project’s publications webpage.

The system can also serve as a simple, lightweight institutional repository. Epistemio Outcomes allows authors to add links to pre-prints/post-prints/archived PDFs. One of the simplest ways to publish such a PDF is to place it in a Dropbox folder, ask Dropbox to create a link for it, and then add the link to the publication on Epistemio. This simple method can rapidly enable an institution to create a repository of its publications.

Start using Epistemio Outcomes

Talking at the conference “Technologies Transforming Research Assessment”

Posted by on March 13, 2014

Next week I will talk about research assessment in Romania at the conference “Technologies Transforming Research Assessment“, organized at the Parliament of Lithuania.

Here is the abstract:

In 2011, the higher education and research systems of Romania undertook major reforms that were praised by the European Commission and led to what Nature editorialists characterized as “exemplary laws and structures for science”. Research assessment was a key focus of these reforms. This included: introducing a habilitation process, for evaluating an individual’s research achievements in order to become eligible to apply for full professorship jobs in the universities; minimal scientometric standards for an individual’s eligibility for the various levels of faculty jobs in universities, and for the eligibility for submitting grant applications for the major research funding programmes; the assessment of grant applications, which started to use mostly foreign reviewers; a national assessment exercise for the classification of universities and for the ranking of the universities’ study programmes, for which research was a major component; and a national assessment of research institutions. I present the background and the constraints that led to the design of these research assessment processes, and I discuss the choices that have been made. I also discuss some new tools and processes for research assessment that were designed to solve some technical problems encountered during these processes.

To some extent, Epistemio’s features were informed by the issues encountered during the 2011 reforms of the higher education and research systems of Romania, when I have been an adviser to the minister of education and research. For example, the data about scientific publications submitted by universities for the national assessment exercise included many errors, to such an extent that the ministry had to request a re-submission of data. This happened because universities were lacking suitable research information systems to allow them to have accurate information about their scientific publications. Epistemio Outcomes, that we launched in 2014, solves this problem and helps universities to easily aggregate their lists of publications authored by their scientists.

The minimal standards that were introduced in 2011 in Romania were much discussed, and an important issue was to find suitable standards for scientific domains where citations-based metrics, such as the article influence score, were not available or not applicable. Such domains are computer science and, to some extent, some areas of engineering where conferences, rather than journals, are the main vehicle, or an important one, for publishing original research results; and humanities and some social sciences, where there are few citations, and books are the main vehicle of publication, or an important one. The research councils that designed the minimal standards spent much time trying to find suitable equivalents, for these domains, of the article influence score that was used for natural sciences. For example, because there was no article influence score for conferences, an equivalent has been established by using a classification into three categories established by the Australian Research Council. Because there was no citation information for books, an equivalent has been established by the National Research Council by counting the number of WorldCat libraries where the books were available.

The problem of establishing ad-hoc equivalents between inherently distinct metrics would not have appeared if there would have been available a common metric for all types of publications. An obvious common metric are the ratings given by peers. Peer review is the foundation of assessment in science, and metrics based directly on peer review are likely to be much more relevant than any other types of scientometric indicators that are just weakly connected, through proxy intermediaries, to peer review. This is why Epistemio aims to aggregate ratings and reviews provided by peers, especially by those who read anyhow the publications to be rated, for their own research.