Live and recent posts from across the Fediverse tagged #researchevaluation, aggregated by home.social.
Yesterday, I read an interesting paper by Łukasz Remisiewicz in #Scientometrics, introducing the concept of "metric capital": academic indicators that researchers can accumulate and convert into positions, funding, promotions, and recognition.
:doi: https://doi.org/10.1007/s11192-026-05848-y
What I found particularly interesting is the concept of metrics–status decoupling, when strong bibliometric performance does not translate into academic standing.
My exchange with Ole Ellegaard continues in #Scientometrics. This time, the discussion moves to an interesting question: should every useful descriptive bibliometric analysis be published as a peer-reviewed research article, and where should we draw the line?
A useful new paper by Huang et al.: responsible scientometrics requires more than responsible indicators.
:doi: https://doi.org/10.1007/s11192-026-05796-7
Every #metric is shaped by choices made long before evaluation: #data sources, retrieval, cleaning, normalization, analysis, interpretation, and documentation. Responsible research assessment starts long before the assessment itself.
#ResearchEvaluation #ResearchIntegrity #Scientometrics #Bibliometrics #Impact
I presented some of my recent work at the IRIT Research Seminar – from strange bibliometric patterns and GenAI to research integrity and research climate.
📌 https://doi.org/10.13140/RG.2.2.31734.56647
The main question: finding something wrong is one thing. What happens next? Detection needs interpretation. Correction needs institutions. Prevention requires understanding the environment.
#ResearchIntegrity #Scientometrics #GenAI #ResearchEvaluation #bibliometrics
Photos of our poster from the 30th Annual International Conference on Science and Technology Indicators:
📄 Conference paper: https://doi.org/10.5281/zenodo.22670373
📌 Poster: https://doi.org/10.5281/zenodo.22670653
#STI2026 #GAIDeT #Scientometrics #ResearchEvaluation #GenerativeAI #OpenScience
Our GAIDeT poster is on display today at #STI2026 in Antwerp! We explore a simple question: what if information about generative AI use in research became structured, machine-readable metadata?
📄 Conference paper: https://doi.org/10.5281/zenodo.22670373
📌 Poster: https://doi.org/10.5281/zenodo.22670653
#STI2026 #Scientometrics #ResearchEvaluation #AI #OpenScience #GenAI #GenerativeAI
Tracking, Valuing and Evaluating the Diversity of HSS Research Outputs
This was originally a script for a video I recorded for a meeting of the Australian Deans and Directors of Creative Arts (DDCA) group. It is addressed to those who are in institutional positions who need to make judgements about the qualities of research outputs in Humanities and Social Sciences with a particular focus on outputs that are “non-traditional” (actually often very “traditional” in their field!) or otherwise “not easily measurable by a STEM audience”.
In my view there are two key roles for those specifically engaged in supporting those doing HSS research. The first is to advocate for financial systems that can support small and numerous research projects and the funders who want to support them. The second key role is to defend the diversity of research practices, outputs and impacts in HSS and to create a meaningful narrative around their qualities for institutional audiences.
I’m going to focus on the second here, but I wanted to link both to underline the point that what is of value in HSS research is precisely its diversity, and its difference from the homogeneity and scale of conventional STEM research and its outputs. There is not infrequently a greater diversity of research outputs from single HSS groups, centres and departments than there is from entire faculties of science, engineering and medicine.
That diversity is at the core of the value that HSS research and scholarship brings to the table.
Every social epistemology of knowledge from Latour to Longino via Merton, Ravetz, Kuhn and Fleck have at their heart the idea that the core is the negotiation of knowledge across boundaries, or between groups. This places diversity, of all kinds, at the heart of this process, and linked to it the institutional systems that keep it alive. Arguably it is the purpose of a university to support and bring into contact diverse approaches, experiences and ways of knowing.
The issue of course is that managing that diversity is a challenge. At the scale of a university, systems are a necessity. Consistency is needed and economies of scale are real. Particularly in HSS but across other disciplines as well, it is impossible for any leader to be across the detail of how good every specific body of work is. If you believe in the argument for diversity then this as it should be – anything else would be a sign of insufficient diversity. That doesn’t mean that leaders can’t be effective at telling the stories about why diverse research matters. Nor does it mean that they can’t make judgement calls on the distribution of resources. But it means those decisions are subjective and contextual. And that the evidence that supports them will be specific to each case.
But that doesn’t solve the problem either. So what are the pragmatic and principled approaches available to addressing this complexity? In my view the key lies in separating out the challenge into two parts, one relatively straightforward and immediately useful, the other more complex but I would argue ultimately tractable.
An important, but oddly overlooked distinction in research evaluation and particularly technology discussions is the distinction between tracking outputs and evaluating them. There is often a tendency to lump these together and to imagine for example, that if scholarly books were better tracked then that would necessarily mean there would be improved citation data for them. The homogeneity of STEM research outputs means that issues with data coverage and completeness are often conflated to the question “is it indexed” using the presence of a specific journal in a specific database both as a reliable marker of quality itself but also of confidence in the quality and relevance of generally accepted performance data.
So, although this is an oversimplification, in STEM the tracking of outputs is standardised and homogenous. You look it up in an index like Web of Science.
In HSS, and particularly for creative works, the situation is far more complex. Institutions generally have very poor data on the scope and volume of creative outputs, or other forms of (so-called) “non-traditional” outputs. Work done by Niamh Quigley, within the COKI team (Quigley, 2022) showed generally used recording systems (systems like Symplectic, Pure and the like) tended to be poor at capturing aspects of research outputs important to creative researchers. Further, there was a tendency for the information that was manually inputted by researchers (that they care enough to type in!) to be stripped away as it made its way through institutional systems to the data warehouses used for high level analysis.
From a systems perspective what is needed is systems that make it easy for creative practice researchers to register outputs, to flexibly record the evidence that they consider important about the qualities of these works, and to also make it easy for institutions in turn to gather that information up. Recent technical developments provide improved paths to making this happen.
The Open Research Contributor and ID (ORCID) system provides a mechanism for researchers to authoritatively identify themselves, and to record outputs they have contributed to. In 2025 ORCID have increased the number of work types (Petro, 2025) to include a range relevant in humanities (including images, video, designs, sound and other creative works) as well as improving the mapping of some other relevant work types including reports. A researcher can choose what works they make visible, and thereby designate as “research”. This is an important distinction for some creative workers. Institutions can then harvest those outputs without requiring further work from researchers.
ORCID doesn’t solve all the problems though. Badly implemented as an institutional system it is yet another place creative researchers have to battle through manually adding their work while colleagues in STEM subjects have it all done for them by journal publishers. Critical to making this work is a commitment from you and your teams that ORCID be the only place researchers have to make sure their work is available and pulling that information from ORCID is an institutional responsibility.
But that still leaves the manual input issue. There isn’t – currently – a complete solution for that (although one – or several – could be built) but there is at least a solution that offers some additional value for creative researchers. Zenodo is a large scale repository, managed by CERN (yes the particle accelerator in Switzerland CERN) that offers a place for anyone to deposit research outputs. It provides a safe, publicly accessible platform from which researchers can showcase their work. It even allows for dark deposit for those creative researchers who need to restrict access to the core of their work for artistic, ethical or commercial reasons.
Most importantly Zenodo provides DataCite DOIs and this means that – while you do have to manually put information into Zenodo – that information can automatically flow through to ORCID. One place to input the information and then it can flow to the other places it needs to be.
This flow of information should be the case for any repository that integrates with DataCite (or Crossref) to provide DOIs. Zenodo is large scale and flexible, but there can be an argument for specialist repositories. I am generally sceptical of the value of creating new local repositories, whether at institutions or nationally, but the key questions to ask of any repository are “does it integrate with DOI providers” and “does it provide the recording functionality creative research practitioners need”? That flexibility is key and Zenodo has some strengths here that can also help us to address, if not yet fully solve, the second problem.
Creative practice researchers know the reasons that their work is important, and can help provide evidence of that. But that evidence is highly diverse.
I know the pressures that institutional leaders face to articulate a comparative quantitative value of research outputs to the rest of the university. And I would argue that we nonetheless need to collectively hold the line that this makes no sense. If you value a diversity of research and researchers, you can’t protect that diversity by homogenising the way that value is measured.
But that doesn’t mean we can’t articulate value. It doesn’t mean we can’t compare how disparate research projects and outputs achieve specific goals. It just means, in the best tradition of HSS scholarship, that the evidence that we use is contextual. That it is diverse.
And here is how repositories like Zenodo can help. Where they support the creation of packages of work, researchers can bundle up the work itself with evidence of its value and impact. This will start unstructured, but over time, and in collaboration with research communities it will be possible to develop some standards. For specific cases. Where appropriate. This might mean giving some background on the gallery that invited an exhibition, the footfall or an audience survey. It might be details of the film festival that selected a documentary for screening, or audience numbers when it was syndicated or broadcast. It might be structured in a particular way to make it easier to process (or even generate). But the key is to build on systems that always allow for the flexibility of a narrative statement that allows for the flexibility that research statements already provide.
This workflow places researchers in control, gives them flexibility they need to provide access and to evidence the value of their work. It provides a single point of entry from which communities, institutions, and others can draw in the data to help track those outputs and collate the evidence of their value. It is not controlled by corporate interests but by academic communities. It is not subject to the funding whims of one national government but an international set of consortia.
Perhaps best of all, this is an area where the humanities and social sciences can lead. It solves a problem by putting researchers in control while supporting institutional needs. It helps and simplifies monitoring while offering a path to gathering more sophisticated evidence to tell better stories. These are not problems that are restricted to HSS, but you could argue that much of STEM evaluation has lost its way, focusing on numbers instead of substance, journal lists instead of journal content, impact factors instead of actual impact. This recontextualisation, and re-situation is where humanities and social sciences scholars, and leadership can show a way to better forms of evaluation.
References
Is Scopus really a curated bibliographic database?
A new Scientometrics study finds 2.7M records without author information, 500K+ records with unknown source titles, inconsistent document types, and even London recorded as an affiliation.
:doi: https://doi.org/10.1007/s11192-026-05778-9
A useful reminder: better research assessment starts with understanding and checking the data - not just pressing a button.
#Scopus #Bibliometrics #ResearchAssessment #Scientometrics #ResearchEvaluation
New in Journal of Informetrics: Myroslava Hladchenko examines how foreign co-affiliations shaped the research performance of Ukrainian universities and the National Academy of Sciences before and during Russia’s full-scale invasion.
An interesting paper by Lutz Bornmann and António Osório on incentives, competition, and undesirable practices in science.
The debate over research metrics is often framed too simply: metrics or no metrics? A better question is: what kind of behaviour does an evaluation system make rational? Good research assessment should make doing good science the rational choice.
:doi: https://doi.org/10.1038/s44319-026-00900-7
#ResearchEvaluation #ResearchAssessment #Scientometrics #ResearchIntegrity
My new paper is out in Research Evaluation. I introduce "assessment creep": a risk that, as research assessment moves beyond simplistic metrics, we keep adding legitimate new criteria without making clear how they should be prioritised or combined:
🔗 https://doi.org/10.1093/reseval/rvag050
The challenge for #DORA, #CoARA & research assessment reform is not complexity itself, but keeping evaluation coherent, transparent & accountable.
#ResearchEvaluation #ResearchPolicy #OpenScience #Bibliometrics
An interesting evolution from Mike Thelwall. A year ago, he was cautiously exploring whether LLMs could approximate expert research assessment. Today, he reports that ChatGPT-5 mini reached correlations of up to 0.905 with expert REF2021 quality assessments across some disciplines, using a dataset of more than 107,000 UK research articles.
:doi: https://doi.org/10.1515/jdis-2026-0058
#ResearchEvaluation #Scientometrics #ResearchAssessment #ChatGPT #LLM #AI #SciencePolicy
When did "local" become a synonym for "second-rate"? A thought-provoking new paper in #ResearchEvaluation argues that locality is a characteristic of research, not a measure of its quality.
:doi: https://doi.org/10.1093/reseval/rvag045
A timely reminder for everyone involved in research assessment.
#Scientometrics #OpenScience #ResearchPolicy #SciencePolicy #HigherEducation
Are university journals really "local"?
Our new preprint shows that national and institutional embeddedness are not the same. University journals may be strongly rooted in national research systems while publishing very few authors from their own universities.
https://doi.org/10.31235/osf.io/pmhdy_v1
#ScholarlyCommunication #UniversityJournals #ResearchEvaluation
Can a national research ranking claim to reward the "most productive researchers" while capping the value of additional scientific output?
I looked closely at the formula behind Ukraine's new government-funded ranking. My conclusion: it optimizes for something quite different. Full explanation on LinkedIn 👇
#ResearchEvaluation #ResearchAssessment #SciencePolicy #OpenScience #CoARA #DORA
Analysing more than 30K cell biology papers 📄, the authors found that after performance-based evaluation systems began emphasizing first and corresponding authorship, the number of co-first and co-corresponding authors increased substantially.
:doi: https://doi.org/10.1016/j.joi.2026.101838
Interestingly, team sizes did not grow in the same way = credit was simply redistributed among more researchers.
#Scientometrics #ResearchEvaluation #Authorship #AcademicPublishing #ResearchPolicy
Most university rankings focus on Harvard, Oxford, and a handful of global stars. A recent paper takes a different approach - compares entire #HigherEducation systems:
📄 https://doi.org/10.1016/j.joi.2026.101839
One interesting finding is that countries such as the UK, Spain, and Italy show a more evenly distributed level of institutional performance, whereas systems like the US and China rely more heavily on a smaller group of elite universities.
Grateful to #VUIAS for the opportunity to present my workshop "Ranked but Not Heard: Metrics and the Position of Ukrainian Scholarship" during the Academic and Public Positioning for Ukrainian Scholars workshop in Vienna.
The presentation explored academic visibility, scientometrics, open scholarly infrastructures, and the challenges Ukrainian researchers face within global systems of research evaluation.
A recent Journal of Informetrics study shows – There is no universal number of “too many authors.”
In some fields, 3–6 may already be unusual.
In medicine – dozens are common.
In physics – large teams are often the norm.
:doi: https://doi.org/10.1016/j.joi.2026.101803
Yes, #hyperauthorship can signal problems (e.g., honorary authorship, metric inflation). But the key question is not “how many authors?” 👉 it is: Is this abnormal for this field and time?
We evaluate science mostly through papers. But researchers report that up to 75% of project effort is data work — collecting, cleaning, documenting, and preparing datasets. A reminder that research outputs ≠ research work.
New paper in Research Evaluation: https://doi.org/10.1093/reseval/rvag008
#ResponsibleMetrics #OpenScience #DataCitation #ResearchEvaluation
Most research evaluation still rewards papers, not the work that makes them possible. Yet researchers say up to 75% of a project can be data work: collecting, cleaning, curating, documenting.
:doi: https://doi.org/10.1093/reseval/rvag008
Maybe it's time to stop pretending that publications alone represent research.
#OpenScience #ResearchEvaluation #DataCitation #ResponsibleMetrics #Scientometrics
New paper in Research Evaluation explores how researchers actually cite data. Key insight: data citations are far more complex than simple indicators of data reuse.
:oa: https://doi.org/10.1093/reseval/rvag008
They reflect scientific practice, community norms, attribution, and even reputation-building. A timely reminder: metrics alone cannot capture the real value of data work.
#OpenScience #DataCitation #ResearchEvaluation #ResponsibleMetrics #Scientometrics
"Many of the loudest Open Science advocates are deeply embedded in the very systems they critique such as traditional publishing, prestige-driven academia and grant-dependent research cultures. They speak the language of reform while continuing to “play the game” remarkably well. Researchers who sit on advisory boards talk about preprints but then celebrate publishing their latest Nature paper"
https://www.themodernpeer.com/people-the-problem-in-open-science/
#OpenScience #OpenData #ScienceReform #Metascience #ResearchEvaluation #UniversityRankings #PublishOrPerish
A new article by Chloe Patton shows how debates about #OpenScience often slip into absurdity – like demanding #replication from the #Humanities. You can’t replicate history, culture, or interpretation the way you replicate a physics experiment. It’s a different kind of knowledge.
:doi: https://doi.org/10.1093/reseval/rvaf052
Forcing STEM-style standards onto the humanities doesn’t improve #science – it just adds bureaucracy and limits academic freedom.
Today at my alma mater, I spoke about how research evaluation is quietly shifting from citations to ChatGPT-style predictions.
👉 https://doi.org/10.13140/RG.2.2.30585.12642
AI can already “detect quality” from text alone, and sometimes performs better than classic metrics. But it doesn’t evaluate science: it rewards what sounds like good science. We may be heading from “publish or perish” to the new absurdity: “write ChatGPT-friendly or perish.”
#AI #ChatGPT #ResearchEvaluation #Scientometrics #LLM #OpenScience
The recent debate in JoI highlights a key issue often ignored in research evaluation - the impact of document types on citation indicators:
:doi: https://doi.org/10.1016/j.joi.2025.101738
When all publication types are counted, normalized metrics become inconsistent and misleading. But once we restrict the analysis to articles and reviews, correlations rise sharply, and results become robust and reproducible.
#ResearchEvaluation #Bibliometrics #SciencePolicy #Ukraine #Metrics
I currently have about a dozen papers under review. Now imagine: a drone hits my window — and who will keep emailing editors and reviewers then? 😅
Stewart Manley published his brilliant idea in #ResearchEvaluation the “exclusive option”. Authors could submit to multiple journals at once, and interested editors request an exclusive right to review.
:doi: https://doi.org/10.1093/reseval/rvaf027
No duplicated #peerreview. No endless delays. This could shake up academic publishing!
Honored to receive an Award of Appreciation from the Ministry of Education and Science of Ukraine for my contribution to the evaluation of research projects. Proud to stand with Ukrainian science.
#UkraineScience #ResearchEvaluation #ScienceForUkraine #OpenScience #PeerReview #DistributedPeerReview
A new study by Liu & Wang (2025) uncovers a shocking issue: millions of Scopus records have affiliation addresses but are labelled as "country-undefined."
:doi: https://doi.org/10.1002/asi.25011
Why does "USA" ≠ "United States"? How can "United Kingdom" get lost? A critical problem for research evaluation and policy decisions!
Time for Scopus to act! ✅
#Bibliometrics #Metadata #AcademicPublishing #ResearchEvaluation #DataQuality #Scientometrics
Most people still don't know: publishing in real scientific journals is usually free.
A new study shows how pseudo-journals in 🇮🇳 exploit researchers, offering "cheap" publications for just $25.
:doi: https://doi.org/10.1093/reseval/rvaf017
The authors of the #ResearchEvaluation study advise: carefully check open access models and journal fee policies. Education and critical thinking remain our main tools against academic fraud.
#OpenAccess #OpenScience #AcademicIntegrity #PredatoryJournals #OpenJournals
A brilliant new paper in #ResearchEvaluation 📄 compares how the UK, Norway, and Poland implemented research impact assessment - with very different results:
🇬🇧 UK built infrastructure, #REF became a strategic tool.
🇳🇴 Norway took a soft, formative approach.
🇵🇱 Poland copy-pasted, spent big - got confusion, “point-chasing” and no culture shift.
:doi: https://doi.org/10.1093/reseval/rvaf010
🇺🇦 A lesson Ukraine risks ignoring again.
#ImpactAssessment #SciencePolicy #ResearchImpact #AcademicCulture