Dimension 1

Problems

To realise Dimension 1 (D1) of the Lag-Score pipeline, it must first be established that the underlying data quality is adequate for the intended use case. Before a bibliographic database can be adopted as the single source for the scientific layer, several methodological problems have to be identified and, where possible, empirically validated. This chapter outlines the five central problems that must be addressed prior to finalising the data source decision.

Data Coverage

No single bibliographic database covers all natural science disciplines to an equal extent. Gusenbauer's large-scale comparison of 56 databases demonstrates that disciplinary coverage varies substantially both within and across databases, with the largest database in a given subject frequently covering more than 30% more records than the second-largest alternative, and with gaps exceeding 50% in fields such as Environmental Science [1, p. 2706]. If the disciplinary coverage of the chosen database is not comparably balanced across the eight target disciplines of this project, differences in corpus size would be conflated with genuine differences in the temporal gap between scientific activity and market attention, thereby producing a Lag Score that reflects database artefacts rather than the underlying phenomenon.

This risk is not merely theoretical for open, publication-level indexes. Abramo, Cicero, and D'Angelo show that OpenAlex, compared with Web of Science (WoS) and Scopus, indexes a substantially larger and more heterogeneous publication portfolio, but that this additional coverage is not neutral: publications exclusive to OpenAlex are, on average, considerably less cited than the shared core set, and the average citation level of the aggregated OpenAlex portfolio is lower than that of WoS or Scopus [2, p. 3433]. This indicates that broader coverage in OpenAlex is unevenly distributed across venues and, by extension, potentially across disciplines, reinforcing the need to verify disciplinary balance empirically rather than assuming it from aggregate database size.

Classification Granularity and Accuracy

It is essential that the chosen database classifies works into disciplines with sufficient accuracy. If, for example, a query for Mathematics returns results in which fewer than 50% of the hits are actually attributable to Mathematics, every second result is effectively noise, and any subsequent measurement built on this classification would be unreliable. Gusenbauer formalises this concern through the concepts of keyword precision and recall, showing that classification accuracy is highly keyword- and subject-dependent: the keyword "branes," for instance, achieved 90.3% precision for Physics and Astronomy, while broadly used terms such as "evolution" proved unsuitable due to cross-disciplinary ambiguity [1, pp. 2699–2700]. This illustrates that classification accuracy cannot be assumed uniformly across a taxonomy and must instead be checked per discipline.

This concern is amplified for OpenAlex specifically, since it classifies works at the publication level using 4,516 topics grouped into 252 subfields, in contrast to the journal-level classification schemes used by WoS (254 categories) and Scopus (333 ASJC categories) [2, p. 3418]. While publication-level classification is in principle more precise than inheriting a journal's category, Abramo et al. document a concrete failure case in which a publication was assigned to the OpenAlex subfield "Museology" instead of "Business and International Management," resulting in a citation-normalisation factor (FWCI) of 286.0, compared to 33.7 in Scopus and a Category Normalised Citation Impact of 9 in WoS for the same publication [2, pp. 3429–3430]. Although this example originates from a citation-impact context rather than disciplinary coverage measurement, it demonstrates that fine-grained, text-based classification carries a real risk of misclassification that a coarser, journal-level scheme would not produce. Consequently, the accuracy of the topic classification used for each of the eight target disciplines must be validated after the fact, rather than assumed from the classification scheme's stated granularity.

Multi-Attribution

A single publication can be related to more than one topic or category, and the way this multiplicity is counted materially affects the resulting discipline size. Gusenbauer addresses this by relying on fractional, single-attribution counting rather than full multi-attribution counting, since the latter would artificially inflate the apparent coverage of highly specialised databases; the study reports subject-specific multi-attribution factors in Scopus ranging from 1.38 for Medicine to as high as 2.75 for Decision Sciences, against an average of 1.59 across all subjects [1, p. 2702]. Ignoring this variation and applying a uniform attribution assumption across disciplines would systematically distort the relative coverage estimates used for the Lag Score.

Abramo et al. further note a structural asymmetry between databases in this respect: in OpenAlex, the association between a publication and a subject category is one-to-one, whereas in WoS and Scopus this association is not necessarily unique, since journals classified under multiple categories/ASJC codes are typically assigned to a single "prevalent" label for evaluative purposes [2, p. 3430]. This difference means that multi-attribution behaviour cannot be assumed to transfer between databases and must be explicitly accounted for when the classification field of the chosen source (e.g., OpenAlex's primary_topic.field) is used for discipline delineation.

Preprint vs. Journal Dating

Different databases use different conventions for what counts as a publication's first issue date, which directly affects any measurement of a temporal gap. Abramo et al. document this explicitly: OpenAlex indexes publications based on the online-first date, i.e., the date associated with DOI registration, whereas WoS and Scopus generally use the source year, referring to the year of the journal issue in which the article is formally published, which often corresponds to the print version [2, p. 3418]. Since the Lag Score is, by construction, a measurement of a temporal gap, the dating convention adopted by the chosen database is not a peripheral metadata detail but a factor that directly shapes what is being measured. It is therefore necessary to identify which dating convention the chosen database applies and, where feasible, to validate it against an independent reference rather than treating the reported date as ground truth.

Metadata Accuracy

The chosen database must provide sufficiently accurate metadata, particularly with respect to abstracts, institutional information, and record identity, since these fields underpin both the classification process and the corpus's overall trustworthiness. Gusenbauer's validation methodology illustrates one general approach to testing this dimension, distinguishing internal validity (consistency of estimates across different access systems for the same underlying database) from external validity (accuracy of estimates against an independent control dataset) [1, pp. 2719, 2721]. A comparable, database-specific concern emerges from Abramo et al., who report an affiliation mismatch between official institutional records and the affiliations indicated in the bibliographic database itself: in Scopus, this occurred for 2.2% of the professors in their dataset, and in OpenAlex for 5.4% [2, p. 3425]. Beyond static metadata quality, the same study identifies a dynamic integrity issue specific to OpenAlex's work-based data model: upon re-extracting citation data approximately nine months after an initial extraction, nearly 12% of the original records appeared to be missing, not because they had been removed, but because they had been reassigned to different OpenAlex identifiers as preprints, early-access papers, or other provisional versions were merged into finalised journal article records [2, p. 3433]. This identifier instability has direct implications for any pipeline step that relies on persistent record identity, such as preprint-to-publication linking, and must therefore be accounted for through a re-harvesting or identifier-versioning strategy rather than assumed to be stable across time.

---

References

[1] M. Gusenbauer, "Search where you will find most: Comparing the disciplinary coverage of 56 bibliographic databases," Scientometrics, vol. 127, pp. 2683–2745, 2022, doi: 10.1007/s11192-022-04289-7.

[2] G. Abramo, T. Cicero, and C. A. D'Angelo, "Data source effects in research performance assessment of individuals and institutions: comparing OpenAlex with established bibliographic databases," Scientometrics, vol. 131, pp. 3411–3441, 2026, doi: 10.1007/s11192-026-05638-6.

Tools that where used to write the article: Claude Science, Claude Sonnet.