Discussion Papers,
Norwegian School of Economics, Department of Business and Management Science

Abstract: This paper addresses grade comparability across exam cohorts when assessors and item sets change from year to year. Ordinal item scores reflect a mixture of student ability, item difficulty, and assessor severity; separating these components requires linking assumptions rarely verified empirically. We fit a sequence of Bayesian cumulative logit models to item-level scores from nine cohorts of an undergraduate statistics course. The setting is fully observational with no cross-grading and only partial assessor overlap, so cross-cohort alignment relies on repeated content used as anchors and on shared assessors. Sequential model expansion guided by posterior predictive checks reveals that treating anchors as having fixed difficulty across cohorts is inconsistent with the data. A soft-linking formulation, where linked items share a baseline difficulty but admit cohort-specific deviations regularised toward zero, removes the systematic misfit without discarding anchor information. Approximate cross-validation confirms that each modelling step improves out-of-sample predictive accuracy. Student ability estimates are robust to anchor specification (pairwise correlations exceeding 0.996), whereas cohort location estimates shift materially, which is a finding with direct consequences for grading policy. Using the recovered ability scale, we construct counterfactual grades and show that assessor severity is the dominant predictor of individual grade movement.

Keywords: comparable grading; many-facet measurement; soft anchors; cumulative logit model; Bayesian hierarchical models; PSIS-LOO cross-validation

JEL-codes: C11; C25; C52; I21; I23

Language: English

32 pages, May 22, 2026

Full text files

5518775  Full text

Download statistics

• Norwegian School of Economics, Department of Business and Management Science
• Ordering Working Papers
• Home page for this series

Questions (including download problems) about the papers in this series should be directed to Stein Fossen ()
Report other problems with accessing this service to Sune Karlsson ().

RePEc:hhs:nhhfms:2026_006This page generated on 2026-05-29 11:37:07.