Reference Integrity Is an Institutional Risk — and It Is Finally Measurable

Every university has a quality stack for doctoral work. There is a supervisor, an examination panel, a similarity checker, and a set of submission regulations. Between them they cover originality, argument, method and presentation. There is one dimension almost none of them measure: whether the references actually hold together.

Not whether the thesis is plagiarised. Not whether the sources are good. Something far more basic — whether every in-text citation resolves to an entry in the reference list, and whether every entry in that list is actually cited in the text. It sounds trivial. In a 300-page thesis with 250 references accumulated over four years, across chapter drafts merged and re-merged, it is not trivial at all. And because nothing in the standard quality stack looks at it, problems surface at the worst possible moment: in the viva, in the hands of an external examiner reading closely.

Four things go wrong, and only two are visible

When you pair a document’s in-text citations against its own reference list, every citation lands in one of four states.

• Matched — the citation and a reference agree. This is the healthy majority.
• Questionable — a pair resting on weak evidence. A bare year marker, two different citation styles in the same document, a contested reference. The pair may well be correct, but it is not resting on anything solid.
• Missing — cited in the text, absent from the reference list. The reader is pointed at a source that the document never identifies.
• Orphan — listed in the bibliography, never cited anywhere in the text.

A careful examiner will eventually catch a missing reference, because the citation makes a claim and offers no way to check it. Almost nobody catches orphans, because an uncited entry in a bibliography looks exactly like a cited one. And questionable pairs are invisible by construction — they look matched until somebody interrogates the evidence underneath.

A measure, not an accusation

This is the point institutions most need to understand before adopting anything in this space, so it is worth stating plainly.

The Ref[In] Score — short for REFerence [IN]tegrity — is a referencing-quality measure. It reports whether each citation and each reference found its counterpart in this document. It is not a similarity score, and it is not a finding of misconduct.

That sentence is printed on the cover of every report, not buried in documentation, and the reason is institutional rather than cosmetic. A similarity percentage carries an implication — a high number invites suspicion. A referencing-quality percentage carries a different one: this document needs another pass before assessment. Confusing the two would be damaging in both directions. It would put students under suspicion for what is usually administrative drift across drafts, and it would let institutions treat a referencing tool as an integrity investigation, which it is not.

A document can be entirely honest and score badly. That is the normal case. Four years of drafting, chapters merged, a reference manager that fell out of sync somewhere around the second year — none of that is misconduct, and all of it produces gaps.

How the number is built

A score that governs decisions has to be legible, so the arithmetic is published rather than proprietary. It has two halves, and it is weighted deliberately.

• Citation resolution — the share of in-text citations that resolve to an entry in the reference list. This counts triple.
• Reference utilisation — the share of reference-list entries that are actually cited. This counts once.

Nothing else moves the number. The weighting encodes a judgement worth making explicit: a citation with no reference makes a claim the reader cannot verify, while an uncited entry in a bibliography is untidy but harmless. One is a broken promise to the reader; the other is a loose end. Weighting them equally would flatter documents with clean bibliographies and unverifiable claims.

Evidence is reported, not penalised

Shared identifiers such as DOIs, citation styles in use, and reference completeness are all reported in the audit — but they are reported as evidence, and they do not move the score. This is a deliberate design decision with direct consequences for fairness across a university.

Consider a law faculty, or a humanities department working largely with primary archival sources and monographs. DOIs are rare in both. If identifier coverage fed the score, entire disciplines would be marked down for the publishing conventions of their field rather than for anything about the quality of their referencing. A thesis in a field that rarely publishes DOIs is not a worse-referenced thesis. Because completeness is reported as evidence rather than scored, one instrument works across a whole institution without quietly disadvantaging its non-STEM departments.

A fixed scale is what makes it administratively usable

The bands run from Excellent down through Very good, Good, Minor gaps, Attention needed, Major gaps and Substantial gaps to Critical. The important property is not the labels — it is that the scale is fixed. It does not adapt to the document, the cohort or the discipline.

That is what makes the instrument usable for administration rather than only for individual feedback. Two reports compare directly. The same thesis at two points in its life compares directly. A department can look at a distribution across a cohort and read it as a distribution, because every document was measured against the same ruler. An adaptive scale would make each report locally sensible and institutionally meaningless.

Three levels, and what each one certifies

The same report is issued at three levels, and the distinction matters for anyone designing a submission policy.

• Level 1 · Organising — the deterministic audit, with citations and references not yet found, dismissed, replaced or edited.
• Level 2 · Reference checking — after missing references have been found, dismissed, replaced or edited.
• Level 3 · Final export — in-text citations and references list updated and validated.

The level records how far the work had gone when the report was produced — not how good it is. It is independent of the score, and any level can come out green, amber or red. A Level 3 report can still be red. A Level 1 report can be green if the document arrived in good order.

For an institution this separation is the useful part. A submission requirement can ask for a Level 3 report — evidence that the work was actually carried through — without that requirement becoming a demand for a particular score, which would create precisely the wrong incentive.

What the supervisor actually receives

The report is available as a PDF download or a shareable link, and the link stays live for a supervisor after the student’s organising session has closed. That detail matters more than it first appears: supervision conversations rarely happen at the same moment as the work, and a report that expires with the session is a report that arrives too late to discuss.

Alongside the score, the activity summary states every metric of the run in a single table — citations extracted, reference-list entries, matched, missing, orphaned, cited-but-unpaired, unresolved alignments, AI-advisory proposals, questionable matches, pairs matched on a shared identifier, pairs written in incompatible styles, and chaff screened at extraction. A supervisor does not have to take the headline number on trust; the composition behind it is on the page.

Honesty about what was not checked

One design choice deserves particular attention from anyone evaluating tools for institutional use. Every run closes with a conservation ledger, reconciling what entered the process against what left it. Where the review accounting audit has not run, the ledger reports itself as unaudited — never as balanced.

It would be easy, and much more flattering, to show a green tick and stay quiet about an unrun check. Refusing to do that is the difference between an instrument you can put in front of an examination board and a dashboard that makes people feel good. The same principle runs through the rest of the system: nothing is discarded silently, and an unplaced date is screened into an unresolved bucket with its reason attached rather than dropped.

Where the AI sits

Cross-checking is deterministic and free, and can be re-run as often as you like. No model decides whether a citation matches a reference. AI appears in an advisory role only — tidying messy in-text forms, proposing corrections on weak pairs, helping locate references that are genuinely missing — and every suggestion is approved or dismissed by a human. Nothing is rewritten in the thesis automatically.

For procurement and research-ethics review, that boundary is usually the first question asked and the one on which adoption turns. Client and institutional content is never used to train models, and data-processing agreements are available for institutions.

Where to start

Most institutions begin with a single cohort rather than a policy change — a department runs its current submissions through and looks at the distribution. That is usually enough to establish whether reference integrity is a live problem locally, and the answer varies considerably between disciplines.

• Walk the whole workflow in the product tour, or read the full CitationLab Product Guide (PDF).
• See how the report reads from the other side of the supervision relationship in the Ref[In] report for supervisors.
• Review institutional terms and credit pricing on the pricing page, or check the FAQ.
• For data-processing agreements or a departmental pilot, talk to us directly.

Reference integrity has been unmeasured for so long that it is easy to assume it is unmeasurable. It is not. It is simply arithmetic that nobody was doing — and once it is a number on a fixed scale, it becomes something an institution can actually manage.