Why a Citation Checker Should Not Be an LLM

There is an obvious way to build a citation checker in 2026. Take the thesis, take the reference list, hand both to a capable language model, and ask it which citations resolve. It would take an afternoon to prototype and it would demo beautifully.

It would also be the wrong instrument — not because language models are bad, but because of what an audit is for. This is worth setting out properly, because the distinction determines whether a tool’s output can be put in front of an examination board or merely looked at.

An audit that changes its mind is not an audit

Start with reproducibility. If a student runs a check on Tuesday and gets 68%, and a supervisor runs the same check on the same file on Thursday and gets 71%, the number means nothing. Neither party can act on it. The disagreement is not evidence about the thesis — it is noise from the instrument.

Language models are probabilistic by construction. Even pinned to temperature zero, behaviour shifts across model versions, context arrangements and prompt phrasing. For creative and analytical work that variability is often a feature. For an audit it is disqualifying. An audit’s entire authority rests on the claim that anyone re-running it on the same input gets the same answer.

So in CitationLab, the matching is deterministic. Every citation is paired against the reference list on a fixed backbone, applied in a fixed order:

1. Shared identifiers — DOI, PMID, legal instrument or standard number.
2. Exact author-year alignment.
3. The reference line as written.

No model decides whether a citation matches a reference. That is why the cross-check can be free and re-run as often as you like: it is a rule, not a judgement, and rules are cheap to apply and stable across runs.

The specific failure that matters most

There is a sharper problem than variance. Language models generate plausible text, and a plausible-looking reference is exactly the thing they are good at producing. Fabricated citations are now a well-documented failure mode — correctly formatted, authors who exist, a journal that exists, and a paper that does not.

Now consider what happens if that capability sits at the centre of a checker. A student uploads a thesis with a genuinely missing reference. The model, asked to resolve it, produces a clean, complete, entirely fictional entry. The student, who came to the tool precisely because they could not verify things themselves, accepts it. The thesis now contains a fabricated reference that it did not contain before, introduced by the software bought to prevent that class of problem.

A checker that can invent references is worse than no checker at all, because it converts a visible gap into an invisible fabrication and attaches false confidence to the result.

Verification comes before judgement

The structural answer is ordering. Before any AI judgement is invited, six independent databases — Crossref, OpenAlex, Semantic Scholar, PubMed, Google Scholar and Google Books — establish whether a citation is genuinely missing, as opposed to merely misspelled, hyphenated or accented differently.

That ordering removes the most dangerous case entirely. Most “missing” references are not missing — they are the same author spelled two ways, or a mangled author string, or a diacritic that survived one copy-paste and not another. Resolving those against real bibliographic records is retrieval, not generation. A model asked to fill the same gap from its weights has no way to distinguish recall from invention.

PubMed is only consulted where a citation reads as biomedical — a small detail that reflects the same discipline: ask the source that could actually know.

Your library outranks the internet

Your own uploaded sources — PDFs, DOCX, BibTeX — are searched before any external database. This is a correctness argument, not a convenience one. The PDF in your folder is the edition you actually read and cited. A record retrieved from an index is a plausible candidate for what you might have meant. When those two conflict, the one sitting on your disk is the better evidence, and it should win.

Evidence you can inspect

A deterministic system can show its work in a way a probabilistic one structurally cannot. Every pairing carries the evidence it rests on — which identifier matched, whether the two sides are written in compatible styles, what was and was not present in the reference line.

This is what makes a disputed result productive. A supervisor who disagrees with a row can look at the specific evidence and form a view. Against a model’s opinion there is nothing to examine — only a confidence figure that was itself generated.

The same principle explains why evidence is reported rather than scored. Identifier coverage, styles in use and reference completeness all appear in the audit but do not move the number, so a thesis in a field that rarely publishes DOIs is not marked down for the publishing conventions of its discipline.

Saying “I cannot judge this”

The hardest thing to build into an automated system is a competent refusal. Match review reports verified, questionable and not-judgeable counts — and not-judgeable is a first-class outcome rather than an error state.

An unresolved alignment is one the system suspected but declined to decide — resolving it by hand is always better than a false Missing or a false Orphan. A tool that always produces an answer is not more useful than one that sometimes declines; it has simply relocated its uncertainty into your thesis, where you can no longer see it.

The conservation ledger applies the same rule at the level of the whole run. It reconciles what entered the process against what left it — and where the review accounting audit has not run, it declares itself unaudited, never balanced. Claiming a balance it has not verified would be the single easiest way to look more trustworthy and be less so.

So where does AI actually belong?

None of this is an argument against AI in academic tooling. It is an argument about placement. There are jobs in this workflow that rules handle badly and models handle well — all of them advisory, none of them decisive:

• Cleaning messy in-text forms — spacing, ampersands, inconsistent et al., entries that do not read as citations. Fuzzy, contextual, exactly what models are good at.
• Proposing corrections on weak pairs — the alignment audit re-reads pairs resting on thin evidence and proposes, rather than quietly accepting.
• Helping locate genuinely missing references — after six databases have confirmed they are genuinely missing.

In every case the model proposes and a human disposes. Nothing is rewritten in the thesis automatically; every suggested change is approved or dismissed by the author. The AI never gets to decide what is true — only to suggest what might be worth looking at, which is a role it can perform honestly.

The commercial design follows the same logic. Deterministic work — upload, extraction, review, cross-check — is free and unlimited. AI steps cost credits, and the price is stated next to the button before you press it. You are never billed for a judgement you did not choose to invoke.

The unglamorous conclusion

The engineering here is mostly not exciting. It is careful parsing, an ordered matching backbone, six database integrations, and a great deal of attention to what happens at the edges — bare year markers, tools that assume everything is APA, and testing across thousands of real theses rather than tidy synthetic ones.

That is the actual work. A model asked to eyeball a bibliography would produce something impressive-looking in a fraction of the time, and you would have no way to know when it was wrong. For a document that gets examined once, by someone reading closely, being right and boring beats being clever and occasionally fictional.

• Read how the method is stated in the CitationLab Product Guide (PDF), or walk it in the product tour.
• More on the principle in deterministic checking and nothing gets lost.
• Credit costs for the advisory steps are on the pricing page; method questions are answered in the FAQ.