A score with nothing behind it is a rumour with a number attached. If Arbi tells you a candidate is an 84 percent match and cannot say which sentence in which document produced that, you have not saved any work. You have moved the reading from before the decision to after it, when someone asks you to justify the shortlist.
This post is about the part of the system that makes the number checkable. Internally we call it the evidence layer, and it is roughly half the engineering in screening.
The problem with a bare score
The naive implementation is one prompt: here is a profile, here are the criteria, return a score. It works. It demos beautifully. It also fails in three specific ways that only show up at volume.
- It cannot be audited. When a recruiter disagrees with a verdict, the only available response is to re-run it and hope.
- It degrades with document length. A forty-page CV plus three years of GitHub activity does not fit comfortably in one pass, and quality falls off in the middle of long contexts in ways that are hard to detect from the output.
- It is not stable. Two runs of the same profile against the same criteria produce different numbers, and there is no diff you can inspect to find out why.
The fix for all three is the same: stop asking for a judgement about a person and start asking for a judgement about a passage.
Retrieval before judgement
Every profile is decomposed into spans: a role, a project description, a paragraph from a cover letter, a repository README, a certification record. Each one carries a stable identifier and a pointer back to its source.
For each criterion, we retrieve the spans most likely to bear on it. This is hybrid: a dense vector search so that "operated containerised workloads" finds a criterion about Kubernetes, plus a lexical pass so that exact tokens like a specific licence number or framework version are not lost to paraphrase.
criterion → retrieve k spans (dense + lexical, reciprocal rank fusion)
→ judge each span independently: supports / contradicts / irrelevant
→ aggregate to a verdict with the supporting span ids attached
Judging spans independently is the important part. It bounds the context each judgement sees, it makes the unit of work small enough to run in parallel, and it means a wrong verdict can be traced to a specific span rather than to a vibe about the whole document.
Citing a span, not a document
Anyone can attach a source link. The useful thing is a character range.
Every verdict carries the span ids it rested on, and every span id resolves to an offset in the original document. That is what makes the review panel work: clicking a requirement highlights the exact sentence in the CV that satisfied it, in place, with the surrounding paragraph visible.
It also gives us the only regression test that matters. When a recruiter marks a verdict wrong, we capture the criterion, the spans, and the verdict as a labelled example. That corpus, currently a bit over 90,000 human-corrected judgements, is what we evaluate model and prompt changes against, and it is considerably more valuable than any public benchmark for this task.
What we do when the evidence is thin
The most consequential design decision in the whole system is what happens when retrieval comes back with nothing good.
The tempting behaviour is to let the model reason from context: the candidate was at a company that certainly uses Kubernetes, in a role that would certainly involve it, so mark it a pass. This is exactly the behaviour that makes a screening tool untrustworthy, because the inference is invisible and frequently wrong.
Arbi returns not evidenced and says so. It is a distinct state from a fail, it renders differently, and it is the correct answer surprisingly often. Resumes are lossy documents, and plenty of true things about a candidate are simply not written down anywhere in them.
Cost and latency
Judging every criterion against six spans for four hundred candidates is a lot of inference. Three things keep it viable.
- Spans are shared across criteria. Decomposition and embedding happen once per profile and are cached; re-running a stage with edited criteria only re-runs the judgement step.
- Cheap models do most of the work. Span-level relevance is a small classification problem. It does not need a frontier model, and routing it to one is how teams end up with a screening bill larger than their ATS.
- Escalation is selective. Judgements near a decision boundary, and criteria the recruiter has marked as high importance, get a second pass from a larger model. Everything else does not.
Median wall-clock for a 400-profile stage against eight criteria is a little under four minutes. The recruiter who kicked it off is generally still in the tab.
Where this is still weak
Two places, both known.
Cross-span reasoning is limited by design. A criterion like "has grown a team from three to fifteen" requires assembling a fact from several places in a document, and our aggregation step handles the easy version of this and misses the hard one. We would rather miss it and mark it not evidenced than hallucinate it.
And the evidence layer is only as good as the source. A profile that is out of date is out of date, and no amount of retrieval fixes a document that does not mention the last two years. That is a data problem, not a modelling one, and it is the honest limit on what screening can tell you before someone picks up the phone.
