Getting Structure Out of an LLM

Resume-Screening-System takes a stack of CVs and a job description, retrieves the relevant chunks with FAISS, and asks a local Mistral model to score and justify each match. Retrieval was never the hard part — getting output I could trust was.

Ask for a schema, not an essay

Free-form answers look impressive and parse terribly. I ask for a fixed shape instead: score, matched skills, missing skills, one-line justification — then validate it with Pydantic before anything reaches the UI. If it doesn't validate, it retries once and fails loudly after that.

The retry matters as much as the schema. A single validation failure is noise; a silent fallback to free-text is a system that sometimes works and can't tell you when. One retry, then a visible error, means the failure mode is a decision someone makes, not a wrong ranking nobody notices.

Keep the model on a leash

A model that invents a skill is worse than one that says nothing. So every justification has to quote the chunk it came from, and any claim without a quote gets dropped. This one rule converts the model from a plausible writer into an evidence machine.

Running Mistral locally keeps this cheap enough to run across hundreds of CVs, and it keeps private documents private — a real constraint in any hiring context. The setup is the same stack as my other RAG work: LangChain for the pipeline, FAISS for retrieval, GGUF quantization so the model runs on CPU.

The scoring rubric does the thinking

Asking for a "match score" without defining it gets you numbers with moods attached. I pin the rubric in the prompt: what a 9 needs, what a 5 means, what scores low. Consistency across hundreds of CVs is a rubric problem, not a model problem — the same model produces wildly more consistent rankings once the scale has anchors.

Ranking, not deciding

The system ranks; a human decides. Every output is a suggestion with its evidence attached, because automated ranking without traceability is how you end up with a bad decision nobody can explain. The goal of the system is to make the reviewer faster and better-evidenced — never to close the loop on its own.