LangSmith Adds Jev for Typed Agent Evaluations
The new TypeSafe provider scores agent traces with probabilities and structured choices, giving LangSmith teams another option between deterministic checks and generative judges.
Edited by Tyronne Panaino
LangChain added Jev as an evaluation judge in LangSmith on September 21, creating a third path for teams that need to score open-ended agent behaviour. Instead of checking only deterministic conditions or asking a generative language model to reason in free text, Jev returns typed answers and probabilities that LangSmith can store as feedback.
The integration is aimed at narrow, repeated decisions such as whether a trace contains personally identifiable information, which intent category applies or how strongly a response matches an ordered score. It does not make code-based checks or language-model judges obsolete, and LangChain's own evidence remains an early first-party test rather than an independent benchmark.
A typed judge between rules and generative reasoning
Code-based evaluators are fast and predictable when a team can specify the expected condition in advance. Generative judges can handle more open-ended behaviour, but their free-text reasoning must be converted into a structured result and can vary between runs.
In the LangChain announcement, Jev is described as a System One model rather than a traditional text-generating language model. It evaluates a supplied state, such as an agent trace or message, and returns typed answers with probabilities. The supported question forms are a yes-or-no probability, a choice from a defined set and a score on an ordered scale.
That design gives developers a constrained output contract. It is most relevant when the decision is narrow, the categories are known and the check must run repeatedly across many traces. A team that needs a written explanation of a nuanced decision may still prefer an LLM judge, which the source explicitly presents as the better fit for open-ended criteria that require reasoning alongside the verdict.
How the LangSmith integration works
TypeSafe AI is now available as a model provider inside LangSmith. A team adds a TypeSafe API key, creates an evaluator, selects TypeSafe as the provider and chooses `jev-latest` as the model. The evaluator maps trace or thread variables into a state, then defines each criterion as a typed question. Every question appears as a separate feedback key that can be filtered, charted or connected to alerts and automations.
Jev processes the questions in one request, so the integration is designed to evaluate several related criteria against the same state. LangChain positions this as useful for online evaluation, where live traffic may need checks for issues such as data leakage, prompt injection or toxicity. Those are examples of possible evaluation criteria, not proof that the integration prevents any of those failures.
Data retention and evidence limits matter
The setup guide carries an operational warning: TypeSafe does not currently offer zero data retention, so prompts and outputs sent for evaluation may be retained by the provider. Teams handling confidential, regulated or customer data should account for that boundary before sending production traces to Jev.
LangChain also reports an internal comparison against three language-model judges, but describes it as one test on one agent. That evidence is useful for forming a test plan, not for assuming the same accuracy, speed or cost relationship across different agents, trace lengths and rubrics. The safer checkpoint is a team's own labelled dataset, repeated-run consistency checks and a retention review for the data it intends to evaluate.
Status
Confirmed. LangChain documents Jev as available in LangSmith Evals through the TypeSafe provider. Confidence is medium because availability and performance details come from the product maker's announcement and have not been independently validated across production workloads.
Sources
Update note: Last reviewed 2026-09-22. We will revise this post if LangChain changes the integration path, supported output types or provider-retention terms.
Sources
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.