🔍 Read the full analysis: Z.ai’s Legal AI Benchmark Results: How About 20% Passed Still Ranks Near The Top on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models in the archived listing. The score is strict: a task counts as passed only when every rubric criterion is met, while the model passed 92.5% of individual criteria in the same run.
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria pass rate on Harvey’s Legal Agent Benchmark, placing seventh among 81 models on Vals AI’s archived leaderboard. The result means the model met every requirement on about one in five tested tasks—not that it got only one in five individual answers right—and puts a strict measure of legal-work completion into focus for firms considering AI tools.
The score is listed by Vals AI, which evaluates models using Harvey’s benchmark and grading protocol. Z.ai’s October 13 blog post names Vals as the independent evaluator; Vals’ archived leaderboard lists GLM Legal at 19.17% and $0.15 per test. Z.ai rounds the result to 19.2% in its developer documentation.
Harvey’s benchmark treats each task as complete only when every rubric criterion passes. A task can require an agent to produce a work product from client-matter materials, with criteria covering items such as facts, conclusions, citations, deadlines, recommendations and formatting. Under that all-or-nothing rule, GLM Legal missed at least one criterion on roughly four out of five tasks.
The same Vals run reports a 92.5% individual-criteria pass rate, a different measure of performance. Z.ai says the legal model improved by 12.5 percentage points over the base GLM-5.3-Flash model. The archived leaderboard supports that arithmetic: GLM Legal scored 19.17%, compared with 6.67% for the base model.
Why Task Completion Matters
The gap between a 92.5% criteria pass rate and a 19.17% complete-task rate shows why benchmark definitions matter. A system may satisfy many individual requirements while still failing a whole legal work product because one required fact, citation, deadline or conclusion is wrong. For professional use, partial accuracy may not be enough when the output is intended to support a complete task.
The ranking is also a comparison within a field where end-to-end performance remains limited. Harvey reported that the best all-pass result among frontier models in its May 2026 baseline was 7.1%, and said those models completed fewer than 10% of tasks end to end in aggregate. That was a separate run, so it should not be treated as a direct comparison with Vals’ GLM result. It does, however, underline that a high placement on this benchmark does not by itself establish readiness for unsupervised legal work.
For law firms, the results make verification and data handling practical questions alongside benchmark rank and cost. Z.ai presents the model for professional workflows, but firms would still need to assess how its outputs are checked, how client information is processed, and whether the tool fits their own confidentiality and compliance requirements.
Products Worth Considering
Ledger Nano Gen5 Crypto Wallet, Black
Trezor Safe 7 Crypto Hardware Wallet
Ledger Flex Crypto Wallet
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as an open-source evaluation on May 6, 2026. The public benchmark contains 1,250 tasks across 24 legal practice areas, graded against more than 75,000 expert-written, binary pass-or-fail rubric criteria. Harvey’s example of a change-of-control task includes 57 separate criteria.
Vals’ leaderboard uses a private holdout of 120 tasks that mirrors the public benchmark’s mix, with internet access disabled. According to Vals, two AI judges—GPT 5.5 and Claude Sonnet 4.6—grade submissions; the reported task pass rate averages their results. The leaderboard’s strict method is designed to test whether an agent can complete a work product against a detailed rubric, rather than reward a high percentage of correct fragments.
Z.ai describes GLM-5.3-FlashX-Legal as built on GLM-5.3-Flash and optimized for legal document understanding, reasoning and task execution, including research, contract review and drafting. Its documentation says the model underwent legal-domain mid-training and post-training, including reinforcement learning in task environments modeled on legal workflows. Z.ai recommends connecting it to services for laws, cases and company data. Its pricing page lists $0.37 per million input tokens and $1.25 per million output tokens; Vals lists a separate benchmark test cost of $0.15.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.””
— Z.ai, in its developer documentation
Limits of the Published Evidence
The leaderboard result does not establish how the model performs across every type of legal work or in live practice. Vals’ reported score comes from 120 private tasks and a particular judging setup; performance on a firm’s own documents, jurisdictions and workflows may differ. The score also does not show how often errors are severe, or whether human review would catch them.
Z.ai calls the result state-of-the-art among open-source models, but the cited documentation does not define the comparison set. The source material also reports no published weights for the legal model, so its status as open source cannot be independently assessed from the available details. A small four-task spot check described by the source author found citations to rulings that could not be located; that check is not a benchmark and cannot establish an error rate.
Questions about client-data processing, applicable rules and citation verification remain unanswered in the cited materials. Z.ai’s pages describe drafts for legal professionals, but the source author reports finding no “not legal advice” notice on the pages reviewed. It is not clear from the materials whether the model is intended to be used only with human review or what safeguards are built into its deployments.
What Firms Should Verify
The next useful evidence would include clearer disclosure of the model’s comparison group, its availability and weight-release status, and more detail on data handling and safeguards. Further results across legal tasks and independent evaluations could help establish whether the benchmark performance carries over to practical workflows.
Firms evaluating the tool will need to test it on representative tasks under their own review procedures, with particular attention to citations, factual support and deadlines. The benchmark provides a measured result, not a substitute for professional verification. No further benchmark release or product milestone is specified in the cited material.
Key Questions
What does the 19.17% score mean?
It is the share of benchmark tasks on which GLM Legal passed every required rubric criterion, according to Vals AI’s archived leaderboard. A task with one or more failed criteria does not count as a pass.
Why is the criteria pass rate much higher?
The 92.5% criteria pass rate counts individual rubric criteria, while the 19.17% score counts only tasks completed with every criterion met. A model can pass most criteria on a task and still fail the task overall.
How did GLM Legal rank?
Vals AI’s archived Version 1 leaderboard placed it seventh among 81 models. The listing reported a 19.17% task pass rate and a benchmark cost of $0.15 per test.
Does the result show the model is ready to handle legal work without review?
No. The benchmark measures performance on a defined set of tasks; it does not establish suitability for unsupervised legal practice. The cited material leaves questions about error severity, client-data handling and required human review unresolved.
What does Z.ai mean by a 12.5-point improvement?
Z.ai compares GLM Legal’s 19.17% score with the base model’s 6.67% on the same archived leaderboard. The difference is 12.50 percentage points, not a 12.5% relative increase.
Source: ThorstenMeyerAI.com
