🔍 Read the full analysis: Does Z.ai’s Legal AI’s Benchmark Rank Tell More Than Its Pass Rate? on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models in the archived results. That task-level score requires every rubric criterion to pass, while the same evaluation reports a 92.5% pass rate across individual criteria. The ranking and pass rate describe different things; neither establishes that the model can reliably produce legal work without human review.
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh of 81 models in the archived results. The score counts a task as passed only when every required rubric criterion is met, so it is not a measure of how many individual legal details the model gets right; the same evaluation lists a 92.5% criteria pass rate.
The leaderboard entry reports that the model completed all required criteria on 19.17% of benchmark tasks, at a listed cost of $0.15 per test. Vals AI’s archived leaderboard was last updated October 18, 2026. Z.ai’s October 13 blog post identifies Vals as the evaluator, and the company’s developer documentation describes the model as designed for legal research, contract review, document understanding and drafting initial work products.
The task score uses all-pass grading. Harvey’s benchmark rules say a task is complete only if every criterion passes; the criteria can cover facts, conclusions, citations, deadlines, recommendations and formatting. Vals reports a separate 92.5% individual-criterion pass rate for GLM Legal in the same leaderboard version, with a criterion counted only when both of its AI judges pass it. A high criterion score can coexist with a much lower task score when one missed requirement is enough to fail the whole task.
Z.ai says the legal model improved by 12.5 percentage points over its GLM-5.3-Flash base model. The comparison matches Vals’s listed figures: 19.17% for GLM Legal against 6.67% for the base model. Z.ai also calls the result state of the art among open-source models, but the supplied announcement does not specify the comparison set, and the source material says published weights for this legal model could not be found.
Why Task Completion Scores Matter
The difference between a task score and a criterion score matters to law firms evaluating whether a model can support real work. The 19.17% figure means the model met every required condition on roughly one in five tasks in the evaluation; it does not mean it got only 19.17% of legal details right. Conversely, the 92.5% criteria figure does not show that the model completed most assignments correctly from beginning to end.
In legal work, a missing citation, incorrect deadline or overlooked issue may undermine an otherwise polished draft. Harvey’s benchmark was designed around that risk: its scoring treats a task as incomplete if any required criterion is missed. The leaderboard can help buyers compare systems under a shared protocol, but it does not establish that any model is ready to handle client matters without professional verification.
Cost is another point of interest, though the numbers need care. Vals lists $0.15 per test for GLM Legal, compared with higher listed costs for several models above it. That is a benchmark cost, not a complete estimate of deploying the system in a firm. Z.ai’s separate pricing page lists charges per million input and output tokens, and the source material does not establish that benchmark cost and production costs are directly comparable.
Popular right now
Philips 2200 Series Fully Automatic Espresso Machine with Classic Milk Frother, 2 Coffee Varieties, Intuitive Touch Display, Ceramic Grinder
Hisense 65-inch U6 Pro Mini-LED 4K UHD Smart Fire TV
KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, Almond Cream
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 legal practice areas. Harvey says those tasks are graded against more than 75,000 expert-written, binary criteria, and that a private holdout set is used for leaderboard evaluations. Vals describes its holdout as 120 tasks, with internet access disabled.
According to Vals, two AI judges—GPT-5.5 and Claude Sonnet 4.6—grade each submission. Each judge calculates a task pass rate, and the reported score is the average. This is an evaluation design, not a direct review of the model’s performance in a particular firm or jurisdiction. Harvey’s own first baseline, published May 26, reported a best overall all-pass rate of 7.1% among the frontier models it tested. That was a different run, so it provides background on the benchmark’s difficulty rather than a direct ranking comparison.
Z.ai’s documentation says the model underwent legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of simulated legal-workflow task environments. It recommends using the model as an agent connected to services providing laws, cases and company data. The documentation says it can assist with work from research and contract review to initial drafts; those are company descriptions of intended uses, not independent evidence that the system is accurate in each setting.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its benchmark’s scoring rule
Limits Behind the Leaderboard Result
The published score does not show which kinds of errors drove GLM Legal’s missed tasks, how performance varied across legal practice areas, or whether the model’s ranking changes on a different set of matters. The 120-task private holdout is a limited sample compared with the benchmark’s 1,250-task public release, and the source material does not provide uncertainty intervals or repeated-run results for this entry.
The source article also describes a small, four-task spot check in which its author says GLM Legal cited rulings that could not be found. That check was not a benchmark and cannot establish an overall error rate. It also reports that two Claude models cited a real ruling for the opposite of what it held. These observations underline citation verification concerns but should not be treated as a controlled comparison.
Other open questions include what client-data protections and processing terms apply to a firm’s use of the service, and what review controls are expected before output is used externally. Z.ai’s documentation describes drafts for legal professionals, but the source material says the pages reviewed carried no explicit “not legal advice” notice. That observation does not settle the product’s legal status or the terms that apply to a particular customer.
What Firms Should Verify Before Use
The immediate next step is for prospective users to review the model’s deployment terms, data handling rules and integration requirements, then test it on representative matters under qualified legal supervision. Firms will also need policies for checking citations, authorities, deadlines and factual assertions before work leaves the organization. The benchmark ranking alone does not answer those procurement and governance questions.
Z.ai’s AutoClaw blog post, dated October 13, 2026, said a beta was open by application. The supplied materials do not provide a schedule for broader access, additional benchmark results or publication of model weights. Further updates could clarify availability and technical details, but until then, the leaderboard provides a specific benchmark result, not a general guarantee of performance across legal tasks.
Key Questions
What does GLM Legal’s 19.17% score mean?
It is the share of evaluated tasks on which the model met every required criterion, averaged across the two judges described by Vals AI. A task fails if even one required criterion is missed.
Why is the criteria pass rate 92.5%?
The 92.5% figure counts individual rubric criteria rather than whole tasks. Vals says a criterion counted as passed only when both AI judges passed it. It is a different measure from the all-criteria task rate.
Did GLM Legal rank first on the benchmark?
No. In Vals AI’s archived leaderboard, GLM Legal ranked seventh among 81 models. The entry showed a 19.17% task pass rate.
Does the benchmark show the model is ready to give legal advice?
No. The result measures performance on a defined benchmark and does not establish that the model can reliably advise clients or complete legal work without review. Z.ai describes its output as initial drafts for legal professionals; human verification remains necessary.
What should a law firm check before adopting the model?
Firms should examine client-data processing terms, applicable controls, integration setup and review procedures. They should also test citations and other consequential details on representative work before relying on outputs.
Source: ThorstenMeyerAI.com
