🔍 Read the full analysis: 42 AI Setups Faced 15 Legal Questions. The Law Model Got 8/100 on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A test by ThorstenMeyerAI.com gave GLM-5.3 FlashX Legal 8 out of 100 across 15 questions on German, U.S. and EU law; it fully solved none and produced no answer on two tasks. Claude Opus 5.5 led with 79, but the test author says the results are a spot check, not a formal benchmark, and discloses that his company runs the test and Claude agents graded responses.
GLM-5.3 FlashX Legal scored 8 out of 100 in a 15-question legal AI test published by ThorstenMeyerAI.com, whose operator says the model fully solved none of the tasks and gave no answer on two. The result raises questions about using legal-branded chatbots for high-stakes work, but the test’s author says it is a spot check, not a formal benchmark, and discloses that his company runs it.
The Lexassay test gave 42 AI setups the same 15 questions, covering German, U.S. and EU law. Ten questions are public and five are hidden; the questions include deadline calculations, recent legal changes, court decisions and false-premise prompts. Models answered without internet access or tools, aiming to measure what they could produce without external research.
According to the published results, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leading setup fully met every criterion on only eight of the ten public tasks. The author reports that GLM-5.3 FlashX Legal had 21 citations judged real, 25 misattributed and one invented. Mistral Large 4 scored 10, with the author saying citation errors accounted for much of its lost credit.
The test penalizes citations that are invented or do not support the claim they accompany. Each such citation costs a quarter of a task’s points, according to the method description. Answers without a legal-advice disclaimer or referral to a lawyer also lose 5% under the scoring rules. The results page publishes answers, gold answers and citation checks for readers to inspect.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Errors Can Matter
The results point to two separate risks for people using AI in legal matters: incorrect legal analysis and citations that look authoritative but do not support the answer. The test’s examples include a mistaken appeal deadline and an outdated account of litigation over the EU–U.S. Data Privacy Framework. Those are not merely wording issues: a missed filing deadline or reliance on a misrepresented legal rule can affect a person’s options.
In the deadline example, the test describes two German civil cases with judgments served on April 27, 2027. The appeal period would ordinarily run for one month, but Corpus Christi falls on May 27 and is a public holiday in Bavaria, not Berlin. The author says the Munich deadline therefore moves to May 28, while the Berlin deadline remains May 27. He reports that one model gave Berlin an extra day and another placed the deadline for appeal grounds a month late. This illustrates why location and procedural details can change an answer.
The findings do not establish how a model would perform with a lawyer’s review, a web search, legal databases or repeated attempts. They do show that a legal label or confident citation is not, by itself, evidence that an answer is safe to rely on.
Popular right now
Sony BRAVIA XR8B 65-Inch 4K OLED Google TV (K-65XR8B)
De'Longhi Dinamica Plus Fully Automatic Espresso Machine
eufy C28 Robot Vacuum Cleaner and Mop for Pet Hair, 15,000 Pa
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Models
Lexassay’s questions span several legal systems and include five hidden tasks intended to reduce the chance that future models can memorize the published answers. For each task, responses are compared with a reference answer using three to six weighted criteria. The author says citations are checked against official databases and categorized as real, misattributed, invented or not findable.
The test author disclosed that ThorstenMeyerAI.com runs Lexassay and that several Claude agents graded responses. He also noted that Claude Opus 5.5, a Claude model, ranked first. That creates a potential conflict of interest, which the author says is stated on the method page; publication of the answers and checks is offered as a way for readers to review the grading. Separately, he says the test uses one answer per model and task, so small score differences may reflect noise.
The source also cites a separate Vals AI legal-agent benchmark with different conditions: models work from supplied documents. In that ranking, Mistral Large 4 placed eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. The contrasting results are not directly interchangeable because the tests assess different setups and capabilities.
“optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution”
— Z.ai, as quoted in its developer documentation
What the Scores Cannot Establish
The results are based on one response per model per task, according to the test author, and the source says small score differences may be noise. The material does not provide enough detail to independently evaluate every grading decision, model configuration or the effects of repeated runs. Although public answers and citation checks are available, the five hidden tasks are not disclosed in full.
The test also excludes internet access and tools. It therefore does not establish how the systems perform when connected to current legal databases or used within a professional workflow, nor whether a human lawyer could identify and correct errors. The source does not report a controlled comparison between standalone models and lawyer-reviewed answers.
The disclosed grading relationship also warrants attention: the site running the test uses Claude agents to grade, and a Claude model ranked first. Meyer identifies that conflict, but the supplied material does not describe an independent replication of the results. The scores should be read as findings from this particular test, not a definitive ranking of legal AI products.
How Readers Should Use the Results
The published material points readers to Lexassay’s method page, answers, reference answers and citation checks for further review. It does not announce a date for another test or an independent audit. Any future results would be more informative if they included repeated runs, clearly specified model versions and settings, and independent grading alongside the current disclosures.
For readers facing a real legal deadline or dispute, the source advises treating chatbot output as a starting point, checking each cited law and ruling in the underlying source, and asking a lawyer when money, a deadline or an authority is involved. The test is not legal advice, and its scores do not establish that any model can safely handle a person’s case.
Key Questions
What scored 8 out of 100?
GLM-5.3 FlashX Legal, a model Z.ai describes as optimized for legal workflows, scored 8/100 in the Lexassay test. The test author says it fully solved none of the 15 tasks.
Which model ranked highest?
Claude Opus 5.5 scored 79, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The test author disclosed that Claude agents graded the answers and that a Claude model ranked first.
What did the test ask the AI systems?
The 15 questions covered German, U.S. and EU law, including deadlines, recent legal changes, court decisions and questions based on false premises. Models answered without internet access or tools.
Does the test prove that legal AI systems are unreliable in every setting?
No. The author describes Lexassay as a spot check with one answer per model and task. It does not show how models perform with legal databases, repeated attempts or lawyer review.
Can I rely on a chatbot for a legal deadline?
The source advises readers not to take a deadline from a chatbot. Check the applicable law and court documents, and consult a lawyer when a deadline, money or an authority is involved.
Source: ThorstenMeyerAI.com
