🔍 Read the full analysis: OpenAI Training Agents In Software: The Terms To Examine At Ironclad on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI described training GPT-6 Astra in hosted copies of Ironclad’s contract-management software, using 11 selected legal, commercial and procurement tasks. Astra met an average 55% of task rubric criteria, while OpenAI’s estimated completion times were simulations rather than measured customer results.
OpenAI said on October 6 that it trained and evaluated GPT-6 Astra on selected workflows in hosted copies of Ironclad’s contract-management software, offering a concrete example of frontier-model training inside a specialized business product. Astra met an average 55% of rubric criteria across 11 tasks, according to OpenAI; the company’s estimated task times are simulations, not measured customer savings.
The tasks were selected by Ironclad staff and OpenAI employees familiar with the product. They covered legal, commercial and procurement work, including setting up nondisclosure agreements, creating procurement approval processes and changing a reusable contract clause based on a requester’s selected jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes on each task.
OpenAI scored each task against a rubric of 8 to 50 criteria, depending on its complexity. It reported that GPT-6 Astra met an average 55% of criteria, compared with 41.6% for GPT-5.6 Sol in the high setting. An internal model used during Astra’s development reached 63.7%. On one showcase task, Astra met about 94% of criteria. These are rubric scores, not percentages of tasks completed successfully.
For training, Ironclad supplied hosted product copies where models could practise. OpenAI said it created synthetic tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information. It said it did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data. OpenAI also reported estimated times of 19.2 minutes per Astra attempt and 37 minutes for GPT-5.6 Sol, while explicitly describing those figures as simulated estimates based on assumed processing and generation speeds.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Contract Workflow Accuracy Matters
The reported results show both the promise and the limits of asking AI agents to perform work inside business software. A model may complete multiple steps, but a partial score can hide a consequential omission. In procurement, for example, missing a required Finance approval above a spending threshold or skipping a Security review could leave a purchase outside the company’s rules. In contract work, getting most rubric items right does not establish that the final workflow is safe to use.
OpenAI’s account points to a developing role for software companies as training and evaluation partners: they can supply difficult workflows, domain experts and secure test environments. For buyers, the practical question is not only whether an agent can operate an interface, but whether it follows business rules, records its actions and flags work requiring human review. The results do not establish that agents are ready to run these processes without oversight.
There is also a strategic question for vendors. If agents increasingly mediate how customers use a product, a vendor’s value may depend less on its screens and more on its business rules, data structures, audit records and controls. OpenAI’s post frames the work as evidence for the continuing need for a full contracting platform. That is a rationale in the company’s account, not an independently demonstrated market outcome.
As an affiliate, we earn on qualifying purchases.
How the Ironclad Trial Was Set Up
The announcement concerns a specific research effort, not a general claim that OpenAI agents now operate contract systems autonomously. The evaluation involved 11 selected tasks in one specialized product. OpenAI estimated that an experienced user could complete each in 30 to 40 minutes; the model-time figures were calculated simulations and do not measure how long customers actually took or how much time a deployed system would save.
The scoring method also matters. Each task had multiple criteria, with the number varying from 8 to 50. OpenAI’s reported 55% is the average share of those criteria met, not a 55% task success rate and not a guarantee that a workflow met every required control. The company itself noted that losing track of a business rule can constrain what a software company should ask an agent to do, and that human oversight remains important.
OpenAI said it is inviting a small number of software companies to partner on workflows that current agents cannot reliably complete. It described the requested contributions as a concrete failing example, people with deep knowledge of the work, a secure test environment and data that can safely be used for research. The source account says OpenAI published another item on October 6 about mathematics manuscripts, but this article focuses on the Ironclad development and its reported evaluation.
What the Evaluation Cannot Establish
The reported findings do not establish how Astra would perform across the full range of Ironclad customers, contracts or unusual cases. The work covered 11 selected tasks, and the source material does not provide a detailed breakdown of which criteria the model missed on each task. Without that information, readers cannot tell whether the average shortfall involved minor formatting issues or errors in required approvals and other controls.
It is also unclear how performance would change with live customer data, varied company policies, system changes or repeated use in production. OpenAI said it did not use non-public Ironclad customer data, and its time figures were simulated. The source does not give a customer deployment, independently verified results, or a timeline for broader availability. The reported scores should not be read as proof of safe use without review.
What Partner Testing Could Show
OpenAI’s stated next step is to work with a small number of software companies that can bring workflows current agents struggle to complete, along with domain experts, secure test environments and research-safe data. Further partner evaluations could show whether models improve on harder tasks and which kinds of errors persist. The company has not supplied a schedule or named additional partners in the source material.
For software vendors and business buyers, useful follow-up evidence would include task-by-task results, the criteria missed, how often agents need human correction, and performance on live or representative workflows. Any claimed time savings would need to be measured in actual use rather than inferred from simulated processing speeds. Until such evidence is available, the Ironclad results describe an early evaluation, not a demonstrated replacement for human review.
Key Questions
What did OpenAI announce about Ironclad?
OpenAI published details of training and evaluating GPT-6 Astra in hosted copies of Ironclad’s contract-management product, using 11 legal, commercial and procurement tasks.
What does Astra’s 55% score mean?
It is the average share of rubric criteria met across the evaluated tasks. It is not the share of tasks completed and does not show that each workflow met all required business controls.
Did OpenAI demonstrate customer time savings?
No. OpenAI described its per-task time figures as simulated estimates based on assumed processing and generation speeds, not measured customer results.
What data did OpenAI say it used?
OpenAI said it generated synthetic training tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information. It said it did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
Is Astra ready to run contract workflows without people checking its work?
The reported evaluation does not show that. Astra met an average 55% of criteria, and the source account says human oversight remains important. OpenAI has not established from these results that the model can safely handle these workflows without review.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
