Methodology·honest accuracy · CLAUSE legal benchmark

Measured.
Not promised.

Most legal-AI vendors quote one accuracy number averaged across categories. We publish recall per contradiction type against the CLAUSE legal benchmark, including where our pipeline still struggles. Then we show you exactly what a sprint delivers, before you ever book a call.

BENCHMARKCLAUSE legal · public · reproducible
SCOPE5 contradiction types · per-category recall
CLAUSE LEGAL · RECALL BY TYPEbenchmark · v2.0 · reproducible
0.00.51.0
  1. InconsistencySTRONGEST
    0.83
  2. AmbiguitySTRONG
    0.78
  3. OmissionMODERATE
    0.56
  4. DriftWEAK
    0.44
  5. StructuralWEAKEST
    0.22 / 0.60
PRECISION0.21 baseline · 0.55-0.70 after playbook calibration
214Findings shipped
31Contracts reviewed
3Sprints delivered
0Findings without citation
Mateusz Szczepanski
Mateusz Szczepanski · Founder & Principal EngineerRuns every sprint personally
● 02: FIVE CONTRADICTION TYPES

One accuracy number is a marketing artifact.

Recall varies by type. A pipeline that catches 95% of inconsistencies and 22% of structural defects is not “95% accurate.” It is two different things wearing one number. We publish the breakdown.

TYPECLAUSE RECALLWHAT IT IS
  1. 01Inconsistency
    0.83STRONGEST

    Two sections of the same contract contradict each other on a quantitative or factual point: payment term, notice window, liability cap.

    EXAMPLE
    §4.1“Fees are payable for twelve (12) months.”
    vs
    §9.3“On termination, Customer is liable for six (6) months of remaining Fees.”
  2. 02Ambiguity
    0.78STRONG

    A single clause is readable two contradictory ways. Often a numeric reference (“twelve months of Fees”) without a clarified denominator, or a defined term used in an undefined sense.

    EXAMPLE
    §11.1“…limited to twelve (12) months of Fees paid hereunder.”
    vs
    n/aReadable as 12× monthly OR last 12 months of annual fees. Material delta on a €200k contract.
  3. 03Omission
    0.56MODERATE

    A required provision, schedule, or annex is referenced but missing from the delivered document. Frequently sign-blocking under DORA Article 30.

    EXAMPLE
    §5.6“Vendor shall maintain the authorized Subprocessors list in Annex B.”
    vs
    Annex BReferenced annex absent from delivered PDF.
  4. 04Terminology drift
    0.44WEAK

    A defined term is renamed or redefined mid-document: “Service Levels” in §2 becomes “SLA” in §9 with subtly different obligations. Easy for human reviewers to miss after 30 pages.

    EXAMPLE
    §2“‘Service Levels’ means the metrics defined in Annex A.”
    vs
    §9.4“SLAs are measured monthly; breaches trigger service credits.”
  5. 05Structural
    0.22 / 0.60WEAKEST

    Self-contradictory section hierarchies and broken cross-references: “see §3.4(c)” when §3.4(c) does not exist. The hardest type in CLAUSE; we publish two numbers because pair-mode and single-section analysis perform differently.

    EXAMPLE
    §3.4“See sub-clause 3.4(c) for notice requirements.”
    vs
    n/aSub-clause 3.4(c) does not exist. Cross-reference broken.

NOTE · Precision baseline before playbook calibration: 0.21. Calibration to your severity rules and required-clauses list typically improves precision to the 0.55-0.70 range, depending on how strict your rubric is. Numbers above are recall against CLAUSE labels at delivery configuration.

NEXT · 2 MINThese five types, flagged on a real contract: citations, severities, owners.
See what you actually get
● 03: WHAT A FINDING LOOKS LIKE

Five elements per finding.
Missing one? It gets dropped.

  1. CITATIONVerbatim quote, section reference, PDF page. No paraphrasing.
  2. EXPLANATIONTwo to three sentences, plain language. What contradicts what.
  3. SEVERITYMapped to your playbook’s rubric, not our opinion.
  4. ACTIONDrafted recommendation. Your lawyers override; we provide the start.
  5. OWNERSuggested decision-maker. Editable in the delivered XLSX.
FINDING F-001 · VENDOR MSA · ANONYMIZEDHIGH
Fee term contradiction: §4.1 vs §9.3
CITATION
§4.1 · p.6: “Fees are payable for twelve (12) months.”§9.3 · p.14: “On termination, Customer is liable for six (6) months of remaining Fees.”
EXPLANATION
The contract states two different fee obligations on termination. §4.1 implies a full 12-month commitment; §9.3 caps liability at 6 months. Material to renewal accounting on a multi-year term.
SEVERITY
High: quantitative contradiction on a payment term, per your severity rubric §2 (“financial exposure, sign-blocking”).
ACTION
Negotiate: request amendment of §9.3 to reference §4.1 explicitly before signature.
OWNER
Legal (contract owner) · CFO informed. Editable in the delivered XLSX.
● 04: WHERE IT STRUGGLES

What we're not good at, said out loud.

“If a vendor won't show you this table, ask them why not.”

Read it the way your auditor would.

Structural defects in pair-mode (0.22 recall)
When two clauses argue across distant sections, raw pair-mode misses most. Mitigated with single-section structural analysis (0.60), but pair-mode is still where we need the most engineering investment. We tell you which mode your sprint uses, and why.
Multi-language contracts
English is our strongest configuration. DE, NL, SE work but require more manual review in validation, extending the sprint by 3-5 days and adding €3-5K. We price honestly upfront.
Contracts shorter than 8 pages
There simply aren't enough clauses to find cross-section contradictions. We're worth hiring at scale (10+ contracts) or for one document if it's substantial (30+ pages). Single short contracts are not a fit.
What we don't do at all: negotiate on your behalf, give legal advice, replace your counsel's judgment, or speak to regulators on your behalf. Every finding is a draft for your lawyers to validate, override, or reject.
Mateusz Szczepanski, Founder & Principal Engineer at OpsSolved
Mateusz SzczepanskiFounder & Principal Engineer · OpsSolved
● A NOTE FROM THE FOUNDER

A person answers
for these numbers.

Every number on this page is mine to defend. I configure the pipeline, run the validation pass, and sign the findings memo on every sprint. There is no junior team behind a curtain.

That's also why the weak numbers are published next to the strong ones. You're going to find out anyway. Better from this page than three weeks into an engagement.

● 05: SECURITY + DATA HANDLING

One page. Cleared by your InfoSec before signing.

RESIDENCY
Azure EU · Frankfurt region · ephemeral processing tenants
ENCRYPTION
TLS 1.3 in transit · AES-256 at rest · per-engagement keys
ISOLATION
Multi-tenant isolation · per-engagement vector store · per-org chunk scoping
RETENTION
30-day default · 7-day on request · delete-on-demand · contractual
AUDIT LOG
Tenant-scoped · every action recorded · exportable on request
GDPR
Pre-signed DPA covers processor obligations · SCCs where applicable
ROADMAP
ISO 27001 audit scheduled Q1 2027 · SOC 2 Type II following
WE DON'T
Train on your data · share with third parties · retain after deletion
Want the full evaluation protocol? The 13-page methodology brief has every table.13 PAGES · 297 KB · NO EMAIL GATE
Download brief (PDF)
● SEE · IT · FIRST

Judge the output,
not the pitch.

Browse real findings from an anonymized sprint: citations, severities, owners, redlines. If it looks like something your team would actually use, book 20 minutes and we'll scope yours.

FIXED SCOPE21 DAYS FROM KICKOFFFOUNDER-LED · EU DATA RESIDENCY

OpsSolved does not provide legal advice or regulatory representation. We deliver evidence-backed findings and review artifacts; your legal and compliance teams validate findings and make final decisions.

NEXT STEPThe methodology is the proof. The findings are the product.See what you actually get