The finding: useful observations, no public named winner
Weblaw's AI Legal Research Challenge is a useful starting point for a Swiss legal AI evaluation. It does not provide a publicly recoverable provider leaderboard in the nine slides available on the organizer's page. The result columns are anonymous. We therefore cannot use them to say Silex beat Omnilex, or Lawsearch AI beat Jurata. [1] [2] [3] [4]
We have transcribed the visible observations into CSV and JSON. They contain 98 cell-level records across 14 displayed metrics, including entries explicitly marked “not disclosed.” They contain no assigned provider identities, composite scores or estimated missing values. These are observations from public slides, not test results produced by Whisperit.
The practical lesson is to inspect the reasoning and source version after a tool finds the apparently right decision. Retrieval is a necessary starting point; it is not the whole standard for work a lawyer can rely on.
Published challenge slide
Read the challenge’s method before its results
Weblaw’s original slide describes accuracy, sources, full-text access, hallucinations, currency and contradictions. It explains a limited observation exercise; it does not identify anonymous result columns or establish an overall winner.
Source: Weblaw · AI Legal Research Challenge, slide 2
Open full-size imageWhat was tested
The challenge took place on 17 June 2026. Participants were Silex, Omnilex, Jurata, Unplex, Libra, Lawsearch AI and an Opus model combined with OpenCaseLaw. According to Weblaw, each received the same prompt and input conditions for each of four cases. [5] [1]
The public methodology uses six axes: substantive correctness, sources, full-text access and linking, hallucinations, applicable statute versions, and treatment of contradictions or court hierarchy. It distinguishes an invented citation from an existing source used incorrectly. Those axes are more useful than counting how authoritative an answer sounds. [6]
The scenarios concern a driving instructor's permit, creditor/shareholder standing in a debt-enforcement setting, dashcam evidence and data protection, and Bern social assistance. These descriptions and expected answers come from the organizer. We are analysing the published evaluation, not independently endorsing a legal answer to those cases. [7] [8] [9] [10]
The visible results, compiled
| Public result | What the slide shows | What it does not establish |
|---|---|---|
| Case 1: expected decision 2C_149/2024 | Present in 7 of 7 anonymous columns | Equal reasoning quality or full-text verification |
| Case 1: additional decision 2C_131/2023 | Present in 1 of 7 columns | That citing this extra decision improves the answer |
| Case 1: literature | Present in 4 of 7 columns | The authority, licence or correct use of that literature |
| Case 2: Art. 298(2) SchKG | Present in 7 of 7 columns | Correct analysis of standing in every answer |
| Case 2: expected decision 5A_53/2026 | Present in 6 of 7 columns | Which named product omitted it |
| Case 2: literature / nullity issue | Each present in 4 of 7 columns | That longer or broader answers are always better |
| Case 3: BGE 146 IV 226 | Present in 7 of 7 columns | Correct handling of the historical statute version |
| Case 3: agreement with expected answer | 6 full, 1 partial | A general accuracy rate for any provider |
| Case 3: literature | Present in 3 of 7 columns | A comparative quality score |
Source matrices: slides 5, 7 and 9. Positions are transcribed left to right within each slide. We have not established that a position identifies the same vendor across cases; the dataset must not be joined into a vendor score. [2] [3] [4]
Slide 5 shows “X” for the numerical statute-count field. We preserve the field as undisclosed, rather than interpret the marks as six statutes or none. The answer-length fields in slides 5 and 9 are empty. They remain missing values, not zero-word answers. This matters when compiling data: apparent completeness achieved by guessing would make the comparison less trustworthy.
A distinction the broad summary can hide
Weblaw's narrative broadly describes strong agreement with expected solutions and relevant leading decisions being found. The visible case-two matrix is more specific: one anonymous column does not cite the expected decision. Retain that difference rather than flatten the slide into a claim that every tool found every leading case. [1] [3]
The narrative also reports one hallucination of its defined first type in the dashcam case, related to mixing old and new data-protection law, and none of the second type across the challenge. Those are organizer-reported findings for four cases. They cannot establish a zero-error product or predict the frequency of an error in a firm's practice. The public matrices do not identify which vendor produced the reported error. [1]
For the dashcam scenario, the expected-answer slide identifies the old statute version as relevant. Slide 9 refers readers to the next slide for detailed version differences. That next slide is outside the nine-slide public extract. We consequently do not supply a count of products that handled the version correctly. [10] [4]
How much weight should a buyer give this evidence?
It is a real, narrowly scoped comparison published by its organizer. It provides more task-specific evidence than a feature checklist alone. It is also a small June snapshot, and Weblaw operates participant Lawsearch AI. We label that commercial relationship because an organizer-published comparison has a different evidential status from an independent, fully reproducible benchmark.
The cover notes that not all invited parties participated and one challenger declined reproduction of its results. It does not name that challenger. Do not infer the identity from the order of logos, anonymous columns or another vendor's marketing. [5]
The full evaluation is offered free on request. We used the public extract and have not obtained that report or the complete outputs. Without those, provider-labelled results, complete case-four findings and a sufficient reproduction record remain unavailable to this draft. A future update should preserve the original date and distinguish newly supplied evidence from the public extract. [1]
Whisperit, Swiss-Noxtua, CASUS and DeepJudge are not named participants. This means they were not assessed in this reported set; it does not mean they scored zero, performed poorly or were excluded for a particular reason.
Turn the challenge into a better trial
Ask each shortlisted product to handle a matter with a known historical law issue. Require the relevant date, version and transitional reasoning to be explicit. Give a contrary authority and ask whether it changes the conclusion. Include a deliberately absent fact and reward the tool for asking or stating a limitation.
For every material statement, open the cited passage and record whether it supports the claim. Separate “source exists,” “source is relevant” and “source proves this statement.” Then turn the research into the deliverable your team needs and let another lawyer verify it.
That final workflow was not established by the public challenge matrices. Dictation, source subscriptions, document editing, confidentiality architecture, colleague handover and commercial value need their own evidence. Our blank evaluation worksheet adds those checks without pretending they were measured in Weblaw's event.
Use the Swiss legal AI market overview for the participant categories and current product distinctions. Use the Swiss research tools guide for a source-based shortlist; it makes no named winner claim from these matrices.
Frequently asked questions
Which product won Weblaw's challenge?
The public material does not establish a named winner. We will not assign provider names to anonymous columns or manufacture a ranking from the participant list.
Is the downloadable data an independent benchmark?
No. It is Whisperit's transcription of observable facts from Weblaw's public slides, with provenance and missing values retained. The source images remain linked at Weblaw; we have not republished the slide deck.
Why not calculate percentages or a total score?
Finding a particular decision in six columns is not a general 6/7 product accuracy rate. Metrics also differ in significance, and some data are undisclosed. A composite score would conceal those limits and imply a comparison the public evidence cannot support.
Romain de Wolff
