The recommendation: evaluate reviewed work, with evidence you can keep
A Swiss legal AI trial should answer a concrete question: does this configuration help this team produce better reviewed work on its recurring matters? Begin with two or three shortlisted tools, a small set of approved completed matters and a clear baseline. Decide what counts as a useful result before reading the outputs.
Download the blank evaluation worksheet. It is an editorial template created by Whisperit, with all vendor identities, scores and results blank. It is separate from Weblaw's public observations, which record a limited organizer-published comparison.
The research axes below build on Weblaw's published method; the workflow, commercial and procurement checks are our proposed additions. Weblaw did not publish results for those additions in the public matrices. [1]
Decision diagram
Compare evidence on the same work
Define acceptance
Agree on required sources, controls and deliverables.
Repeat the task
Use the same approved files and instructions for every candidate.
Inspect the result
Check citations, omissions, corrections and the export.
Record the decision
Keep evidence, unresolved questions and the reason for your choice.
Keep unresolved confidentiality, source-access or export requirements as decision gates. A high average score cannot compensate for a requirement your firm must meet.
Set up a trial that can inform a decision
Choose matters the firm is permitted to use. Include a research question, a document task and a handover. Prefer completed matters because a knowledgeable reviewer can establish what a good answer should include before seeing the software's response.
Record the vendor, product, plan, enabled collections, model/configuration and run date. Save the exact prompt and input inventory. Source access and deployment must match the offered product; a specially configured demonstration should not be silently treated as the standard subscription.
Write a reference answer with the essential facts, expected authority, contrary material and known uncertainty. The reference is a review aid, not an invitation to require identical wording. A tool that identifies a valid issue the baseline missed should receive an expert assessment rather than an automatic penalty.
Separate the person running the tool from the person reviewing its deliverable where practical. Let the reviewer inspect the result without a sales explanation. Include their correction, source-checking and export time in the outcome.
The criteria
| Criterion | A useful test | Evidence to retain |
|---|---|---|
| Substantive correctness | Apply a known rule to facts with an important exception | Reference answer, output and reviewer explanation |
| Source existence and support | Include a plausible authority that does not prove the requested claim | Exact cited passage and claim-level review |
| Full-text access | Open an entitled commentary and an official decision | Work, edition, passage and access boundary |
| Invented sources | Ask an unfamiliar question where bluffing is tempting | Unresolvable citations and the tool's uncertainty |
| Historical versions | Use a matter where the relevant date changes the applicable law | Version date and transitional reasoning |
| Court hierarchy | Supply apparently conflicting authorities | Explanation of relevance, hierarchy and contradiction |
| Missing facts | Deliberately omit a decisive fact | Questions, assumptions and visible qualifications |
| Multilingual equivalence | Produce French, German and English deliverables from equivalent facts | Bilingual lawyer's substantive corrections |
| Document quality | Revise a structured draft with definitions and cross-references | Original, revised file and tracked corrections |
| Handover and access | Have another authorised user continue; test restricted material | User roles, outputs and continuation effort |
| Data-path approval | Inspect the exact storage, inference and retention arrangement | Approved configuration and contractual evidence |
| Commercial value | Price the demonstrated system and record reviewer effort | Scoped quote, retained tools and measured minutes |
The first six research themes correspond to Weblaw's method. The remaining criteria make the trial specific to a firm's complete purchasing decision. They are not universal benchmark weights. [1]
Use a simple score without hiding failures
For task-quality criteria, use an optional 0–3 scale: 0 means unusable for the intended task; 1 means major corrections; 2 means useful after ordinary substantive review; 3 means it meets the firm's predefined standard after verification. These meanings are editorial suggestions. They are not measured vendor scores.
Attach an explanation and the evidence file to every score. “Three” does not mean no lawyer review is required. It means the reviewer checked the task against a stated standard and accepted the result.
Mark a critical failure separately. Examples might include an invented decisive authority, a confidently wrong historical version, disclosure of restricted material or a failure to meet a mandatory client processing condition. The firm should agree its own critical criteria before testing. Do not average an unmet mandatory requirement away with attractive scores elsewhere.
Data approval and contractual scope are generally better recorded as approved, unresolved or failed requirements than as subjective quality points. Keep them distinct from the research score. Only aggregate optional quality scores if the firm has deliberately chosen weights that match its work; always retain the individual findings.
Three tasks that expose useful differences
A historical-law research note
Use an approved scenario where today's rule is not automatically the applicable rule. Ask for the relevant date, statute version, leading authority and counterargument. Open the passages. A source link is insufficient if the output applies the wrong period or treats a real authority incorrectly.
Weblaw's public dashcam material illustrates why this matters: the expected answer depends on the historical statute, while the detailed version-results slide is outside the nine-slide public extract. Do not use that gap to assign scores to named products. Run your own controlled task instead. [2] [3]
A document with an amendment
Provide an agreement, an amendment and a party-specific instruction. Ask for a risk memo and an editable revision. Include an apparent issue resolved by the amendment and another that remains unresolved. Check whether the output distinguishes them and preserves definitions, dates and references.
Use the actual plan's editing path. A native Word tool and a separate exported draft may both be useful, but their handoff costs differ. For relevant evaluations, Jurata and Libra publish plan-specific Word access, while CASUS describes native Word review; Whisperit's verified published capability includes export. [4] [5] [6] [7]
An interview-to-colleague handover
Start with approved interview material and supporting documents. Require a factual summary, a missing-information list and a client letter. Change one material fact after the first output. Ask a second lawyer to continue without re-entering everything.
This checks whether context remains useful across the job. Do not assume audio is exclusive to one vendor: Silex publishes audio transcription, and Whisperit publishes a dictation-to-document workflow. Compare correction and continuity rather than the presence of a microphone button. [8] [9]
Test French, German and English meaningfully
Use equivalent legal facts in each language and have a qualified reviewer check the legal meaning. A literal translation can preserve the sentence while changing the legal concept. Include party names, negation, defined terms, procedural dates and an intentionally ambiguous phrase.
Record errors by consequence: stylistic preference, terminology problem, factual change or substantive legal change. Keep those categories separate. The existence of a language selector is not evidence that the product meets the practice's multilingual standard.
Repeat enough to challenge the first impression
Run a subset of critical tasks again, retaining the configuration and instructions. A single impressive answer can reflect a fortunate run; a single weak answer may need diagnosis. Record variation and whether the tool consistently states the same unresolved issue.
Compare against the existing process as well as other AI products. Measure time to the accepted deliverable, not only time to first response. Do not automatically convert every saved minute into billable revenue; value depends on the work that replaces it and the firm's capacity to realise it.
Finish with a decision memo: chosen product and configuration, accepted weaknesses, retained subscriptions, approval evidence and a review date. That is a more useful outcome than an unqualified “best AI” label.
Before setting up the trial, use the AI workspace decision guide to define the job and deliverable. The Swiss market overview can narrow the vendor categories to the two or three candidates worth testing.
Frequently asked questions
Does the worksheet contain scores for the competitors?
No. Every result field is blank. It is a template for the firm's own evidence, clearly separated from the compiled Weblaw observations.
Can we reuse Weblaw's four cases as the entire trial?
They can inspire research checks, but the public extract cannot cover the firm's full workflow, source entitlement, data approval or commercial value. Include the tasks your team actually performs.
How many products should we trial?
Two or three around one defined job is usually a manageable editorial starting point. Add another only when a specific unresolved need warrants it. A smaller completed evaluation is more useful than many unfinished demonstrations.
Romain de Wolff
