Saruman Legal AI

Case analysis for Ohio criminal defense.

It reads a case file and works out what the State’s theory of the case appears to be, what that theory requires them to prove, and where the record doesn’t get them there. Then it sets out the defense theories the file will actually support.

Ohio only, for now. It covers Ohio state courts and the federal courts whose law binds Ohio — the Northern and Southern Districts of Ohio, the Sixth Circuit, and the Supreme Court. No other state. Further jurisdictions are planned, and none is built.

What it does

What it refuses to do

Most of the design effort went into what it declines to say. A tool that is confidently wrong about criminal law is worse than no tool.

How that is tested

Case files are built with a held-out answer key, run blind, then scored against it. Four have been scored on the current rubric:

82Case 014
86Case 015
80Case 016
79Case 019

And two runs with nothing wrong in them

Separate case files are built with no planted defects at all, to measure something nobody publishes: whether the tool invents problems that are not there. Each is registered in writing before the file is uploaded, so the prediction cannot be written after the result.

The first took 1 trap of 20 and declined 7 — but it also reported six disagreements that were not there: five were identical boilerplate appearing in different filings, and one was its own transcript of a recording read as a second, conflicting account. The method that produced them was replaced rather than patched.

The second, on a different case and run after that change, scored 97 out of 100: nine traps engaged and none fallen into, and the genuine issues that clean file did contain were found.

Those control scores are a different rubric from the four above and are not comparable to them — one measures whether the tool finds what is there, the other whether it invents what is not. Two runs on two different cases is also not a trend.

Publishing the control matters more than publishing the scores. A tool that finds problems is easy to build. The hard question is whether it finds problems that are not there — and that is the number nobody reports.

What is not finished

Who is building it

Jodex Saint-Vil, a software engineer in Ohio. Previously worked on legal software at Filevine. Not a lawyer — which is why the tool is built to refuse rather than guess, and why the questions it cannot answer are written down and waiting for someone who can.