In May 2026 the software company Onyx published an unusual dataset on arXiv: slightly over half a million documents from a company that does not exist. Internal messages, email, CRM records, meeting transcripts, support tickets, wiki pages. Along with it came 500 questions an employee would ask on an ordinary working day, and a program that checks, answer by answer, whether the system under test got it right.
It is called EnterpriseRAG-Bench. Two details made me look twice. First: Onyx sells a product of the same kind and states, in its own repository, that it excludes itself from the leaderboard to avoid a conflict of interest. Second: anyone who wants in has to prove the result is reproducible — open-source systems hand over a walkthrough, closed ones hand over access to the live system so Onyx can re-ask the questions itself.
We submitted our document search to EnterpriseRAG-Bench, and on the September 15, 2026 leaderboard, among 22 systems, Skyller ranked 1st in answer completeness, with 87.39, and 2nd overall, with 79.30 points — 1.04 points behind first place. It came in ahead of the evaluated configurations of OpenAI File Search (61.03), Amazon Q (48.96), Azure AI Search (48.42), Vertex AI Search (41.87) and NVIDIA AI Blueprints (37.73). The comparison applies to the tested configurations and corpus; it does not rank every product on the market. The full numbers are on the answer quality page.
What a test like this actually does
The temptation, when someone talks about evaluating artificial intelligence, is to picture an exam: maths, logic, essay writing. It is nothing like that. The corpus is built to look like a real company, defects included: files stored in the wrong place, old versions living next to new ones, near-duplicates spread across different systems, and documents that openly contradict each other.
The 500 questions are split into ten types. There is the easy one, answered by a single document. There is the one nobody would phrase using the words in the file. There is the one that requires stitching together distant parts of a long document. There is the one you can only answer by gathering everything that exists about a project. And there is a group of twenty questions whose answer simply is not in the corpus — the assistant has to say it did not find anything, instead of inventing something plausible.
That last group is the one I value most, and the rarest to see measured. Every company that has put AI to work on internal documents knows the problem: the answer that is well written, confident — and wrong. A test that scores "it isn't here" is measuring reliability, not eloquence.
The grading program does not belong to the vendor being assessed. It compares the answer with an expected answer and with a list of facts that have to show up. Then it looks at the documents the system returned and decides, by a vote across three independent evaluations, which ones were genuinely necessary.
Why almost every quality figure is useless
In corporate software, quality figures tend to be born inside the house of whoever is selling. The vendor picks the question set, picks the pass criterion, runs it in its own environment and publishes the result in a sales deck. There is no way for the customer to verify any of it — and, to be fair, no way for the vendor to prove it did not pick the test it was going to do well in.
I have seen comparisons where the competitor was configured by the party doing the comparing. I have seen accuracy percentages with no denominator: accurate on how many questions, asked by whom, over which corpus. And I have seen a lot of numbers with no date, which is an elegant way of never going out of date.
The Onyx test solves that simply: it publishes everything. The corpus, the questions, the expected answers, the grading program and the leaderboard with its history. Anyone who doubts our result can download the material and redo the arithmetic. Anyone who doubts a competitor's result can too.
What has to be in place
If you are evaluating AI tools for your company, these are the mechanisms that separate evidence from advertising. None of them requires technical vocabulary.
A public dataset. If the test runs on documents only the vendor has, the result is not verifiable. Ask where the data lives and who else can use it.
A grader that does not belong to the party being assessed. The program that decides whether an answer is right has to be the same for every participant, and preferably written by a third party.
Reproducibility required at the door. A serious leaderboard does not take a file of answers on trust. It asks for the route to redo the test — in the code, if it is open; in the live system, if it is closed.
A date next to the number. A leaderboard is a snapshot of one day. An August position is not valid in December, and repeating an old position without its date is, in practice, saying something false.
The whole score, not the flattering slice. An overall average hides what matters. The breakdown by question type shows where a tool is strong and where it will leave your team stranded.
The test running on the product you will actually use. There is an enormous gap between a lab result and how the live product behaves. Our submission went through the same conversation route customers use every day. Onyx did not settle for the file: it sent 80 questions we had never seen and checked the answers before publishing.
What the result showed
The overall score combines two things: whether the answer was correct and whether it was complete. Broken down, it shows where we went furthest.
1st in answer completeness. Every question comes with a list of facts the answer needs to include, checked one by one. Our answers covered 87.39% of those facts — the highest mark among the 22 systems. For whoever asks, it is the difference between an answer that settles the matter and one that means asking again.
The right documents, in the middle of the mess. The evaluation also checks whether retrieval brought back the documents that held the answer. Our search brought back 86.5% of them; the leaderboard's best on this measurement is 86.55%. Practically a tie, on a corpus built to confuse.
Correct in four out of five answers. The grader judged 81.6% of the answers correct — figures, names and dates matching the expected answer.
The result for each of the 500 questions is published in the leaderboard repository, next to our answers, including the ones we got wrong. New versions and other environments need new measurements; this submission stands as dated evidence.
One last observation about reading leaderboards. An open leaderboard moves: new systems come in and others send new versions. That is why a position always comes with its date — and why I distrust any sales material that shows a ranking without saying when it is from.
Three questions to take to your next meeting
- Who calculated that figure, and where is the data? If the answer is "our internal lab", treat it as a vendor estimate, not a measurement. Ask for the public address.
- What is the date, and how many systems were on the list? A position on its own tells you nothing. Second out of 22 and second out of 3 are completely different results.
- Where did the tool do badly? A vendor that only shows the good scores chose the slice. Ask for the breakdown by question type and read the bottom row first.
A public test does not replace the experience of working with your own files, in your own language, with your team's questions — and that part is free, and takes an afternoon. What the leaderboard does is something else: it separates those who accept being measured by a third party from those who prefer to measure themselves.
