Why I Revisited My AI Model Selection

When I first designed Document Workspace (working title), I had a rough idea of which AI models I’d use. One primary model, plus a fallback in case something went wrong. My criteria were simple. These were models that had scored well in the public evaluations available at the time. If they’d done well across several of them, I figured they’d be a good choice in a real product too.

Then, partway through building, a thought stopped me.

Every user’s computer is different. Is the model I picked really the best choice in every environment?

Document Workspace runs AI models on the user’s own computer. Some people will have a powerful machine. Others won’t. The same model might run quickly and reliably in one setup and be a struggle to run at all in another.

So is picking whichever model scores highest on an official benchmark enough? I couldn’t let that question go. In the end I decided to build a benchmark tailored to my product, even a simple one.

How I Decided to Compare the Models

First, I divided computers into a few performance tiers. Whether a model can run at all, and how smoothly, depends on the machine. I also decided not to judge models only on how many answers they got right. Document Workspace isn’t a chatbot where what matters is how well it answers general questions. It’s a product that finds and organizes information across multiple documents. So I set the criteria like this:

1. Can it run? Does the model work properly on that hardware tier?
2. Hard Gate. Does it follow the required answer format, run without errors, and resist malicious instructions hidden in documents?
3. Product Fit. How well does it do the work the product actually needs, like answering or summarizing based on the documents? I ranked models by a score weighted for how important each task is.
4. Speed and stability. How long does an answer take, and how often do errors come up during a run?
5. A fallback. I keep a model from a different family on hand in case the chosen one runs into trouble.

The reasoning was straightforward.

No matter how high a model scores on an official benchmark, it’s hard to give users a good experience if it can’t return results in the format a task needs, fails repeatedly mid-run, or makes up content that isn’t in the document and presents it as fact.

And a good result on one computer doesn’t mean you’ll get the same result on every user’s machine.

Results I Didn’t Expect

Some of the results surprised me. At first, one of the newly released models was my leading candidate, since it was also rated well on official benchmarks.

In my own tests, though, it ranked last in my tests. It had the longest processing time and used the most memory and CPU. It also failed far more of the automatic checks that most of the other models passed.

The benchmark also has a step where I check by hand the items that are hard to judge automatically. For example, an AI may seem to have stated something that isn’t in the document as fact, but automatic evaluation alone can’t say for sure. I don’t mark those as right or wrong on a guess. I hold them back and check them myself.

But the model I'd had the highest hopes for had already failed so many automatic checks that the difference was clear before I even needed to review the ambiguous cases.

That alone doesn’t mean it performs worse overall than the others. Results can shift with the test environment, the settings and which items you evaluate. All I can say is that, for my product’s requirements and this round’s conditions, it didn’t deliver what I’d hoped.

Then I Changed the Model Again

I picked my models and kept building. Then, while validating how the product handles PDFs, I ran into a new problem.

A PDF holds more than text. It has images and charts, too. To find what’s needed in a document and generate an answer, the model had to understand and analyze what’s in those as well.

Extracting content from a PDF is one task. How accurately the AI model understands the information in images and charts is a separate question I had to check. So I added another round of validation for it, and the ranking changed again.

Luckily, I didn’t have to go looking for a brand-new model. My first and second choices simply traded places. But that small change meant a lot.

When I first built the benchmark, I thought I’d covered the tasks the product really needed. As development went on, new requirements emerged, and my earlier judgment shifted with them.

Finding the Model That Fits the Product

At first, I was asking which AI model is the best. This experience changed the question a little.

Which model fits best in the product I’m building?

Development is still going, and I’ll run into new problems. When I do, I’d rather judge by actual results than hold on to my first choice.