Cost per Correct Answer: A Better Way to Compare AI Model Pricing

Imagine that Model A costs half as much per request as Model B. It sounds like the obvious choice until the work begins: Model A occasionally omits required fields, sometimes invents references and needs extra prompts to fix formatting. A business that pays for two retries and five minutes of human checking may spend more than it would on a stronger first attempt. The better measurement is cost per accepted answer, with acceptance defined before either model runs.

Pricing pages also distinguish input, output and often cached input. Long documents, extended reasoning and tool calls can change the actual charge. Provider pricing differs by model and service mode, so a fair comparison should use invoice or usage data from the same workload rather than two headline rates copied from marketing pages.

Take one hundred representative tasks and write down what success means. For an invoice extractor, the required fields might be vendor name, date, currency and a correctly calculated total. Run both candidates with the same source material and count the tasks accepted without manual repair. Add costs for retries and external services, then divide that total by the number of accepted tasks. In a hypothetical experiment, $4 spent across eighty accepted answers means five cents per accepted answer; $6 across ninety-five accepted answers is about 6.3 cents. Neither result automatically proves which model is better because the quality and risk of the remaining failures may differ.

For user-facing agents, track time to first useful answer and how often a human must intervene. For coding tools, replace 'answer' with 'accepted patch that passes review and tests'.

A small mistake in a personal to-do list is not the same as an incorrect financial statement or an accidental account change. Critical workflows should use weighted failure categories instead of treating all errors equally. Measure fabricated citations, incorrect calculations, leaked data, unauthorized tool use and failures to ask for clarification. Make the test repeatable: save the prompt, model version, date, configuration and scoring rules.

Artificial Analysis separately reports quality and efficiency measures, which is a useful reminder that one score does not represent every dimension. RecoupRev has not conducted an independent price benchmark for the models described here. The goal is a method readers can run on their own work, rather than an invented price-versus-quality leaderboard.