Benchmark
* Compared to the best-performing standard model, GPT-5.5, as of June 2026.
To compare VibeSolve with standard AI tools, we evaluated its capability to generate a functioning optimisation solver. The input is a plain-English description of the problem. The expected output, in addition to functioning source code, is a solver API deployed using Docker. As with all LLMs, results may vary.
We tested the performance with 10 inputs and ran each 5 times across various LLMs, for a total of 50 runs. All VibeSolve results used the default setup: GPT-5 Mini as the underlying model and up to 10 iterations in the automated fixer loop. VibeSolve can be configured to use different base models and to set a different maximum iteration count. Be aware that different configurations may affect both benchmark results and the average cost per run. In addition to testing the solver functionality, we checked the required token cost.
| Metric | Description |
|---|---|
| Compilation | The generated project compiles successfully. |
| Solver Execution | The solver runs for at least 15 seconds without crashing. |
| Quarkus Startup | The Quarkus application starts successfully. |
| Endpoint Validation | The application’s Quarkus endpoints respond correctly. Partial success is counted as 0.5, complete success as 1.0. |
| Docker Deployment | The complete application starts successfully using the provided Docker configuration. |
Cost comparison
| VibeSolve | Opus 4.8 | GPT-5.5 | GPT-5 | GPT-5 Mini | Haiku 4.5 | |
|---|---|---|---|---|---|---|
| Total Price of Benchmarks (50 runs)* | $7.64 | $23.78 | $25.77 | $9.08 | $1.46 | $3.35 |
| Success Rate** | 92% | 2% | 18% | 0% | 0% | 0% |
*Based on VibeSolve’s default configuration and model pricing as of June 2026. It includes the cost of all 50 runs, both successful and unsuccessful.
**Measured using the Docker Deployment metric.