tastebench

Taste-Bench: measuring the long-horizon judgment of LLM agents at real decision forks

**Taste-Bench measures the taste of an LLM agent: its ability to choose the better direction at a real decision fork in a long-horizon task.** Given the task, the trajectory up to the fork, and two candidate next steps, the model must pick the step that the hidden rest of the trajectory proves right. A wrong choice often looks reasonable at the moment and costs the agent most of its budget later. The 502 questions are mined from software engineering and machine-learning research trajectories, with no expert annotation, and the best frontier model answers 59.7% of them correctly.