STEER

Benchmarks for the economic reasoning of language models

Welcome

Language models are increasingly asked to make economic decisions, or to advise the people who make them. STEER is a way to check whether a model is up to it. Instead of one overall score, it breaks economic reasoning into small, named abilities, such as computing an expected utility, finding a dominated strategy, or applying the law of demand, and tests each one separately.

Across both papers, we observed that models' mistakes often sound right, that bigger models are not uniformly more rational, and that performance can be brittle depending on the context. For more, read the pages for STEER and STEER-ME.

New here? Start with the Tutorial.

Tip

The benchmarks are large, and every question can be sampled again and again with fresh values. We publish a fixed test set and results for many models, but producing them was an undertaking, and a single score was never the point: STEER is about diagnosing where a model fails, on the parts that matter for your use. Our recent work, QuickScope, does that adaptively, searching for where a model is reliably wrong without evaluating everything.

Cite

If you use STEER or STEER-ME, please cite the paper for the benchmark you used.

STEER

Raman, Lundy, Amouyal, Levine, Leyton-Brown and Tennenholtz. STEER: Assessing the Economic Rationality of Large Language Models. ICML 2024. arXiv:2402.09552

@inproceedings{ramansteer,
  author = {Narun Krishnamurthi Raman and Taylor Lundy and Samuel Joseph Amouyal and Yoav Levine and Kevin Leyton{-}Brown and Moshe Tennenholtz},
  title = {{STEER:} {A}ssessing the {E}conomic {R}ationality of {L}arge {L}anguage {M}odels},
  booktitle = {Forty-first International Conference on Machine Learning, {ICML} 2024, Vienna, Austria, July 21-27, 2024},
  publisher = {OpenReview.net},
  year = {2024},
  url = {https://openreview.net/forum?id=nU1mtFDtMX}
}

STEER-ME

Raman, Lundy, Amin, Perla and Leyton-Brown. STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models. NeurIPS 2025, Datasets and Benchmarks Track. arXiv:2502.13119

@inproceedings{raman2025steerme,
  title = {{STEER-ME}: Assessing the Microeconomic Reasoning of Large Language Models},
  author = {Raman, Narun and Lundy, Taylor and Amin, Thiago and Perla, Jesse and Leyton-Brown, Kevin},
  booktitle = {Advances in Neural Information Processing Systems},
  volume = {38},
  year = {2025},
  note = {Datasets and Benchmarks Track},
  url = {https://arxiv.org/abs/2502.13119}
}