Benchmarks
We use benchmark to mean a specification of what is measured and how: a taxonomy of elements, a generator that produces questions for each element from templates, and the rules for scoring answers. A dataset is a fixed sample of questions drawn from a benchmark. The datasets we publish are for results that need to be comparable across papers; the generator, available through the API (and soon in autoSTEER), is for fresh questions.
Two angles on economic reasoning
We provide two benchmarks, each taking a distinct angle on economic decision-making.
In strategic settings the best choice depends on how other agents will respond: a phone manufacturer setting its quantity and price has to anticipate its competitors. This is a well-defined and rich area, but a small part of economics as a field.
In non-strategic settings each agent is too small to move the market and takes prices as given: a small farm choosing between wheat and corn at today's prices is not playing against anyone. These settings make up much of microeconomics: consumers, firms, competitive markets and welfare.
The benchmarks are described in the following articles:
| Benchmark | Description |
|---|---|
| STEER | Strategic and behavioral decision-making. It breaks decision theory and game theory into elements of economic rationality: the foundations, choosing alone under certainty and risk, choosing against other agents in games, and choosing on behalf of a group. |
| STEER-ME | Non-strategic microeconomics. It tests whether a model can reason as a price-taking consumer or firm, add individual choices up into markets, find the prices that clear them, and judge whether the outcome is efficient. |
Both are built the same way: elements grouped into modules and settings, questions that vary in type, domain and perspective, and the same scoring. STEER-ME starts from the same foundations as STEER, so the elements they have in common are published once, in STEER.
Key terms
- Element
- One ability, such as applying the law of demand.
- Template
- A question with labeled fields where the numbers go.
- Question
- A template filled in, with options and a correct answer computed by code.
- Type
- The form a question takes within an element.
- Domain
- Its subject matter, such as finance or medicine.
- Perspective
- The point of view it is written from.