Understand this story
Ai2 (Allen Institute for AI) released the StateBench benchmark on November 4, 2026, to evaluate the ability of language models to track changes in entity information across long documents. The benchmark examines model performance across four difficulty levels: r-trivial, aperiodic, soluble, and general finite. According to an Ai2 blog post, when trained directly on these tasks, recurrent and hybrid architectures generalize far beyond Transformers; meanwhile, frontier models such as Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and Luna still fail on the harder state tracking tasks as input length grows. The open pretrained models tested include the 7B Olmo 3 and Olmo Hybrid, as well as 275M and 2.7B parameter models, all trained with contexts of no more than 64,000 tokens. Unite.AI reported the same day that the accuracy of the aforementioned frontier models on the soluble and general finite tasks drops sharply as input length grows, with the longest general finite cases scoring only 13% or lower; the dataset contains 68,000 rows totaling 1.05 GB, and the code is open-sourced under the Apache-2.0 license.
AI-generated from public reports
Latest development · Report
Ai2 releases StateBench to measure long-context state tracking capabilityDevelopments
1 confirmed developments
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 4
Unite.AI
Ai2 releases StateBench to measure long-context state tracking capability
The Allen Institute for AI (Ai2) has released StateBench, using a four-tier benchmark, dataset, and code to test whether language models can track changing variables in long documents. In testing, Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna saw accuracy plummet on soluble and general finite tasks as input length increased, with the longest general finite test cases scoring only 13% or lower. The dataset contains 68,000 rows totaling 1.05 GB, and the code is open-sourced under the Apache-2.0 license.
Ai2 BlogFirst-party
Ai2 Releases StateBench: Testing How Language Models Track State Changes in Long Documents
Ai2 has launched StateBench to evaluate long-context state-tracking capabilities, examining how well models track changes in entity information across four difficulty levels: r-trivial, aperiodic, soluble, and general finite. When trained directly on these tasks, recurrent and hybrid architectures generalize far beyond Transformers; meanwhile, frontier models such as Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and Luna still fail on harder state tracking as inputs grow longer. The open pretrained models tested include the 7B Olmo 3 and Olmo Hybrid, as well as 275M and 2.7B parameter models, all with training contexts of no more than 64,000 tokens.
Ai2 BlogRelated discussion
Ai2 Blog joined this discussion
This post contributes to heat; it is not a formal report or confirmed fact.
Hugging FaceRelated discussion
Hugging Face joined this discussion
This post contributes to heat; it is not a formal report or confirmed fact.
Heat trend for this story
- Current
- 20
- Period peak
- 29 Nov 4 03:00
- 24-hour change
- —
Move the pointer or select the chart to inspect hourly heat. Use the left and right arrow keys with a keyboard.