Omega: Structured Mathematical Benchmarks for Detection of Inference Limits in LLMS

Mathematical reasoning summary

Large language models with long COT inference, such as DeepSeek-R1, show good results on Olympic-series mathematical. However, models trained through supervised fine-tuning or enhanced learning depend on limited techniques, such as repeating known algebraic rules or default values ​​to coordinate geometry in graph problems. Since these models follow the learned inference pattern rather than showing true mathematical creativity, they face challenges through complex tasks that require original insights. Current mathematical datasets are ideal for analyzing mathematical skills that RL models can learn. A large-scale corpus integrates a range of mathematical problems, with varying themes and difficulty levels, thus isolating the challenges of specific reasoning skills.

Limitations of current mathematical benchmarks

Current methods (such as distributed generalization) focus on processing different test distributions than training data, which is crucial for mathematical reasoning, physical modeling, and financial forecasting. Composition generalization techniques are designed to help models systematically integrate learning skills. The researchers created datasets through various methods to benchmark math abilities, including hiring humans to write questions such as GSM8K and Minervamath, collecting exam questions such as Aime and Olympiadbench, and scratching and filtering exams Corpora, such as Numinamath and BigMath. However, these approaches either lack sufficient challenges to modern LLM or fail to provide analysis granularity.

Introduction to Omega: Controlled Benchmarks for Inference Skills

Researchers at UC AI2, University of Washington and DModel.AI have proposed Omega, a benchmark that aims to evaluate Boden’s creativity type inspiration to evaluate distributional generalizations in three dimensions. It creates matching training and testing pairs designed to isolate specific reasoning skills in three dimensions: exploratory, compositional, and transformative. Omega’s test and train problems are built using well-designed templates that allow precise control over diversity, complexity, and the specific reasoning strategies required for the solution. In addition, it employs 40 template generators in six math fields: arithmetic, algebra, combinatorial, numerical theory, geometry and logic and puzzles.

Evaluation of Frontier LLM and Enhanced Learning Settings

The researchers evaluated four border models at different levels of complexity, including DeepSeek-R1, Claude-3.7-Sonnet, OpenAI-O3-Mini, and OpenAI-O4-Mini. For RL generalization experiments, the framework applied the GRPO algorithm for 1,000 training problems using the QWEN2.5-7B-INSTRUCT and QWEN2.5-MATH-7B models. Explore the level of complexity in training limited and evaluate higher complexity problems. Composition summarizes training models involving individual skills related to isolation and tests their ability to incorporate and effectively apply these skills. Transformation generalizes training conventional solution approaches and assessing the performance of problems requiring unconventional strategies.

Performance observation and model behavior patterns

Inferential LLMs tend to perform poorly as the complexity of the problem increases, often finding the right solution as early as possible, but spending too much token on unnecessary verification. RL is only applied to low-complexity problems, which can enhance the generalization of medium-complexity problems, while the benefits on in-domain examples are greater than the distribution, indicating the effectiveness of RL in enhancing familiar patterns. For example, in the zebra logic domain, the basic model achieves only 30% accuracy. However, in the absence of SFT, RL training increased performance in in-domain examples by 61 points, while in the case of overscore examples, 53 points.

Conclusion: Promoting Change Reasoning

In summary, the researchers introduced Omega, a benchmark that separates and evaluates three axes of distributed generalization in mathematical reasoning: exploratory, compositional, and transformative. Empirical research reveals three insights: (a) RL fine-tuning significantly improves the performance of distributed and exploratory generalization tasks, (b) RL has limited benefits for constituent tasks, and (c) RL is unable to induce a true new inference pattern. These findings highlight a fundamental limitation: RL can expand the breadth and depth of problem solving, but is crucial to transformational reasoning in making creative leaps. Future work should explore course scaffolding and meta-fusion controllers.

Check Paper, project pages and GitHub pages. All credits for this study are to the researchers on the project. Also, please stay tuned for us twitter And don’t forget to join us 100K+ ml reddit And subscribe Our newsletter.