one New research Munich LMU Munich researchers, Munich Machine Learning Center and Adobe Research reveal weaknesses AI Language Model: They strive to understand long documents in ways that may surprise you. The research team’s findings show that even state-of-the-art AI models cannot rely on simple word matching, so it is difficult to connect information when connecting it.
Hidden Problems of Artificial Intelligence Reading Skills
Images try to find specific details in long-term research papers. You might browse it, creating a psychological connection between the different sections to piece together the information you need. It turns out that many AI models are simply unavailable. Instead, they usually rely heavily on finding exact word matches, similar to using CTRL+F on computers.
The research team developed a new benchmark called Nolima (no literal matching) to test various AI models. The results show that when the AI model processes text with more than 2,000 words, its performance is greatly reduced. When they reach 32,000 words (about the length of a short book), most models perform halfway. This includes testing the main model GPT-4O,,,,, Gemini 1.5 Proand Camel 3.3 70b.
Medical researchers consider using AI to analyze patient records, or legal teams using AI to review case files. If the AI misses critical connections because the relevant information uses different words than the search query, the consequences can be huge.
Why is the word matching not enough
Current AI processes text using something called attention mechanisms. The system helps AI focus on different parts of the text to understand the relationship between words and thoughts. This works well when using shorter text. However, research shows that this mechanism becomes overwhelming as the text grows, especially when it cannot rely on exact word matching.
The Nolima test reveals this limitation by asking the answers to an AI model question that requires understanding the context rather than finding matching words. The results show. Although the models perform well with short texts, they greatly reduce the ability of these connections as text length increases. Even specialized models dedicated to inference tasks have a precision score of less than 50% when working with longer documents.
No word matching crutches, AI model efforts:
- Connect related concepts using different terms
- Follow a multi-step reasoning path
- Find relevant information when it appears after a critical context
- Ignore misleading word matching in unrelated parts
The numbers tell the story
The results depict how AI models handle sharp images of longer texts. GPT-4O shows the strongest performance, maintaining validity up to about 8,000 tokens (approximately 6,000 words). However, even this highest performance showed a longer text drop. Most other models, including the Gemini 1.5 Pro and the Llama 3.3 70B, experienced sharp performance between 2,000 and 8,000 tokens.
Performance decline becomes more pronounced when the task requires multiple inference steps. For example, if a model needs to establish two logical connections – for example, knowing that the character lives near a landmark, and the landmark is in a specific city – the success rate is greatly reduced. Research shows that such multi-step reasoning becomes particularly challenging in texts with over 16,000 tokens, even with techniques designed to improve reasoning, e.g. Tips after thinking chain.
What makes these findings particularly noteworthy is that they challenge claims about the ability of AI models to handle novels. Although many models are advertised in a wide context window, Nolima’s benchmarks suggest that effective understanding will be reduced until these theoretical limitations are met.
Source: Modarressi et al.
When AI misses the forest of trees
These limitations have serious implications for how we use AI in real-world applications. Consider a legitimate AI system that searches through case law. It may miss relevant precedents simply because they use different terms than search queries. Instead, the system can focus on fewer related cases of sharing more words with search terms.
The impact on search and document analysis is particularly concerning. Current AI-driven search systems often rely on a technology called Search Authorized Generation (RAG). Even if these systems successfully retrieve documents containing the correct information, the AI may not recognize its relevance if the wording is different from the query. Instead, AI may tend to share less relevant documents with surface-level similarity with search terms.
For AI users, these findings raise some important considerations:
The firstshorter queries and documentation may produce more reliable results. When using longer text, breaking them down into smaller, focused segments may help keep AI performance.
secondUsers should be particularly careful when requiring AI to establish connections across different parts of a document. Research shows that when AI models need to piece together information from different parts, especially when connections are not obvious through shared vocabulary.
at lastThese limitations highlight the ongoing importance of human supervision. Although AI can be a powerful tool for processing and analyzing text, it should not be used as the only means to determine important connections in long or complex documents.
These findings remind you that despite rapid advances in AI technology, these systems are still very different from what humans process information. Understanding these limitations is still crucial to the effective use of AI tools and understanding human judgment.
What will happen next
Understanding the limitations of current AI models’ ability to handle long-term texts opens up important issues regarding the future of AI development. Research behind Nolima Benchmark shows that our current approach to AI text processing may require a lot of improvement, especially in how models handle information in longer paragraphs.
The current solution shows only partial success. Chain tips that encourage AI models to divide their inference into steps help improve performance. For example, when using this technique, Llama 3.3 70b shows better ability to handle longer contexts. However, this approach is still lacking when dealing with texts of over 16,000 tokens, which suggests that we need more basic solutions.
The attention mechanism forms the backbone of the current AI model for processing text and needs to be rethinked. Thinking about it is like trying to hold a conversation in a crowded room – the longer the conversation takes, the harder it is to keep track of all the important points mentioned above. Our current AI model faces similar challenges, but larger in scale.
With a future in mind, researchers are exploring several promising directions. One approach involves new ways to organize and prioritize information for AI in long texts, rather than simple word matching to understand deeper conceptual connections. This may be more like a psychological map of how humans create information based on meaning rather than just sharing vocabulary.
Another area of development focuses on improving how AI models deal with what researchers call “potential hops” – the logical steps required to connect different information. Current models have difficulties with these connections, especially in longer texts, but new architectures may help bridge this gap.
For those who use AI tools today, these findings suggest several practical approaches:
When considering using AI, consider breaking down longer documents into meaningful segments. This helps create logical parts that retain important backgrounds. For example, if you analyze a research paper, you can put the method and the results part together because they usually contain relevant information.
When asking AI to analyze longer text, specify the connection you want to make. Rather than asking a wide range of questions, guide AI to explore specific relationships that interest you. This helps to make up for the current limitations of the model when establishing these connections independently.
Perhaps most importantly, maintain realistic expectations about the capabilities of AI. Although these tools are very helpful for many tasks, they should not be considered as a complete replacement for human analysis of complex documents. Humans’ ability to maintain context and establish cross-text conceptual connections are still superior to current AI capabilities.
Advances in AI development in this area are both challenging and exciting. When we understand these limitations better, we can work towards truly understanding long texts rather than just processing them. Before that, using AI effectively means dealing with its current limitations while appreciating its advantages.
