Small AI models match SIFT algorithm

In developments that can reshape AI efficiency, Eth Zurich researchers have created an algorithm that allows smaller language models to match system performance at 40 times their scale, solving one of the most lasting challenges of AI.

The method, called SIFT (selecting information data for fine-tuning), targets fundamental questions of uncertainty like large language models (LLMS), such as Chatgpt can provide excellent insights and ridiculous answers with the same confidence.

“Our algorithms can enrich the general language model of AI through other data on the relevant topic areas of the question. Combining specific questions, we can extract rich data from model depth and enriched data that are most likely to produce the correct answer.

Despite the ongoing focus on reliability, this breakthrough is at a critical moment even as AI systems are increasingly deployed in the industry. The difference between SIFT and conventional methods is its complex approach to choosing complementary information over redundant data.

Current search methods usually use the “nearest neighbor” method, which tends to accumulate duplicate information that often appears in training data. Consider a two-part query about Roger Federer’s age and child – the nearest neighbor method may flood the results of multiple changes in their date of birth while completely ignoring information about their child.

SIFT instead analyzes the relationship between information vectors in multidimensional space. These vectors – basically arrows pointing in different directions based on semantic relations – allow the algorithm to identify data that supplements the problem rather than copying existing information.

“The angle between vectors corresponds to the correlation of content, and we can use angle selection to reduce uncertainty in specific data.”

This geometric approach to information retrieval may be particularly valuable for professional applications where general AI models lack specific domain knowledge. Andreas Krause, head of the research group and director of the ETH AI Center, noted that the approach “is especially suitable for companies or other users who want to use general AI in areas of expertise that are only partially or not covered at all in AI training data.”

In addition to improving response quality, SIFT also provides a potential solution to another pressing challenge: computing efficiency. The system can dynamically evaluate uncertainty and determine the additional data required for each query, thereby adjusting the computing resources accordingly, rather than always running at maximum capacity.

The team showed that this “test time training” approach allows smaller models to achieve results comparable to the latest systems. In benchmarks, when enhanced by SIFT, the model is 40 times higher than the current leading system.

For investors watching the AI ​​infrastructure space, this development marks a potential shift from race to evolving models. If smaller systems can provide similar results through targeted data selection, the computational requirements may be stable rather than continuing to grow exponentially.

This approach not only improves AI response. Krause suggests that the technology can determine which data points are most important for a particular application: “We can track which enriched data screening choices. They are closely related to the problem and therefore are particularly relevant to the subject area. This can be used in medicine, for example, to study which laboratory analyses or measurements are important for a particular diagnosis and which fewer ones.”

The work “Effective Learning at Test Time: Active Fine Tuning of LLM” was introduced at the International Learning Performance Conference in Singapore recently, and was previously recognized by the “Modern Machine Learning Finetuning” seminar at the Neurips annual conference seminar.

For organizations deploying AI systems, especially in areas of expertise where accuracy is critical, SIFT represents a practical approach to improving reliability without the need for large amounts of computing resources. Researchers have positioned its implementation as a 𝚊𝚌𝚝𝚒𝚟𝚎𝚏𝚝 (Active Survey) library, positioning it as a permutation replacement for standard nearest neighbor search methods.

As AI systems continue to expand to sensitive applications across healthcare, finance, and critical infrastructure, approaches like SIFT can systematically reduce uncertainty while increasing efficiency, which may be essential for responsible deployment. The geometric approach to information retrieval shows that sometimes smarter algorithms can replace original computing power, a lesson that goes far beyond the AI ​​industry itself.

Related