Contrasting language image pre-training (editing) becomes important for modern vision and multi-modal models, enabling applications such as zero-camera image classification and visual encoder in MLLM. However, most editing variants (including meta-editors) are limited to English-only curated data, ignoring the large amount of non-English content in the global network. Scaling clips to include multilingual data has two challenges: (a) the lack of effective ways to curate non-English data at scale, and (b) the decline in English performance when adding multilingual data (also known as the Curse of Multilingual). These problems hinder the development of a unified model for optimization of English and non-English tasks.
Methods such as OpenAI editing and meta editing depend on English-centric curation, and the distillation-based approach introduces bias from external teacher models. Siglip and Siglip 2 try to take advantage of data from Google Image searches, but their dependency on proprietary sources limits scalability. Multilingual clipping models (such as M-CLIP and MCLIP) use distillation technology, using English-only clips as visual encoders and trained multilingual text encoders with low-quality data. Furthermore, hybrid approaches such as slip and ignition combine language supervision with self-supervised learning (SSL) to balance semantic consistency and visual representation. Despite these efforts, none of these approaches address the core issues.
Researchers from META, MIT, Princeton University and New York University have proposed Meta Clip 2, the first method to train clip models from scratch using local global image text pairs without relying on external resources such as private data, machine translation or distillation. It eliminates the performance trade-off between English and non-English data by designing and co-scaling metadata, data curation, model capacity, and training. META Clip 2 maximizes compatibility with OpenAI Clip architecture, thus ensuring the universality of the clips and their variants. In addition, its recipe introduces three innovations for scaling the world: (a) scalable metadata across more than 300 languages, (b) per-language curation algorithms for balancing concept distribution, and (c) advanced training frameworks.
To address the first challenge, the researchers used globally curated data, and to solve the second challenge, they developed a global clip training framework. The framework follows the training setup and model architecture of OpenAI and Meta clips, including three additions: multilingual text tokens, scaling of visible training pairs, and analysis of minimum viable model capacity. To ensure generalization, the training sets VIT-L/14 models of the Meta Clip using OpenAI clips and has been modified in multiple languages. Furthermore, studies on minimal model expressivity show that even OpenAI’s VIT-L/14 fights the curse even due to limited capabilities, VIT-H/14 is a turning point, achieving significant growth in both English and non-English tasks.
Meta Clip 2 outperforms its English (1.0×) and non-English (1.3×) and non-English (1.3×) tasks in English and multilingual tasks, when training global data and scaling visible pairs at VIT-H/14. However, the curse persists in non-scale settings or in smaller vit-l/14 (e.g. vit-l/14). Transitioning from English-centric metadata to global equivalents is essential. For example, deleting an English filter on Alt-Texts results in a 0.6% reduction in the accuracy of the imaging network, highlighting the role of language isolation. Replacing English metadata with merged global metadata initially reduces English performance but enhances multilingual skills. Evaluation of zero-shooting classification and a small number of geolocation base benchmarks suggests that proportional proportions from 13B English to 29B globally can improve results, in addition to saturation performance in GEODE.
In short, the researchers introduced Meta Clip 2, the first editing model trained from scratch on global image text pairs. It shows that scaling metadata, curation and training abilities can break the “multi-linguistic curse” and thus bring mutual benefits to English and non-English performance. Meta Clip 2 (VIT-H/14) outperforms its English-only English-only on zero cameras (80.5% → 81.3%) and has a multilingual benchmark for a single unified model such as XM3600, Babel-In and CVQA. By opening up its metadata, curatorial approaches and training code, Meta Clip 2 enables the research community to move beyond an English-centric approach and possess the potential of a global multimodal network.
Check Paper and Github page. Check out ours anytime Tutorials, codes and notebooks for github pages. Also, please stay tuned for us twitter And don’t forget to join us 100K+ ml reddit And subscribe Our newsletter.
Sajjad Ansari is a final year undergraduate student from IIT Kharagpur. As a technology enthusiast, he delves into the practical application of AI, focusing on understanding AI technology and its real-world impact. He aims to express complex AI concepts in a clear and easy way.
