A New Architectural Paradigm for Multimodal Retrieval
In the evolving landscape of vision-language models, the industry has largely relied on repurposing massive generative models to serve as encoders. While functional, this approach is fundamentally inefficient. These legacy pipelines often chain together independent components—a specialized vision tower and a causal decoder—that were never optimized for the retrieval task, leading to significant compute waste and architectural bloat.
H Company has disrupted this standard with the release of NeoMME, a family of 260M and 800M parameter encoders designed from the ground up for high-performance visual document retrieval. By moving to a single-tower, bidirectional architecture, NeoMME eliminates the overhead of separate vision towers and unused generative decoders, effectively compressing the logic of multimodal processing into a unified Transformer.
Unified Multimodal Processing
NeoMME treats both text and image data as first-class citizens within the same transformer stack. Text inputs are processed through an ALBERT-inspired factorized embedding, while visual data is decomposed into raw 32×32 RGB patches. These patches are then projected into the model’s internal representation space via a 2-layer MLP, which is trained alongside the rest of the model rather than being loaded from a pre-existing vision backbone.
The architecture is built for scale, supporting a 16,384-token context window. This capacity allows for the simultaneous processing of two standard 4K UHD images. The model utilizes advanced techniques like grouped-query attention, gated attention, and 2D rotary position embeddings, ensuring that it remains performant even when handling complex, visually dense documents.
Benchmark Performance and Efficiency
NeoMME delivers competitive retrieval results on the ViDoRe v3 benchmark. The 260M model achieves a 0.523 nDCG@10, which matches the performance of significantly larger, legacy-style models like ColQwen2.5 (3.75B parameters) while maintaining a footprint that is 14.4x smaller. The 800M variant pushes this even further, hitting 0.556 nDCG@10.
| Model | Parameters | ViDoRe v3 nDCG@10 | Throughput (2048px Pages/s) |
|---|---|---|---|
| NeoMME-260M | 260M | 0.523 | 51.3 |
| NeoMME-800M | 800M | 0.556 | 21.2 |
| ColQwen2.5-v0.2 | 3.75B | 0.525 | 16.1 |
Deployment and Optimization
H Company has focused on making these models immediately production-ready. All weights are licensed under Apache 2.0, with first-class support in Hugging Face Transformers. Beyond pure accuracy, the project introduces practical storage optimizations for late-interaction indexes, allowing developers to reduce the index size of a page from 1.5 MB down to as little as 6 kB using hierarchical token pooling and asymmetric quantization, all while retaining over 95% of the baseline retrieval quality.
Summary of Innovations
- End-to-End Training: Models are initialized randomly and trained as a masked diffusion denoiser rather than fine-tuning pre-existing weights.
- Architecture Efficiency: By removing the causal decoder and separate vision tower, the model avoids unnecessary compute paths.
- High-Throughput Indexing: The 260M model offers roughly 2x the throughput of comparable baseline models on standard NVIDIA L40S hardware.
- Flexible Quantization: Built-in support for hierarchical pooling allows developers to trade off negligible precision for massive storage savings.
