Redefining Edge AI Benchmarking
In the current AI landscape, model performance is often judged within the sterilized vacuum of high-end servers. Developers typically evaluate foundation models based on full-precision metrics that offer little predictive power for actual field behavior. Liquid AI has introduced Pipette, an open-source benchmarking suite designed to bridge this divide by measuring on-device AI performance in realistic, resource-constrained environments.
The Shift: Holistic System Evaluation
The core philosophy behind Pipette is that on-device AI efficiency is an emergent property of the entire deployment stack, not just the model weights. The platform shifts the industry focus toward a comprehensive unit of measurement: model + quantization + runtime + hardware. By treating these four pillars as a single configuration, Liquid AI enables developers to understand how specific combinations behave on heterogeneous hardware, such as the MacBook Pro (M5 Max), iPhone 17 Pro, and Galaxy S26 Ultra.
Performance Divergence and Key Findings
Pipette’s release dataset, verified by Artificial Analysis, provides crucial insights into how deployment contexts radically alter the performance of seemingly identical models. Notable takeaways include:
- Context Scaling Disparity: At Q4_K_M quantization on a Galaxy S26 Ultra, the Granite-4.0-H-350M hybrid model retains 78.4% of its decode throughput when scaling from 256 to 4,096 tokens, whereas the non-hybrid Granite-4.0-350M retains only 33.8%.
- The Sparse Activation Trap: While sparse models like the LFM2.5-8B-A1B decode significantly faster (up to 2.6x faster than dense competitors), they still command a large memory footprint because all expert weights must remain resident in RAM.
- Speed-Quality Trade-offs: High throughput does not guarantee superior output. For example, on an iPhone 17 Pro, MiniCPM5-1B outpaces the LFM2.5-1.2B in latency, yet the latter scores higher on standard MATH-500 benchmarks.
Infrastructure and Accessibility
Pipette is fully open-source, providing a suite of tools for diverse development needs, from individual developers to large OEMs. The ecosystem includes:
| Component | Function |
|---|---|
| pipette-mgmt | Manages benchmark infrastructure and submission ingestion. |
| pipette-clients | Executes on-device inference and records performance telemetry. |
| pipette-scores | Handles independent quality evaluations using deterministic metrics. |
Methodology and Transparency
To ensure reproducible results, Pipette enforces a strict testing protocol. Before any measurement, the suite utilizes platform-specific triggers to verify thermal stability and system load. If a device fails these readiness gates, the run is discarded. Evaluations use separate, model-blind scoring on reference H100 systems, ensuring that accuracy metrics remain decoupled from device-specific performance optimizations. With native iOS and Android apps available, as well as a public dashboard, Pipette offers an immediate path for developers to validate their on-device model deployment strategies before moving to production.
