June 10, 2026
The latest open-weight mixture-of-experts AI models released in 2026 – DeepSeek-V4-Pro, MiMo-V2.5-Pro, MiniMax-M3, and GLM-5.2 – can now run on a single liquid-cooled Comino Grando system equipped with eight NVIDIA H200 GPUs and 1,128 GB of pooled VRAM.
In practical terms, this enables organizations to deploy advanced AI models on their own infrastructure, ensuring fast response times, high throughput, and complete control over their data.
• Responsive for real-world use: In live sessions with a warm cache, the first token is generated in 2-3 seconds. Subsequent text generation proceeds at 80-87 tokens per second with speculative decoding, exceeding typical reading speeds.
• Built to support teams: Under typical workloads, a single node sustains approximately 1,200 tokens per second in aggregate, or about 100 million tokens per day.
• Full models at FP8 precision: The 1,128 GB of pooled VRAM in an 8× NVIDIA H200 configuration supports running any of these flagship models at FP8, the standard precision for AI model serving. The system also accommodates real-world context lengths and multiple concurrent users.
• Private and sovereign: All tokens are generated within the organization’s infrastructure. No data leaves the premises, ensuring full control over privacy, security, and data sovereignty.
The Comino Grando RM is a fully liquid-cooled 4U server designed for standard 19-inch racks or desktop use. It is available in 4× and 8× H200 configurations on a single-socket platform. The 8× configuration described below was used for all benchmarks in this article.
At FP8, the four flagship models discussed here require 700-960 GB for their weights. The 8× H200 configuration offers 1,128 GB of capacity, supporting these models with extra capacity for KV-cache and multiple concurrent users. This matches the operating range tested in these benchmarks.
Customers frequently ask whether a liquid-cooled Comino Grando can run the flagship AI models of 2026 in-house at practical speeds.
To provide measurable results, we tested the four largest open-source MoE models of 2026 on a single liquid-cooled Comino Grando equipped with 8× H200 GPUs. Tests included cold-start, warm prefix-cache, and speculative-decode modes, using context lengths from 8k to 262k tokens.
* Kimi K2.6 was reviewed but not benchmarked in this analysis. Its dense MLA design includes a substantial ~327 GB KV-cache, placing it at the “heavy end” of the spectrum.
An AI model’s ability to fit in GPU memory (VRAM) depends on three main factors, prioritized as follows:
1. Weights. These are the primary and largest consumers of memory. At FP8, models require 700–960 GB for weights. If weights do not fit in memory, the model cannot run.
2. Multi-GPU overhead. Distributing a model across eight GPUs requires additional memory for activations, communication buffers, and framework reserve. This overhead is significant, ranging from tens to over a hundred gigabytes, and is often overlooked in capacity planning.
3. KV-cache. Memory usage increases with context length and the number of concurrent users. This component is most affected by the model’s attention architecture. Like weights, KV-cache can be optimized using quantization and paging.
So, whether a model fits in memory depends first on its weights, followed by multi-GPU overhead, and then the KV-cache. The KV-cache is also where 2026 AI model architectures differ most as context sizes and user concurrency increase.
The table below offers an approximate FP8 memory overview to highlight differences between these models. Actual deployments also include the multi-GPU overhead described above.
Of the four models, DeepSeek-V4-Pro uses the least cache memory, allowing for longer context windows and more concurrent users per system. It is the preferred option when maximizing context length and user concurrency per node.
Engine: SGLang. All data are based on the 8× H200 node.
Responsiveness (time to first token):
Generation speed (around 50 tokens per second for the larger models) remains almost the same regardless of context length. The main delay comes during the prefill stage, when the model processes the input prompt. This means short prompts are answered within seconds, while very large contexts require more time to process before generation begins, making them better suited for batch workloads than interactive conversations.
For larger models, single-stream throughput of about 50 tokens per second is mainly constrained by GPU interconnect bandwidth, not compute power or memory. With 8-way tensor parallelism, all eight GPUs must synchronize for each generated token, making communication throughput the main performance bottleneck. Models with lighter activations, such as MiMo (~93 tok/s) and MiniMax (~101 tok/s), are less affected because they activate fewer parameters per token. Much of this additional performance can be achieved through software optimization. Speculative decoding (Multi-Token Prediction / EAGLE) alone can provide significant improvements:
* Baseline vs speculative decode – DeepSeek-V4 49.8→81.0 tok/s (+63%), MiMo-V2.5 62.9→86.7 tok/s (+38%)
20–40% of the decoding speed can be gained through software optimization rather than hardware upgrades.
Cold-start results reflect only the initial request. In practice, during ongoing chat sessions, Retrieval-Augmented Generation (RAG) with a shared system prompt or repeated queries, the prefix cache is typically warm. With an 80% prefix cache at a 32k context, the system achieves the following performance:
* Cold vs warm time-to-first-token at 32k context – all four models drop to 0.5–3.2 s warm
In live dialogue, the first token arrives within 2–3 seconds. This reflects the typical user experience and provides a more relevant measure than cold-start results.
As concurrent users increase, overall throughput rises until it reaches a performance ceiling and then stabilizes. The following results were measured on DeepSeek-V4:
* Throughput vs concurrency for DeepSeek-V4 – aggregate output climbs to ~1,199 tok/s at 256 streams (~100M tokens/day) then latency collapses at 512
The practical performance limit is approximately 256 concurrent streams, delivering about 1,200 tokens per second, or 100 million tokens per day (around 3 billion per month). Exceeding this limit results in significant latency. In practice, a single node can reliably support 10-15 users with short queries simultaneously, or 3-6 users working on long-context tasks. The system maintains stability under load by distributing available throughput among users.
* These figures are based on concurrent request streams generated by a load-testing tool, which is the industry standard for measuring multi-user capacity. Per-stream results reflect the performance experienced by each user.
Every token described above is generated entirely within your own infrastructure. No prompts, documents, or customer data are sent through a third-party API. For a growing number of organizations, this is no longer just a preference — it is a business and regulatory requirement:
• Regulated industries such as finance, healthcare, legal, and the public sector require that sensitive data not be processed in foreign cloud environments.
• Sovereign and national AI initiatives across the EU and beyond that require both data and computing resources to remain within national borders.
• R&D and other sensitive projects require that prompts, outputs, and AI models remain confidential.
A Comino Grando 8× H200 makes private deployment of frontier AI models a practical reality. It runs full models at FP8, with full context and full throughput, while keeping all data and AI processing entirely under your control.
At this point, liquid cooling is more than a technical feature. Each Comino Grando is fully direct-liquid-cooled, covering the CPU with VRM, full-cover GPUs, and HBM memory. An independent controller with self-diagnostics and protection mechanisms manages the cooling system, enabling the server to regulate its own thermal conditions without relying on large-scale room cooling.
This design expands the environments where each configuration can be deployed:
• The 4× H200 build can live where the data is generated – in a laboratory, clinic, design studio, or branch office. It requires no hot/cold-aisle containment or large banks of high-noise fans to remove several kilowatts of heat. Its quiet, compact design allows it to operate close to users, bringing the model to the data instead of sending the data to the cloud.
• The 8× H200 build is a rack machine designed as a high-throughput node for a server room or on-premises rack. It remains fully liquid-cooled, sovereign, and on-premises, while offering the size, power, and performance of a dedicated enterprise server rather than a desk-side system.
Liquid cooling is particularly important for LLM inference because the decoding phase relies heavily on HBM memory bandwidth, and HBM performance is sensitive to heat. By keeping HBM operating within its specified temperature range during sustained workloads, direct liquid cooling helps maintain both inference performance and long-term system stability.
Speed and capacity become meaningful when they support real business use cases. At approximately 100 million tokens per day, a single Grando can reliably power:
• A private AI assistant for the entire team – employees can query a frontier model using internal documents throughout the day, without per-seat API costs and without organizational data leaving the company.
• Token-intensive AI agents at scale – including workloads that consume large volumes of tokens, such as reviewing code, summarizing contracts, processing support tickets, and generating reports in overnight batches. In practice, a small group of engineers running coding agents continuously, 24/7, can fully utilize the system.
• A copilot for proprietary data – providing answers based on your codebase, knowledge base, and customer history through a model that never connects to the public internet.
In practical terms, one system is intended to support a team, department, or the internal needs of a mid-sized organization. It is not designed for reselling public API capacity at hyperscale. Within its intended scope, it can replace recurring cloud costs with a fixed, private, and sovereign infrastructure asset.
Frontier AI models in 2026 no longer require hyperscale cloud infrastructure. A single liquid-cooled Comino Grando with 8× H200 GPUs and 1,128 GB of pooled VRAM can run these models privately at FP8, at production speed, and on-premises, eliminating the need for a full data center build-out. By choosing a model that fits available memory and maintaining a warm cache, one system can support an entire team.
Configure:
Grando 4× H200 – https://configurator.grando.ai/configurator/28
Grando 8× H200 – https://configurator.grando.ai/configurator/27
