Supermicro, IBM and NVIDIA’s 56× claim concerns time to first token at a 130,000-token prompt in a vendor benchmark. It does not establish that an entire answer arrives 56 times faster, or that an enterprise can cut its GPU bill by that amount. The useful question is how much repeated context a serving system can retrieve instead of recomputing.
Supermicro’s November 5 announcement describes a jointly tested reference architecture. The supporting IBM Redbook supplies the benchmark conditions. Together they make a case for testing persistent shared KV cache, while leaving workload-specific economics unproven.
Key takeaways
- The 56× figure is a first-token latency comparison for one tested long-context case.
- Shared flash storage adds a reusable cache tier; it does not replace GPU compute or offer HBM-like latency.
- The concurrent test included an initially empty cache, but its request mix still determines the result.
- A purchase needs measured reuse, miss costs, tail latency and a complete hardware-and-software quote.
Where shared storage fits
The design combines Supermicro Petascale all-flash servers using AMD EPYC 9005 processors, IBM Storage Scale, NVIDIA Dynamo and Spectrum-X Ethernet. The vendors’ mechanism is to persist reusable key-value states outside local GPU memory, then fetch matching context rather than repeat its prefill computation. That benefit depends on compatible previously computed context being available.
The Redbook calls shared enterprise storage G4. Its hierarchy puts GPU HBM at G1, system DRAM at G2, local NVMe at G3 and NVIDIA CMX pooled flash at G3.5. These tiers have different latency, capacity and persistence roles. G4 extends the places a cache can live; the GPU remains responsible for inference computation.
The tested storage cluster had eight ASG-1115S-NE316R nodes, each with 16 Micron 7.68 TB E3 drives and an AMD EPYC 9535 processor. Five heterogeneous clients and three Spectrum-X SN5600 switches completed the setup. Inference ran from one SYS-212GB-FNR client with four RTX PRO 6000 GPUs; the first-token test itself is identified as a single-GPU benchmark.
The serving stack used Storage Scale ECE 6.0.0.1, Dynamo 0.9.0+, vLLM 0.14.1 and AIPerf with openai/gpt-oss-120b. Model weights used MXFP4 and the KV cache used FP16. These details matter when deciding whether a prospective deployment resembles the tested configuration.
Reading the 56× figure correctly
Without cache, the Redbook reports first-token latency increasing from 0.572 seconds at 10K tokens to 32.14 seconds at 130K. With the shared cache, it gives a 0.193–0.57-second range across tested prompt lengths and reports 56× at 130K. Dividing 32.14 by the rounded 0.57 upper value yields approximately 56.4. The prose does not separately label that upper value as an exact 130K endpoint.
The press release expresses a different comparison: as prompt length increased from 30K to 130K, latency increased by 29.55 seconds without cache and by 0.377 seconds with cache. Those are changes over the interval, not absolute latency at 130K. Dividing those two increases does not test the Redbook’s endpoint ratio.
| Test | Reported comparison | Scope |
|---|---|---|
| Single-request TTFT | 56× at 130K tokens | First token, not complete response |
| Concurrent requests | 0.19 → 4.26 requests/second | 200 requests, 28 connections |
| Competing network traffic | 0.19 → 3.6 requests/second | One disclosed contention scenario |
Source: IBM Redbook, vendor-authored benchmark. Results are specific to the disclosed system and request mix.
What the throughput tests add
The concurrency test sent 200 requests through 28 connections, randomly drawing from 100 unique prompts representing 24 million tokens and 825 GB of KV-cache data. The G4 cache began empty, and hits accumulated during the run. Its reported average therefore includes cold start.
The authors report 4.26 requests per second with Storage Scale versus 0.19 without cache, calling the improvement 22×. Total processing time fell from 1,048 to 46.94 seconds. This demonstrates the outcome for that request mix, without establishing how often another organization’s prompts will hit the cache.
Four other clients then generated continuous competing network I/O of 200 GB/s. The storage-backed result was 3.6 requests per second and 55.56 seconds for the 200 requests; the authors describe an 18× throughput improvement over the no-cache baseline. Their stated 18% throughput loss against the clean run is not reproduced by the displayed rounded rates, which imply about 15.5%. The paper does not reconcile that difference.
The eight-node cluster is also a tested configuration, not a universal minimum purchase. The Redbook’s sizing discussion spans three to 256 storage servers and considers usable capacity, read bandwidth, erasure-coding overhead, concurrency, context-window size and GPU-client count. An inference team should therefore resist copying the benchmark’s hardware list without matching those requirements to its own workload. The vendors also claim faster deployment through factory integration and on-site services, but the released benchmark does not measure that deployment-time promise.
What an inference buyer should measure
A pilot should reproduce the buyer’s model, context lengths and repetition pattern, then measure hits, evictions, fetch latency and miss costs. First-token and complete-response tail latency need separate attention. Failure and recovery behavior, sustained throughput and performance under competing traffic also need testing beyond the disclosed scenario.
The sources provide no system quote, license price, measured GPU-hours avoided or buyer-specific cost per token. Price storage servers and flash, Storage Scale licensing, networking, power, cooling, redundancy, monitoring, retention, security and integration together. The storage tier’s stated 315 GB/s headroom is not measured application throughput: the documented client-facing configuration offered up to 80 GB/s.
For repetitive long-context workloads, shared G4 storage could shift some cache capacity and spending toward flash, fabric and operations. Whether that defers GPU purchases remains a question for the pilot. This is a vendor-authored, jointly validated study; the reviewed materials contain no independent reproduction.
Related: AI News Fab’s earlier KV-cache coverage addresses a separate model-level footprint and cost claim.
Sources and image disclosure
Primary sources: Supermicro announcement; IBM Redbook; Redbook PDF. Source refresh: November 10, 2026. Both images are AI-generated editorial illustrations, not photographs of the tested installation or benchmark evidence.