Paper · November 4, 2026
On a unified-memory device, the model and every agent's history share one memory. Hibana co-designs the model, its memory, and its decoding so many agents run fast on a single DGX Spark.
1Cornell University2University of Illinois Urbana-Champaign3Independent researcher4National Yang Ming Chiao Tung University*Equal contribution
Building Hibana · 27.9 tokens/s
Qwen3.8-Flash-Next on one DGX Spark, against the fastest of vLLM, llama.cpp, and TensorFold.
Local agents extend long histories, pause for tools, and run side by side. On a unified-memory device such as the DGX Spark, the weights, the active requests, and every saved history share one 128 GB pool, and every generated token streams the weights through limited bandwidth. Hibana attacks both limits at once.
Model surgery. Edit the trained model to fit the hardware: 3-bit weights, INT8 recurrent state, and a 2-bit draft head, so each token reads fewer bytes.
Unified-memory management. Keep every agent's state across turns, spilling it to the SSD when GPU memory runs out, so a returning agent resumes instead of recomputing.
Adaptive speculation. Share experts across drafted tokens and draft only as far as acceptance pays for, so each weight read yields more tokens.
Decoding one request is bound by memory bandwidth, so the fastest token is the one that reads the fewest bytes.
Each group of 32 weights snaps to an 8-entry FP8 codebook, so each weight is stored as a 3-bit index. The kernel looks up the codes and feeds FP8 tensor cores directly.
Eight values per group, three bits per weight. Codes take 3 bits and each group of 32 adds one 8-bit scale: 3 + 8/32 = 3.25 bits per weight. The codebook here is fitted to the group for illustration.
The drafter's output head scores every token in the vocabulary and is the largest matrix the drafter reads. A 2-bit copy is cheap but noisy, so Hibana uses it only to shortlist candidates, then rescores the shortlist in FP8.
Shortlist in 2 bits, decide in FP8. Context: for i in range(. The served head shortlists 16 of over 100,000 tokens and agrees with the FP8 head on 99.76% of tokens.
Agents take turns. By the time an agent returns, other agents have usually pushed its state out of GPU memory.
Discard an evicted agent's state and its next turn recomputes the whole context. Hibana spills the KV cache, recurrent state, and draft state to the SSD instead, and reads them back when the agent returns.
Spill, don't discard. Illustration with 8 agents and room for 4 in GPU memory. On Qwen3.8-Flash-Next with 192 conversations, the SSD tier lifts cache hits to 80–98% and cuts time to first token by up to 42× against the baselines.
A drafter proposes several tokens and the target checks them in one pass. Hibana makes each of these rounds cheaper and sizes it to what will be accepted.
In a mixture-of-experts layer, verifying drafted tokens loads every expert any of them picked. Hibana gives the round a shared expert budget, so it reads fewer weights.
Fewer experts per round. Toy example: 6 tokens, 12 experts, top-2 routing. Shaded columns are the experts read from memory; ↺ marks a second choice replaced by one inside the budget.
A longer draft can yield more tokens per round, but each extra draft token costs time to draft and verify. Hibana picks the draft length that maximizes expected tokens per unit time:
Draft only as far as acceptance pays for. Idealized model. Bars show E[Nd] / T(d) for each draft length; the red bar is d*. High acceptance earns long drafts.
Four models on one DGX Spark against vLLM, Flare, TensorFold, and llama.cpp.
Faster for one user, and for many. Left: SPEED-Bench with 16k prompts, one point per concurrency level; up and to the right is better. Right: AgentPerf-Local agent replay. Hover a point for its numbers.
| GPQA | LCB | IFBench | AIME25 | AIME26 | |
|---|---|---|---|---|---|
| Exact | 90.7 | 87.7 | 85.3 | 100.0 | 100.0 |
| + 3-bit weights | 90.4−0.3 | 85.4−2.3 | 83.3−2.0 | 99.2−0.8 | 99.2−0.8 |
| + Expert sharing | 90.4−0.3 | 85.4−2.3 | 83.3−2.0 | 99.2−0.8 | 99.2−0.8 |
| + Relaxed verif. (fast) | 89.1−1.6 | 84.6−3.1 | 81.7−3.6 | 99.2−0.8 | 99.2−0.8 |
Within 3.6 points of the exact model. Task accuracy (%) on Qwen3.8-Flash-Next as each lossy optimization is added; small numbers are the drop from the exact model.
Install uv, then:
# the repository opens with the paper git clone https://github.com/sharar-project/hibana && cd hibana uv sync uv run hibana serve 27b # Qwen3.8-27B on http://127.0.0.1:30001 uv run hibana serve flashnext # Qwen3.8-Flash-Next (125B MoE) uv run hibana launch claude # point a coding agent at it
OpenAI- and Anthropic-compatible APIs, 262K context, one Spark or two over ConnectX. Add --exact for the model's own outputs.
@misc{emara2026hibana,
title = {Hibana: Fast Agentic Inference on Unified-Memory Devices},
author = {Emara, Yahya and Chang, Chi-Chih and Fei, Anthony and
AbouElhamayed, Ahmed F. and Chen, Chong-Yan and Chiang, Hung-Yueh and
Wu, Kai-Chiang and Abdelfattah, Mohamed S.},
year = {2026},
note = {Preprint}
}