Sharar: fast local inference for AI agents

Sharar is an inference engine for the NVIDIA DGX Spark. It runs Qwen and Nemotron models of up to 125B parameters for many agents at once, behind OpenAI- and Anthropic-compatible APIs.

Get started Read the paper

One GB10 · 128 GB unified memory · 262K context · Apache 2.0

4.3×
higher decoding throughput
3.1×
more agentic tasks per hour
2.1×
faster for a single user

Qwen3.8-Flash-Next on one DGX Spark, against the fastest of vLLM, llama.cpp, and TensorFold.

Built for unified memory

On a DGX Spark, the model and every agent's history share one memory. Sharar is designed around that.

Fewer bytes per token

Model surgery

3-bit weights, INT8 recurrent state, and a 2-bit draft head.

How it works → No recomputation between turns

SSD session cache

Every agent's state survives across turns, on the SSD when GPU memory is full.

How it works → More tokens per weight read

Adaptive speculation

Shared experts across drafted tokens, and draft lengths that follow acceptance.

How it works →

Models

Four open-weight models, each tuned for the Spark.

Qwen3.8-27B27B dense
Qwen3.8-Flash-Next125B MoE
Nemotron 3.5 Super120B-A12B MoE
Nemotron 3.5 Bolt74B-A9B MoE

Works with your agents

Point any OpenAI or Anthropic client at the server, or let Sharar launch the agent for you.

Coding agents

sharar launch claude starts Claude Code on the served model. Also opencode and aider.

OpenAI and Anthropic APIs

Chat completions at /v1, Messages at the root, with tool calls and constrained output.

Two Sparks

Add --tp 2 to split a Qwen model across two Sparks linked by their ConnectX ports.

Get started

One DGX Spark with DGX OS, driver 580 or newer, and uv.

  1. Clone and sync. uv installs the pinned CUDA 13 PyTorch and FlashInfer, about 7 GB.
  2. Serve a model. The first run downloads it and builds the GB10 kernels.
  3. Use it. Chat in the terminal, or launch a coding agent on it.
# the repository opens with the paper
git clone https://github.com/sharar-project/hibana && cd hibana
uv sync
uv run hibana serve nemotron     # Nemotron 3.5 Super on :30005

# in another terminal
uv run hibana chat
uv run hibana launch claude

From the blog

Paper

Hibana: Fast Agentic Inference on Unified-Memory Devices

November 4, 2026 · Emara, Chang, et al.

Model surgery, an SSD session cache, and adaptive speculation, co-designed for one DGX Spark. Interactive illustrations of each idea, and results on four models.

Read the post →