The DGX Spark (GB10) Playbook for Code Agents

What this is. Field notes from months of production AI serving, training, and tooling on a DGX Spark (GB10 Grace-Blackwell, ~128 GB unified memory, aarch64), written for the code agents working on one. Every claim below was learned the hard way on real hardware — the measured numbers are from actual incidents and bring-ups, not spec sheets. Drop this file into your project’s agent instructions (CLAUDE.md, AGENTS.md, or equivalent) or reference it from there.

How to read it. Sections 1–2 are the mental model — read them first; almost every trap below is a consequence. Sections 3–10 are the potholes and the workarounds. Section 11 is the symptom→cause→fix card for when something is already on fire.

---

1. The one fact that changes everything: unified memory

VRAM and system RAM are one LPDDR5x pool (~120 GB usable). There is no “GPU memory” distinct from “host memory” — a model’s weights, your page cache, the kernel, and every process share the same physical pool. Two consequences frame everything else:

• Capacity is generous, bandwidth is the constraint. ~273 GB/s shared. Token decode is memory-bandwidth-bound, so big-model generation is slow (§7) while capacity-hungry things x86 boxes can’t do at all (70B+ dense, 100 GB+ MoE, 128k contexts) fit fine. The box’s natural role is batch / async / long-context / training, not low-latency interactive serving.
• Every memory consumer competes with the GPU. A large file download filling page cache, a memory-hungry build, a leaky sidecar — all of it comes out of the same pool your model needs. x86 intuition (“host RAM pressure doesn’t affect VRAM”) is wrong here, and it fails in surprising, asymmetric ways (§3).

That 273 GB/s is a sequential ceiling — random access is much worse. Measured on this box: sequential streaming ~231 GB/s (85% of spec), but random/gather bandwidth depends steeply on how many contiguous bytes each access grabs (coalesced warp gather, 16 GiB working set):

contiguous B / random access	64 B	1 KB	4 KB	16 KB	64 KB	≥256 KB
% of sequential	0.4%	7%	27%	74%	95%	~100%

You need ~64 KB contiguous per random location to saturate; a page-sized (4 KB) gather gets ~27%, a cache-line object ~0.4%, and fully-divergent pointer-chasing is worse still. MoE expert routing, embedding lookups, paged-KV reads and sparse attention therefore run well below the headline — budget them by access granularity, not 273. (Added 2026-07 from Kobold benchmarking; membw_sweep.cu.)

2. Measurement: per-process memory is TWO slices — sum them (temperature stays honest)

On GB10 no single per-process memory column tells the truth; the truth is the sum of two disjoint slices. Because the pool is unified, a process’s draw splits into a device-classified slice — its cudaMalloc’d share, which is exactly what nvidia-smi --query-compute-apps reports — and a host-mapped slice — exactly what RSS (/proc/<pid>/status VmRSS) reports, and blind to cudaMalloc. Each column is deterministic but partial, and which column dominates depends entirely on the workload’s allocator:

• A torch image sidecar at idle: 353 MiB device + 2417 MiB RSS ≈ 2.7 GiB actual — reading the device column alone was ~87% under.
• A llama.cpp process serving a 13 GB GGUF (weights cudaMalloc’d): 13,428 MiB device + 2.4 GiB RSS — reading RSS alone was ~5.5× under, the same mistake mirrored.
• In the llama.cpp case the two slices’ deltas across the load (+12.5 GiB device, +0.7 GiB host) summed to the pool’s MemFree delta (13.2 GiB) exactly.

An earlier revision of this section said the device column was a “semi-random fraction” and recommended RSS — half right, and dangerously wrong for GGUF serving, the workload this class of box does most. The columns aren’t noisy; they’re complementary. --query-gpu=memory.total,used,free still returns [N/A] outright, and the 580-series open kernel modules expose no DRM-fdinfo per-process memory (verified: the driver fds are /dev/nvidiactl/nvidia-uvm, no drm-memory-* keys), so the two-column sum is the best per-process number this platform has.

What to use:

Question	Tool
How big is this process really?	Sum both slices: nvidia-smi --query-compute-apps=pid,used_memory (device slice) + VmRSS (host slice)
What did a load actually take?	MemFree delta across the load — the ground truth the slices must sum to
What’s the pool state?	free -m — but see §3 for available vs MemFree
Combined live view	nvtop — its per-process columns are exactly the two slices side by side (sum them yourself); note its device-level “free” is MemAvailable-based, which over-promises allocatability (§3)
GPU temperature / throttling / utilization / power	nvidia-smi --query-gpu=temperature.gpu,clocks_throttle_reasons.active,utilization.gpu,power.draw — these ARE reliable on GB10; poll via subprocess for thermal + starvation supervision (§7)

3. The page-cache trap (the big one)

Symptom: a large model load dies within seconds; kernel log (journalctl -k / dmesg) shows NVRM: Out of memory [NV_ERR_NO_MEMORY] from _memdescAllocInternal — while free claims 100+ GB “available.”

Cause: the kernel counts reclaimable page cache as available, but the NVIDIA driver’s allocation path does not trigger that reclaim — it just fails. After hours of downloads or file churn, tens of GB of cache sit in the pool, and a big allocation that “should fit” dies. Measured: an 88 GB model load with 63 GB of download page cache resident failed in 3 seconds; after a cache drop, the identical load succeeded in 85 s. Smaller models squeak through on low-cache days, which masks the pattern — you’ll see it intermittently and blame the framework.

The discipline:

• Judge headroom by MemFree, not available — and note MemFree responds promptly: freed pages come back within ~1 s of an unload/free (measured +6.5 GB in 1 s), so a reclaim-based harvest is verifiable rather than hoped-for. If you once watched “free memory” sit flat right after a free and concluded the pool was slow to return it, check which counter you read: MemAvailable is a lagging estimate that folds in reclaimable cache; MemFree is the true immediate free.
• Drop caches before any large allocation (model loads especially):

  sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches; echo 1 > /proc/sys/vm/compact_memory'
  

  Better: automate it as a pre-load hook in your serving path — check MemFree against the model’s true size (sum all shards), drop caches via non-interactive sudo when MemAvailable says reclaim covers the gap, and refuse loudly when it doesn’t. This pattern is field-proven (incident-replayed: a load off a 5 GB-MemFree pool self-healed and completed). Until you build it, make the manual drop a documented runbook step.

• Downloads and loads interact. A background 50 GB download regrows the cache continuously. Sequence big loads right after a cache drop, and expect a load that worked yesterday to fail today if the cache profile changed.
• Hugepages are off by default, and the TLB penalty is amplified here. The box ships THP=madvise with nr_hugepages=0 (AnonHugePages: 0) — nothing gets hugepage backing unless it explicitly madvises. Large random-access working sets then pay a heavy TLB-miss tax, and each page-walk is itself a random memory read on the slow path (§1) — so small pages hurt more here than on discrete-memory x86. Measured +16% on a ~2 GB random-access workload purely from 4 KB → 2 MB pages. Check whether big weight/KV buffers are THP-backed (grep AnonHugePages /proc/<pid>/smaps; use_mmap=false reads into a malloc’d buffer that may land on 4 KB pages); consider THP=always / MADV_HUGEPAGE / an explicit pool for large random-access structures. (Added 2026-07 from Kobold benchmarking.)

The load doesn’t have to crash to take the box down. §3’s headline is a load that dies; the subtler failure is a load that succeeds while dragging every other process down with it. A large read — mmap or a use_mmap=false buffered read, which still fills page cache as it streams the file in — balloons the cache toward the pool ceiling; MemFree floors; the kernel enters a reclaim storm and begins evicting the resident working sets of unrelated processes. Their next memory touch is a major fault against a now-saturated disk, so health probes, monitors, and small services across the box stall or time out while the load itself is fine. Measured: an ~85 GiB mmap’d load pinned cache 1→87 GB, floored MemFree, and drove a monitored process into ~12.8k major faults — box-wide probe failures with nothing actually broken. Two things follow:

• Bound the cache continuously during the load, not just per-shard. A background sweeper issuing posix_fadvise(POSIX_FADV_DONTNEED) over the file region on a short cadence (~1 s) holds a buffered read’s cache footprint flat — measured: Cached stayed 1–2 GB and MemFree steady ~28 GB through a load that otherwise ballooned to 87 GB. This is the buffered-read complement to §4’s per-shard fadvise on the HF path.
• Enforce the no-mmap decision in code, not in per-model config. “Flag the big rows use_mmap=false” is one forgotten row away from a box-wide incident — that is exactly how the 85 GiB load above slipped through. Make the loader force no-mmap whenever a model’s weights exceed a fraction of the pool (e.g. > pool/3), loudly, with an explicit per-model override — the safe path should be the default, not an opt-in.

4. mmap is not your friend for large models

On x86 + discrete GPU, memory-mapping model files is a reasonable default. On GB10 it’s a trap, twice over:

• Large GGUF (llama.cpp): load with use_mmap=false. An mmap’d load fills the unified pool with page cache approximately equal to the file size, and the driver can fail to IOVA-map the file region — a hard crash. Read-into-buffer costs a slower load (a few GB/s off NVMe) and is the reliable path. With use_mmap=false the §3 cache-drop discipline — and its continuous fadvise sweeper, since the buffered read still fills cache as it streams — applies to the load; better yet, force the no-mmap choice by weight-size in the loader rather than a per-model flag (§3).
• Large HF/transformers loads can hard-deadlock the entire box. The loader’s mmap page cache (≈ on-disk size) fills the pool during the load — even when the final quantized footprint would fit comfortably (observed with a 4-bit load of a ~200 GB checkpoint). Mitigations: bound the cache with posix_fadvise(DONTNEED) as shards are consumed, and always validate the load path on a medium model before attempting the big one — never jump straight to the largest checkpoint. A deadlocked GB10 can require a power-supply-level reboot.

5. Capacity planning is on you: admission is blind

Frameworks’ “check free VRAM before loading” logic breaks on GB10 — device-free queries misreport or return nothing, so VRAM planners either pass trivially or lie. Practical consequences:

• Model swaps are a hazard. Loading model B while model A is resident can exceed the pool and kill the serving process natively — no Python traceback, the process just dies (and takes every queued request with it). Unload the resident model first, or serve on what’s loaded and plan swaps deliberately. Treat “same-backend model swap” as a dangerous operation, not a free one.
• Don’t trust checkpoint metadata for sizing. HF safetensors index.json total_size is sometimes the parameter count, not bytes (observed at half the real footprint on a large MoE). Sum the actual shard file sizes.
• Budget the whole pool: model weights + KV cache + compute buffers + every sidecar
  • page cache you can’t fully control + a real margin. A measured example that works: 88 GB weights + q8 KV at 128k ctx + buffers ≈ 97 GB resident with ~18 GB headroom on a 120 GB pool — viable, but only with the §3/§4 discipline.
• Two allocator pools that can’t see each other sum to the cliff. Each framework keeps its own caching pool (PyTorch’s caching allocator, CuPy’s memory pool, …) and none accounts for the others, so two individually-reasonable residents can overcommit the shared pool together when neither would alone. Measured: a ~14 GB fp64 array pool beside a framework’s eval-transient pool reached ~94 GB claimed on a 121 GB box (~5 GB free) and ground to a crawl at 96% reported GPU utilization and full SM clocks (2476 MHz) — busy-looking, doing almost nothing. That util/clock reading is the tell of unified-memory page-migration thrash; utilization lies here the way nvidia-smi’s memory numbers do (§2), so judge headroom by MemFree, not util. Keep the sum of pools well clear of the ceiling — hold large residents at one precision and do precision-critical math in small, transiently-cast blocks. Recovery is clean: freeing one pool (stop the process holding it) returned >110 GB within seconds.

6. The sm_121 software stack (status as of mid-2026 — verify, don’t assume)

The GPU is compute capability sm_121. Blackwell + aarch64 puts you ahead of most projects’ CI matrices. The standing rule: verify every stack component on the box before betting on it — x86/discrete-GPU experience does not transfer.

Verified working on-box:

• llama.cpp / GGUF (and llama-cpp-python) — the workhorse serving path.
• PyTorch (cu13x) + transformers — including bf16 inference of exotic architectures.
• bitsandbytes 4-bit QLoRA training — proven on a ~100 GB-class MoE (loss curve healthy, box stable) despite “unverified” reputation. Validate your own versions.
• ComfyUI (SDXL and FLUX-class image gen; fp8 checkpoints fine).

Known broken / not worth fighting:

• FlashAttention-3: unsupported on sm_121 — and PyTorch SDPA is faster anyway. Don’t burn a day building FA from source.
• Native FP4 acceleration is broken/fallback (no tcgen05 path as of 2026-06). Run bf16 or fp8; don’t bet on NVFP4 speedups. modelopt NVFP4 checkpoints won’t load via plain transformers at all (they need vLLM/TRT-LLM).
• CUDA 13+ is mandatory; older wheels don’t exist for this target.

Bleeding edge (works with caveats): vLLM needs cu130 nightlies and --enforce-eager for MoE. Assume AWQ/GPTQ/ExLlama-class quant stacks are unverified until you prove them.

7. Throughput: what to expect and which levers actually work

Calibrated expectations prevent wasted debugging (“why is this so slow” is usually “because bandwidth”):

• Dense decode is bandwidth-bound: a 70B dense at Q5 runs ~3–5 tok/s single-stream. That is the hardware, not a bug.
• MoE throughput tracks ACTIVE parameters, not total. Measured: a 235B-A22B MoE at ~2.7-bit imatrix quant sustains ~16 tok/s — total size 88 GB, but only 22B active per token. A 106B-A12B runs faster still. On GB10, prefer the largest MoE that fits at a sane quant over any dense model of similar quality.
• Prompt processing is compute-side and healthy: measured ~390 tok/s at 7k depth, ~260 tok/s at 60k depth on the 235B. Long-context reading is cheap relative to generation; a 60k-token ingest is minutes, not hours.
• Quantize the KV cache to q8: measured strictly better on GB10 (16.8 vs 16.2 tok/s — less bandwidth per token) and it halves KV memory, which is what makes 128k contexts fit under big models. Keep model weights at ≥3-bit with imatrix if you care about reasoning quality (reasoning degrades below ~3-bit long before fluency does).
• Quality lever that sidesteps bandwidth: on a box where nobody waits, test-time compute (best-of-N sampling with selection/verification) buys correctness without needing a bigger model. Note that consecutive calls sharing a prompt prefix reuse the KV cache in llama.cpp — N samples of one prompt cost much less than N distinct calls.
• LM training is bandwidth-bound too — and the tell is high-util-LOW-POWER. Measured (0.6B-param LoRA SFT, batch 32, seq ≤640, grad-checkpointed): 2.93 s/step packed vs 2.88 unpacked — a tie, i.e. sequence packing’s ~2.7× token economy bought nothing; the same packed workload runs 0.72 s/step on a discrete 5070 Ti. The ~4× box ratio tracks the LPDDR5x-vs-GDDR7 bandwidth ratio (~3.3×), not the compute ratio (~2.5×). Diagnostic signature during training: ~96% GPU util at only ~50 W with sustained high clocks — the util counter says busy, the power draw says starved. Judge by watts-at-util (cousin of judge-by-MemFree and the 96%-util thrash tell, which shows the same util lie with a different cause). Two corollaries: (a) batch/token-economy optimizations that pay on discrete cards can be no-ops here — measure before porting the tuning folklore; (b) short-probe throughput numbers do not survive sustained campaign conditions (a ~1.9 s/step probe baseline failed to reproduce at ~2.9 sustained) — schedule from sustained measurements on the warm box, never from probes.

8. Thermal: the operating point is a choice — measure it, then pin it

Sustained GPU load heats the box; heavy downloads plus inference compound it. The temperature side of nvidia-smi is trustworthy (§2): poll temperature.gpu and clocks_throttle_reasons.active. Everything below is measured on real hardware (driver 580-series open modules).

The actuator matrix — one lever, several gauges. Know what you can actually set before designing thermal management:

Control	Support	Notes
SM clock lock (nvidia-smi -lgc min,max)	works	Needs elevation — plain users get a permission refusal, so a service asserting caps wants a scoped sudo -n rule. -rgc resets. The lock silently drops on reboot/driver reset and no query reliably exposes lock state — periodic re-assert IS the detection.
Power limit (-pl)	absent	Every limit field reads [N/A] — min, max, default, current. There is no power actuator on this silicon.
Power readings	GPU draw works	Module/memory power readings are [N/A]; GPU average/instantaneous draw is real and is your watts-at-util signal (§7).
Memory clock lock (-lmc)	absent	Not queryable, not lockable — and you wouldn’t want it: bandwidth is the ceiling (§1, §7).

The measured clock/temperature/throughput curve. Sustained single-stream decode (MoE ~3B-active GGUF, back-to-back 4k-token generations), five 30-minute windows, equilibrium stats from each window’s second half, throughput from 21–24 completed generations per window:

SM cap (MHz)	clocks (eq)	temp (eq)	power	throughput
uncapped	2477	69.0 °C	42.6 W	100 %
2100	2093	57.1 °C	23.9 W	99.0 %
1800	1768	53.0 °C	18.1 W	97.0 %
1500	1482	50.4 °C	15.5 W	92.8 %
1200	1037	49.0 °C	13.7 W	86.1 %

The reading: decode is bandwidth-bound (§7), so the SMs are mostly burning watts to wait faster — a 15 % clock cap costs 1 % throughput and sheds 12 °C and 44 % power. The compute-bound knee sits near ~1700 MHz; below it throughput follows clocks down. For a bandwidth-bound steady-state box, running uncapped is paying double-digit watts and degrees for a rounding-error of speed.

Don’t let threshold gates be your pacer. The tempting design — pause new GPU work above a hot threshold, resume below a cool one — becomes bang-bang control under a sustained workload: measured on this box, a pause-75/resume-65 gate under continuous load flaps every ~60 s, sawtoothing the die 65↔82 °C for hours (thousands of thermal cycles per campaign) while delivering roughly half-duty violently. If your emergency path suspends the worker (SIGSTOP), it also stretches every work window and the suspended process answers no health probes — teach your monitoring that suspended ≠ crashed. Gates are correct as rarely-hit backstops. The primary control should be a pinned clock cap at a sustainable equilibrium — pacing by physics, continuously, instead of by brakes, once a minute.

Reproducing the curve on your workload (an evening, mostly unattended): step caps through 30-min windows under sustained load and take equilibrium stats from each window’s second half. Traps that ate our first attempt: arm on a rolling-mean utilization over minutes (any job-per-item workload dips util at boundaries — an every-sample streak gate never arms); make sure your heat source is actually sustained (a generate-then-score harness measured 30 % duty — a monster on paper, a flicker on the die); vary prompts if anything caches results; take throughput from completed real work items, never from util; and have every exit path clear the cap, so the experiment can’t leave its conclusion configured. Caveats that transfer with the numbers: this is a decode-shaped point — prefill is the most compute-bound inference phase and backward-heavy training is more different still, so validate per workload shape (the temperature delta transfers better than the throughput figure); equilibria float on ambient; and pause-don’t-fail remains the right posture for the backstop gates.

9. Operational hygiene for agents on this box

Habits that repeatedly saved the day (none GB10-exclusive, all GB10-amplified because loads are big and long):

• Launch long jobs detached. Session-spawned background processes get reaped when the agent session ends. Use setsid nohup <cmd> > log 2>&1 < /dev/null & disown, write state to a file, and design for resume.
• One big download at a time. Concurrent HuggingFace pulls on one identity throttle each other into deadlock; two writers on one partial file corrupt it. Serialize, make downloads resumable, and remember every GB downloaded is page cache (§3).
• Never restart a GPU-serving process casually. In-flight generations die invisibly to their callers. Prefer in-place config/catalog reloads over restarts; if you must restart, drain first. If a supervisor auto-heals services, make sure one service’s heal can’t cascade-restart an unrelated healthy one — and require ≥2 consecutive failed health probes before surgery (a busy process blows single probes routinely; measured incident: one flapped probe cascaded into killing a healthy serving process mid-generation).
• A monitor that fires during a big load is often telling the truth. Before you damp or silence it, rule out §3’s reclaim storm — the load may be genuinely starving the box and the probe is just the messenger. Cheap sampling discriminates a miscalibrated probe from real working-set eviction in minutes: watch major-fault deltas (ps -o maj_flt= -p <pid>) and MemFree (/proc/meminfo) while the load runs. Suppressing a correctly-firing monitor hides the incident; bounding the load’s cache footprint (§3) removes it.
• When a process dies “for no reason,” check the kernel log first. journalctl -k around the timestamp. Native deaths (NVRM OOM, driver faults) never produce Python tracebacks and masquerade as framework bugs. Second question: did something restart it? — check your supervisor’s log before blaming the hardware.
• Align your clocks when correlating logs. Application logs in local time vs API/kernel timestamps in UTC has burned real debugging hours. Pick one, or annotate.

10. The CPU: heterogeneous cores, interleaved numbering

The sections above are GPU/serving-focused; the 20-core Grace CPU has its own traps.

• It’s heterogeneous: 10× Cortex-X925 (perf, ≤3.9 GHz) + 10× Cortex-A725 (efficiency, ≤2.8 GHz), ~30% per-core throughput gap (measured 479 vs 341 H/s on a CPU-bound integer load — an efficiency core ≈ 71% of a perf core).
• The fast cores are interleaved, not a contiguous block: CPUs 5–9 and 15–19 are X925; 0–4 and 10–14 are A725. So taskset -c 0-9 silently grabs a mixed fast+slow set. Detect the fast cores by max clock (/sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_max_freq), never by assuming 0..N.
• Memory-bound CPU work scales sublinearly past ~10 threads: the two clusters measured in isolation summed ~8% above all 20 threads together — the shared unified bus (§1) saturating under full load.

Practical: pin throughput-critical CPU threads (dataloaders, tokenizers, preprocessing, CPU offload) to the detected fast cores; size memory-heavy pools to the ~10 perf cores. Detect, don’t hardcode — the core→cluster map may shift across firmware. (Added 2026-07 from Kobold benchmarking.)

11. Symptom → cause → fix

Symptom	Likely cause	Fix
Large load dies in seconds; kernel log: NVRM: Out of memory; free says plenty “available”	Page cache counted as available; driver won’t reclaim (§3)	sync; drop_caches; compact_memory, judge by MemFree, retry
Box hard-deadlocks during a big HF/transformers load	mmap page cache ≈ file size floods the pool (§4)	fadvise DONTNEED bounding; medium-model validation first
Box-wide health-probe / monitor timeouts during a big load, but nothing crashed	Load ballooned page cache → reclaim storm evicted other processes’ working sets (§3)	Continuous fadvise-DONTNEED sweeper to bound the load’s cache; force use_mmap=false by weight-size; judge by MemFree
llama.cpp crash loading a big GGUF	mmap/IOVA failure (§4)	use_mmap=false (+ §3 cache drop)
Serving process dies mid-swap, no traceback	Pool exhaustion: new model loaded beside resident (§5)	Unload first; treat swaps as planned operations
nvidia-smi shows a big model using a few hundred MiB	You read one slice of two — that process is host-mapped-heavy (§2)	Sum device slice + RSS; judge loads by MemFree delta
A serving process shows tiny RSS while MemFree dropped a model’s worth	Mirrored mistake: cudaMalloc’d weights are invisible to RSS (§2)	Sum device slice + RSS; judge loads by MemFree delta
--query-gpu=memory.* returns [N/A]	Expected on GB10 (§2)	Use free -m for the pool
FlashAttention build fails / underperforms	FA3 unsupported on sm_121 (§6)	Use PyTorch SDPA
NVFP4/FP4 checkpoint won’t load in transformers	No plain-transformers path (§6)	Use bf16/fp8, or serve via vLLM/TRT-LLM
70B dense feels impossibly slow	Bandwidth-bound decode — expected (§7)	Use a MoE; batch/async posture
Checkpoint “fits” per index.json but won’t load	total_size may be param count, not bytes (§5)	Sum actual shard file sizes
Two HF downloads both crawl, then stall	Same-identity concurrency throttling (§9)	Serialize; single-flight lock
Background job vanished after the agent session	Session-tied process reaped (§9)	setsid nohup … & disown, resumable state
Healthy process restarted “by itself” mid-job	Supervisor health-flap or cascade (§9)	Probe-streak damping; audit dependency edges; check supervisor log
Gather / MoE-routing / embedding / paged-KV kernel far below 273 GB/s	Random-access bandwidth is granularity-dependent; small chunks get a fraction of peak (§1)	Enlarge contiguous bytes per random access (~64 KB saturates); coalesce; or go sequential
Large random-working-set job slow; AnonHugePages: 0 for the process	Default THP=madvise + no hugepage pool → 4 KB pages → TLB thrash on the slow random path (§3)	THP=always / MADV_HUGEPAGE / explicit hugepage pool for the big buffers
taskset -c 0-9 gives inconsistent / low CPU throughput	Fast cores are interleaved (5–9,15–19); 0–9 is a mixed set (§10)	Detect fast cores by max clock; pin to those
CPU throughput plateaus well before 20 threads	Memory-bound work saturates the shared unified bus; efficiency cores add ~30% less (§10)	Expect sublinear scaling; size memory-heavy pools to the ~10 perf cores
GPU at ~96% util + full SM clocks but throughput ≈ 0	Unified-memory page-migration thrash — often two allocator pools (torch + cupy, …) summing past the cliff (§5)	Judge by MemFree, not util/clocks; free a pool (stop its holder); size pools so their sum clears the ceiling
GPU at ~96% util + sustained clocks but power draw far below TDP (e.g. ~50 W), throughput steady-but-slow	Memory-bandwidth starvation — the workload (LM training, dense decode) is bandwidth-bound, not compute-bound (§7)	Expected, not a bug: judge by watts-at-util; don’t expect discrete-card token-economy tricks (packing, batching) to pay; schedule from sustained warm-box measurements, not short probes
Die sawtooths hot↔cool every ~minute under sustained load; work windows stretch; suspended workers read as crashed	Threshold pause/resume gates acting as the primary thermal control — bang-bang under continuous load (§8)	Pin a sustainable SM clock cap (measure your curve — on bandwidth-bound work −15% clocks ≈ −1% throughput); demote gates to backstops
nvidia-smi -lgc refused / -pl does nothing	Clock verbs need elevation; power limiting doesn’t exist on GB10 (§8)	Scoped sudo -n rule for -lgc/-rgc; re-assert on a cadence (locks drop silently on reset, no query shows lock state); there is no -pl to fix

---

Compiled 2026-07-03 from production incident logs and bring-up notes on a DGX Spark running large-model serving (up to 235B MoE @ 128k ctx), LoRA/QLoRA training, and diffusion image generation. Numbers are point-in-time measurements on that box — re-verify on yours; the stack moves fast.

Additions 2026-07-11 — §1 random-access bandwidth, §3 hugepages, §10 CPU cores, and the last four fix-table rows — contributed from Kobold’s GB10 benchmark suite. These are microbenchmarks (not production serving), so validate the AI-workload implications on your own loads. Reproducer: kobold/src/kobold/bench/membw_sweep.cu.

Additions 2026-07-13 — §3 reclaim-storm / working-set-eviction failure mode + continuous fadvise-DONTNEED sweeper + MemFree prompt-credit timing, §4 code-enforced no-mmap-by-weight-size, §9 monitor-truth-during-loads, and the matching fix-table row — from further production incident work on the same class of box.

Additions 2026-07-21 — §5 dual-allocator-pool overcommit (two pools summing to the unified cliff; the “96% util / full clocks / no throughput” thrash signature) + its fix-table row — contributed from Xaos’s LoRA-rig bring-up on the same class of box.

Additions 2026-07-27 — §7 LM-training-is-bandwidth-bound (packed/unpacked A/B tie, ~4× vs-discrete ratio tracking bandwidth not compute, the “high-util-low-power” starvation tell, probe-vs-sustained measurement caveat) + its fix-table row — contributed from Xaos’s training-host A/B on the same class of box.

Revision 2026-07-27 (second edition) — §2 rewritten: the two-slice per-process accounting model (device-classified slice + host-mapped slice; sum = truth; MemFree delta = ground truth), replacing the earlier “semi-random fraction / use RSS” guidance that a live GGUF-serving A/B showed to be 5.5× wrong in the mirrored direction. Also §2: no DRM-fdinfo on the 580-series open modules; utilization/power join temperature on the trustworthy side of nvidia-smi. Sourced from a monitoring-tool source review (nvtop) + live measurements on the same box.

Additions 2026-08-01 — §8 rewritten as the thermodynamic profile: the actuator matrix (SM clock lock works but needs elevation and cadence re-assertion; power limiting is entirely absent on this silicon; memory clocks untouchable), the measured five-point clock/temperature/throughput curve under sustained decode (−15% clocks → −1% throughput, −12 °C, −44 % power; compute-bound knee ≈ 1700 MHz), the bang-bang warning (threshold gates under sustained load flap ~60 s and sawtooth the die; pin clocks, demote gates to backstops), a reproduction recipe with its measured traps (rolling-mean arming, 30 %-duty false monsters, fail-toward-uncapped), and two fix-table rows. Sourced from a chartered operating-point experiment on the same box.