Blogs

I wanted a physics lab that runs anywhere — no backend, no video uploads, no app install. The result is this site’s /physics page: an AR free-fall lab where a virtual ball falls over your live came...
I wanted a physics lab that runs anywhere — no backend, no video uploads, no app install. The result is this site’s /physics page: an AR free-fall lab where a virtual ball falls over your live camera feed, and real drops can be timed or auto-tracked to measure g. All vision runs on-device in vanilla JS.

The page has one live experiment (Exp 01 — Dropping a Ball) with four modes. Three more cards (projectile, pendulum, air friction) are placeholders.

1. Mode 1: AR simulate — the hook

A virtual ball falls over <video> from your camera, or a dark stage if the camera stays off. Controls: planet preset (Mercury through Neptune + custom), drop height, slow motion (1×/0.5×/0.25×), ball skin, trail toggle.

The physics is semi-implicit Euler at requestAnimationFrame time steps, clamped to 50 ms so backgrounded tabs don’t teleport the ball:
- v += g·dt, y += v·dt
- theory readout t = √(2h/g) alongside live t / y / v
The ball itself is pure canvas: radial-gradient sphere shading, rotating seams (tennis felt speckle is deterministic so it doesn’t flicker), a fixed specular highlight, and a ground shadow that tightens as y → h. A ruler overlay maps frame height to meters, so the fall reads correctly against your room.

One camera lab-wide: opening any stage’s camera releases the others (stopOtherCameras). Phones can’t hold two getUserMedia streams, and this also saves battery. A denied request no longer kills already-working stages — streams are pruned only after the new one succeeds.

2. Mode 2: measure g with a stopwatch — the real lab

Drop a real ball from measured height h, press Start on release, Stop on impact. Each trial computes g = 2h/t², persisted to localStorage with CSV export and per-trial delete.

The honest part is the error discussion, stated on the page: human reaction time (~0.15 s) dominates, so taller drops are better. Stats show mean ± sample-std, median, % error vs 9.81, and n. Corrupt stored entries are filtered on load so one bad write can’t break rendering.

3. Mode 3: auto-track — color centroid + least-squares fit

Point the camera at a plain background, pick the ball color (or click the ball to sample it), press Record just before dropping. Each frame:
1. Downsample to 160×120 into a work canvas.
2. Build a match mask by RGB distance² ≤ tolerance².
3. Flood-fill 4-connected components, keep the largest blob’s centroid — a global average would let scattered background matches drag the point away.
4. Convert px → meters (explicit scale, or full-frame-height = h fallback), keep the falling segment, trim pre-drop loitering, and fit y = y₀ + v₀·t + ½·g·t² by closed-form least squares (Cramer’s rule on the 3×3 normal equations).
Needs 8+ points, >0.1 s span, and >5 cm of motion, or it refuses with a specific message (too few points → brighter ball / closer camera / higher tolerance). The fit result reports g, v₀, RMSE in cm, and % vs 9.81. A red ball on a white wall works best — unsurprising, but worth stating.

4. Mode 4: hand-hold — the hard vision problem

The virtual ball sticks to your hand; lift it above a dashed drop line to release. This mode earned most of the debugging time:
- Skin default via YCrCb rule (133 ≤ Cr ≤ 173, 77 ≤ Cb ≤ 127), no calibration click needed. More lighting-robust than one sampled RGB point.
- Moving-only gate: matched pixels must also change luminance frame-to-frame, rejecting static skin-colored clutter (wood, faces sitting still).
- Sticky lock: once held, blobs near the last position win over bigger blobs elsewhere — otherwise your face, usually the largest skin blob in frame, steals the ball.
- Blue-backdrop chroma key: optionally excludes strongly blue-dominant pixels for the blue-bedsheet-behind-you setup.
- Tracker view: overlays match dots so you can see what the tracker sees when the ball sticks wrong.
Failure guidance is on the page, not in a tooltip: keep faces and bare arms out of frame, the lock prefers your hand but a face is a bigger target.

5. Lab assistant: local model, scoped context

A “Lab assistant” section talks to Ollama on localhost:11434 (same browser-direct pattern as the /chat page). It never measures anything — it reads a small on-device event log (trial table, last auto-track fit, last 5 failures) and answers setup/diagnosis/results questions. Laptop only; the page degrades to “Ollama not reachable” elsewhere.

What I’d do differently

The px-to-meters scale is the weakest link in auto-track: full-frame = h is a rough fallback, and a taped ruler + explicit scale entry is still the honest path. Next up is Exp 02 (projectile) reusing the same tracker, and a pendulum mode where period measurement forgives scale errors entirely.

Cloud leaderboards tell you what a model can do on someone else’s hardware with someone else’s budget. I wanted to know what an open-weights model can do on my machine, inside my agent loop — so I benchmarked devstral:24b (local, via Ollama) on HumanEval and MBPP+, then measured how much of the gap a verify-gated repair loop closes. All runs are reproducible from Zenith’s scripts/ directory.

The headline numbers:

Benchmark	Single-turn base	Single-turn Plus	After gated repair base	After gated repair Plus
HumanEval (164)	86.0%	79.9%	91.5%	84.8%
MBPP+ (378)	75.1%	65.6%	85.2%	71.4%

Setup: one completion per task at temperature 0.2, scored with EvalPlus (base + hidden “plus” tests). The Plus column is the one I quote — more on why below.

1. The harness bug that scored everything 0.000

The first full HumanEval run generated all 164 completions, ran evalplus.evaluate, and reported pass@1: 0.000. Every single task “timed out” — including trivial ones the model obviously solved.

Root cause: EvalPlus workers call setrlimit(RLIMIT_AS, 4GB) to cap memory, and this machine rejects raising the address-space limit (ValueError: current limit exceeds maximum limit). Every worker crashed before running a single test; the harness recorded each crash as a timeout.

The fix uses EvalPlus’s own escape hatch:

EVALPLUS_MAX_MEMORY_BYTES=-1 uv run python scripts/evalplus_devstral.py --limit 164

-1 skips only the memory cap; the other sandbox guards stay on. Real scores: 86.0% base, 79.9% Plus. Lesson re-learned: a benchmark score of exactly 0.000 (or 1.000) is a harness smell, not a model result. I baked the env default into the script so future runs can’t silently score zero.

2. Naive self-debug doesn’t work (I checked)

The obvious next step — feed the failing test output back and ask for a fix — went 0 for 2 on a probe of two failed tasks. Worse, one “repair” turned a clean NameError into an infinite loop (timeout). The model repeated the same wrong logic with different formatting.

That failure motivated the protocol that actually worked: verify-gated repair. Three rules:

Gate on visible tests only. A repair replaces the original only if it passes the base test suite. Otherwise round-0 stands — repairs that regress are discarded, not averaged in.
Sample, then select. Up to 3 candidates per round at temperature 0.6, up to 2 rounds. Diversity beats iteration depth for near-miss bugs.
Never touch hidden tests during repair. The Plus suite is the final exam; studying for it with the answer key would void the score.

3. HumanEval: 86.0% → 91.5% (and 8 of 9 fixes generalize)

Gated repair fixed 9 of 23 base failures. The part I care about: 8 of the 9 also pass the hidden Plus tests — the model actually fixed the logic rather than overfitting the visible asserts. The 9th (HumanEval/91) passes base but still fails Plus: an honest partial win, reported as-is.

Failure clusters were unsurprising: string munging and numeric edge cases. Spot checks confirmed genuine reasoning slips (e.g. a factorization using trial division stepping by 1, dying on the large-prime stress test), not harness artifacts.

4. MBPP+: lower floor, stricter ceiling

MBPP+ (378 tasks) came in lower across the board: 75.1% base, 65.6% Plus. MBPP tasks are shorter but fiddlier — exact output formats and off-by-one traps — and the Plus extra tests are brutal. The base→Plus gap is ~10 points vs ~6 on HumanEval: proportionally more misses are robustness, not core logic.

Repair fixed 39 of 94 base failures with zero regressions (all other 284 tasks byte-identical), lifting base to 85.2%. But only 22 of 39 generalize to Plus — a notably lower rate than HumanEval’s 8/9, suggesting several repairs overfit the visible asserts. That’s exactly why the Plus column is the quotable one: 65.6% → 71.4% is the real gain; the base lift (+10) is flattered.

5. Caveats, stated plainly

One sample per task at temp 0.2 — treat every number as ±3 points. No variance bars were harmed in this post.
Contamination. HumanEval is almost certainly in every model’s training data; assume absolute scores are flattered. The Plus deltas are the more honest signal, and uncontaminated benches (LiveCodeBench) are next on my list.
This measures the model, not the agent. The eval scripts call the provider single-turn; they bypass Zenith’s ReAct loop entirely. Zenith itself runs verify-after-edit, a critic gate with Reflexion retries, and TDD reproduce→fix→verify guards by default — so these numbers are the floor of what the full agent should achieve, not the ceiling.

6. Reproduce it

# single-turn eval (resumable)
/opt/homebrew/anaconda3/bin/python scripts/evalplus_devstral.py --limit 164 --model devstral:24b
/opt/homebrew/anaconda3/bin/python scripts/evalplus_devstral.py --dataset mbpp --limit 378 --model devstral:24b

# gated repair (visible-tests only) + rescore
/opt/homebrew/anaconda3/bin/python scripts/selfdebug_repair.py --dataset mbpp

Next up: BigCodeBench for tool-use headroom, and the big prize — pointing the real Zenith agent loop (not single-turn probes) at tasks, plus real SWE-bench once I have Docker hardware. Terminal-Bench is blocked on this Mac for the same reason.

A chat page on a static GitHub Pages site that talks to a local Ollama model sounds contradictory: there is no server to hold API keys, run tool loops, or keep secrets. I built exactly that for thi...
A chat page on a static GitHub Pages site that talks to a local Ollama model sounds contradictory: there is no server to hold API keys, run tool loops, or keep secrets. I built exactly that for this site’s /chat page, and the constraints shaped every decision. Here is what worked, what bit, and the architecture that fell out.

The core trick: the browser calls Ollama directly

Ollama listens on 127.0.0.1:11434, so a page can POST /api/chat straight from JavaScript — streaming NDJSON tokens included. No backend, nothing leaves the machine. The catch is CORS: Ollama’s default origin allowlist covers localhost (any port, so local Jekyll preview works with zero config) but rejects https://mohankku.github.io. The fix is one environment variable plus a restart:
```
launchctl setenv OLLAMA_ORIGINS "https://mohankku.github.io"
brew services restart ollama
```
For persistence across reboots, the variable belongs in the EnvironmentVariables dict of the Homebrew LaunchAgent plist instead of launchctl setenv. Since Ollama only binds loopback, allowing the origin is safe: only browsers on that laptop can reach it, and anyone else opening the page just gets “Unreachable” — their browser tries their own localhost.

Not all local models tool-call equally

I gave the model a web_search tool for live prices and products. Devstral 24B emits native tool_calls with IDs; Qwen2.5-coder 14B instead emitted the call as JSON inside the message text. The page handles both: native calls first, then a best-effort parse of {"name": "web_search", ...} from content. The loop is standard — decision call (stream: false), execute searches, feed back role: "tool" messages, up to 3 rounds, then a final streamed answer with tools detached. If your small model “ignores” tools, check whether it is actually calling them in prose before blaming the harness.

The key problem, and the loopback proxy

A search API key cannot live in a public static site — every visitor downloads the JS. Browser localStorage works per-browser but means re-entering keys and trusting every tab. So the key moved into a ~200-line stdlib-only Python proxy on the laptop:
- Binds 127.0.0.1 only — off the LAN and internet entirely.
- Origin allowlist (deployed site + local previews); everything else gets 403, verified with hostile-origin probes.
- 30-searches/minute rate limit so a rogue tab cannot burn the monthly quota.
- The Tavily key sits in a mode-600 file the proxy reads; clients never see it.
- Access-Control-Allow-Private-Network: true on preflights, which Chrome requires for public-pages-to-loopback calls.
The proxy also runs scheduled price watches (re-check every 6 hours, min-price extraction from snippets, drop/target-hit badges in the page). Snippet-derived prices are approximate — a stray “$18 accessory” once won “lowest price” — so the UI says so honestly.

Debugging notes
- Ollama once wedged itself: models loaded, every request returned {"done": false} empties. A service restart fixed it; treat empty done: false finals as “restart the server.”
- Devstral is text-only (400 on images), so photo questions route to a small companion vision model (gemma3:4b) for that turn only.
- Test the markdown renderer in isolation: extracting the pure functions and running assertions in JavaScriptCore caught a real bug (text after a closing code fence was silently dropped) before any browser was involved.
Total new infrastructure: one LaunchAgent, one 600-permission config file, zero cloud dependencies. The page degrades gracefully at every step — no Ollama, no proxy, no key — with errors that say exactly which piece to start.
Autonomous coding agents have rapidly evolved from simple chat completions into multi-turn loop runners that read, edit, execute, and verify software repositories. While cloud-hosted frontier model...
Autonomous coding agents have rapidly evolved from simple chat completions into multi-turn loop runners that read, edit, execute, and verify software repositories. While cloud-hosted frontier models (Claude 3.7 Sonnet, GPT-4o) excel at architectural synthesis, relying entirely on remote APIs presents distinct tradeoffs: strict telemetry/privacy concerns for proprietary codebases, cumulative API costs, network latency, and zero offline availability.

I built Zenith (formerly mohan-agent) to explore a local-first paradigm: running an autonomous agent driven by a local quantized open-weights model (such as qwen3.5:9b or qwen2.5-coder:14b via Ollama) on device hardware, equipped with persistent long-term memory, an AST-guided code intelligence layer, and a resilient 5-tier fuzzy edit engine.

Here is how Zenith is architected and the engineering lessons learned along the way.

1. Core Architecture & Execution Loop

Zenith is organized around a ReAct (Reason + Act) loop augmented by token compaction guards and safety checkpoints:
```
                  ┌──────────────────────────────┐
                  │      User Prompt / Task      │
                  └──────────────┬───────────────┘
                                 │
                   [Hydrate Long-Term Memory]
                    (.mohan-agent/memory.md)
                                 │
                                 ▼
            ┌──────────────────────────────────────────┐
            │   Model Router (Local Ollama / Cloud)    │
            └────────────────────┬─────────────────────┘
                                 │ Tool Calls
                                 ▼
            ┌──────────────────────────────────────────┐
            │           Tool Execution Core            │
            │  • File I/O & Multi-Strategy Patch       │
            │  • AST Code Intel (Def/Refs/Symbols)     │
            │  • Bash / Test-Runner (TDD Guards)       │
            │  • Memory Tools (remember_fact / recall) │
            └────────────────────┬─────────────────────┘
                                 │ Pre-mutation Snapshots (/undo)
                                 ▼
            ┌──────────────────────────────────────────┐
            │   Token Compactor & Context Guard        │
            └────────────────────┬─────────────────────┘
                                 │
                   Loop until task completed or test passes
```
Zenith exposes a unified CLI and interactive REPL:
```
# Run one-off command with local Qwen
zenith "analyze repo structure and add missing unit tests"

# Or enter the interactive REPL
zenith
> /diff
> /undo
> remember this project uses pytest and ruff
```
2. Model Routing: Local By Default, Cloud on Demand

A primary goal was ensuring Zenith works completely offline without transmitting a single byte of user source code to third-party servers. By default, Zenith connects to a local Ollama instance running qwen3.5:9b or qwen2.5-coder:
```
# config.yaml
model:
  default_provider: ollama
  default_model: qwen3.5:9b
```
However, complex multi-file refactors occasionally benefit from frontier reasoning. Zenith implements a unified ModelProvider plugin interface:
```
class ModelProvider(ABC):
    @abstractmethod
    def chat(
        self,
        messages: list[dict],
        tools: list[dict] | None = None,
        temperature: float = 0.2,
    ) -> ModelResponse:
        pass
```
With providers registered via a lightweight decorator (@register_provider("provider_name")), switching execution engines is seamless at runtime:
```
# Frontier Cloud
zenith --provider anthropic --model claude-3-7-sonnet-20250219 "architect module"
zenith --provider gemini --model gemini-2.5-flash "profile memory"

# Local / OpenAI-Compatible (vLLM, DeepSeek, Ollama)
zenith --provider openai_compatible --base-url http://localhost:8000/v1 --model deepseek-coder
```
3. Persistent Two-Tier Memory Across Sessions

Most coding agents suffer from session amnesia: as soon as the process terminates, project context, linting configurations, build flags, and architectural decisions evaporate.

Zenith solves this through a two-tier memory architecture stored locally within the repository’s .mohan-agent/ directory:
1. memory.md (Long-Term Semantic Facts): Curated architectural knowledge, directory conventions (e.g. "uses poetry not pip", "run tests with pytest -m unit"), and user preferences.
  - Automatically injected into the system prompt at the start of every session.
  - The agent can explicitly invoke remember_fact(key, value) or recall_memory(query) during execution.
2. history.jsonl (Short-Term Trajectory): Maintains the last 30 interaction turns, providing conversational continuity without overflowing context window token budgets.
```
zenith
> remember this project requires Python 3.11 and ruff format
[Memory updated in .mohan-agent/memory.md]

> /memory
• [env] Python 3.11, ruff format
• [testing] pytest tests/ --import-mode=importlib
```
4. The 5-Tier Resilient Edit & Patch Engine

One of the biggest failure modes of smaller local models (7B–14B parameters) is syntactic edit drift: when asked to replace code, they may output slightly different indentation (2 spaces instead of 4), collapse trailing whitespace, or omit unchanged comments. Standard git apply or exact string matching immediately rejects these diffs, causing agent loops to spin and fail.

Zenith implements a resilient 5-tier search-and-replace pipeline:
1. Exact Match: Instant deterministic slice replacement.
2. Whitespace & Line-Ending Normalization: Normalizes CRLF/LF line endings and trims trailing whitespace before comparison.
3. Indentation Shift Adjustment: Automatically computes the common indentation delta (e.g., model indented by 4 spaces instead of 2) and shifts the replacement block accordingly.
4. Sliding-Window Fuzzy Match: Uses difflib.SequenceMatcher across candidate line windows. If similarity exceeds threshold ($> 0.88$), it locates the intended target block despite comment drift.
5. Pure-Python Unified Diff Fallback: If structured replacement fails, Zenith parses unified diff chunks and reconciles line markers directly in Python, bypassing strict patch utility restrictions.
This multi-tier approach reduced tool execution errors from local models by over 60% in benchmarks.

5. Safety: Pre-Mutation Checkpoints, /undo, and /diff

When an agent operates autonomously with file-writing and bash capabilities, user trust requires instant rollback mechanisms. Zenith takes an in-memory snapshot of every touched file prior to mutation.

In the interactive REPL:
- /diff displays a color-coded unified diff of all edits made during the current session.
- /undo pops the most recent mutation snapshot and restores the affected files instantly.
- /checkpoints lists all rollback points with touched files and timestamps.
- /compact flushes conversation history into memory.md to reclaim context tokens when approaching context limits.
6. Takeaways

Building Zenith reinforced several core principles for edge and on-device AI engineering:
- Small local models punch above their weight with scaffolding: A 9B model equipped with AST symbols, resilient edit engines, and persistent memory frequently accomplishes the same day-to-day coding tasks as frontier cloud models, with zero latency and complete data privacy.
- Memory should be human-readable: Storing memory in plain Markdown files (.mohan-agent/memory.md) allows developers to inspect, edit, or commit agent knowledge directly alongside code.
- Fail gracefully on tool drift: Agent robustness depends less on prompt phrasing and far more on lenient, multi-tier tool execution fallbacks.
The project is open source and actively maintained at github.com/mohankku/zenith.
Getting a neural network off a workstation GPU and onto a phone or headset NPU is not one step — it is a pipeline of graph transformations, each with its own optimizations. This post walks the full...
Getting a neural network off a workstation GPU and onto a phone or headset NPU is not one step — it is a pipeline of graph transformations, each with its own optimizations. This post walks the full path: an FP32 PyTorch model, through quantization and operator fusion, down through lowering to a delegated NPU binary with ExecuTorch.

0. Why this pipeline exists

On an edge SoC, inference cost is dominated by memory traffic, not FLOPs. An FP32 ResNet-50 is ~100 MB of weights; every inference drags all of it through a narrow bus while a small NPU’s MAC array sits idle waiting. Quantization shrinks the traffic 2–4x and matches the INT8 datapaths NPUs are built around. Lowering then reshapes the program itself — fusing ops, pre-laying-out memory, and carving out subgraphs the NPU can execute natively. Each stage exists to remove a different bottleneck.

1. The starting point: an FP32 model and a captured graph

Everything begins with ordinary eager PyTorch, but optimizers can’t work on Python bytecode — they need a graph. The modern capture path is torch.export:
```
example = (torch.randn(1, 3, 224, 224),)
exported: torch.export.ExportedProgram = torch.export.export(model.eval(), example)
```
Under the hood, TorchDynamo traces bytecode into an FX graph of ATen operators with shapes and dtypes specialized. This ExportedProgram (the “ATen dialect”) is the currency every later stage trades in: quantization inserts observers into it, lowering rewrites it, the partitioner cuts it up.

2. Quantization

Quantization maps floats to a low-bit grid:
```
x_q = clamp(round(x / s) + z),        x_hat = s * (x_q - z)
```
where s is the scale and z the zero-point. Two decisions define a scheme: symmetric (z = 0, one scale for a symmetric range — standard for weights) vs asymmetric (nonzero z, tighter fit for skewed activations like post-ReLU), and granularity: one (s, z) per tensor vs per channel (near-mandatory for weights, especially depthwise convolutions where channels have wildly different ranges).

PTQ vs QAT. Post-training quantization picks (s, z) after training by running representative data through the model and recording activation ranges (“calibration”: min-max, moving-average, or histogram/KL methods that trade a little clipping for finer resolution of the dense region). Quantization-aware training instead inserts fake-quantize ops during training so the network adapts its weights to the grid; it recovers the last 1–2% of accuracy on sensitive models but costs a retraining loop. On-device practice is overwhelmingly PTQ first, QAT only where accuracy demands it.

Where it happens in the ExecuTorch flow matters: quantization runs before lowering, in PyTorch land (prepare_pt2e → calibrate → convert_pt2e, with a quantizer such as XNNPACKQuantizer for CPU or a backend-specific one like QnnQuantizer that matches the NPU’s exact numeric behavior). The output is a graph whose linear layers carry quantized weights — the artifact lowering will consume.

3. Optimizations during quantization: fusion

The single most important quantization-time optimization is operator fusion, and batch-norm folding is the canonical example. At inference time batch norm is an affine map, y = γ(x−μ)/σ + β, so it can be absorbed into the preceding convolution algebraically:
```
W' = W · γ/σ,        b' = (b − μ) · γ/σ + β
```
Fusing conv+bn (+relu) before quantizing buys three things at once: the BN parameters and their memory traffic disappear; the quantizer sees one operator with one scale/zero-point instead of three chained ones (every quantize/dequantize boundary injects rounding noise, so fewer boundaries means less error); and the backend later receives a single fused kernel instead of three dispatches. In eager PyTorch this is fuse_modules(model, [["conv", "bn", "relu"]]); in the export-based flow the same fusion is expressed as graph patterns the quantizer recognizes. The principle is identical: never quantize across a boundary you could have erased.

4. Lowering, step by step

Lowering converts the portable graph into an executable artifact. In ExecuTorch it is an explicit, inspectable sequence:
1. Edge dialect. to_edge() rewrites the ATen graph into a constrained opset: functional (no mutation), no data-dependent control flow, no dynamic shapes beyond declared constraints. If your model can’t be expressed here, it can’t run on-device — this stage is where that verdict arrives early instead of at 2 AM on hardware.
2. Decomposition. Complex ops are broken into core ATen ops via the decomposition table (e.g., a layer_norm becomes reductions and elementwise math). Quantized layers appear in their decomposed form — dequantize → fp-op → quantize sandwiches — which looks verbose but is deliberate: it exposes the boundaries the next passes will optimize.
3. Partitioning / delegation. A backend partitioner tags the subgraphs an accelerator supports (“this conv chain can run on the NPU; this custom op cannot”). Each tagged partition is handed to the backend’s ahead-of-time compiler, which returns an opaque blob (compiled NPU binary, Core ML model, QNN graph). Untagged ops stay on CPU (XNNPACK) as fallback.
4. Finalization. to_executorch() emits the .pte file: bytecode for the CPU parts, delegate blobs for the NPU parts, and a static memory plan.
```
edge = exir.to_edge(quantized_program)     # 1+2: edge dialect, decomposed
edge = edge.to_backend(QnnPartitioner())   # 3: delegate NPU subgraphs
with open("model.pte", "wb") as f:         # 4: executable artifact
    f.write(edge.to_executorch().buffer)
```
5. Optimizations during lowering

With the graph fully visible, the lowering passes do the work that makes on-device execution fast:
- Quantize/dequantize cleanup. The sandwiches from step 2 are matched back into single quantized kernels (qconv, qlinear) wherever producer and consumer scales permit — the rounding-error equivalent of the fusion in section 3, now at graph scope.
- Constant folding and propagation. Anything computable at compile time (reshapes of constants, folded scales, static slices) is evaluated once on the host and baked in as bytes.
- Dead-code and common-subexpression elimination. Export often leaves redundant transposes, unused branches, and duplicated shape math; these passes strip them before they cost cycles or bytes.
- Layout transformation. NPUs typically want channels-last (NHWC) while PyTorch defaults to NCHW. The lowering inserts layout conversions and then pushes them to the graph boundaries, so the entire interior runs in the accelerator’s native layout with conversions only at input/output.
- Static memory planning. Because shapes are frozen, every tensor’s lifetime is known at compile time. The planner allocates one arena and reuses buffers whose lifetimes don’t overlap — no allocator, no fragmentation, no malloc in the hot path. This is a large part of why .pte files boot instantly and sip RAM.
- Partition hygiene. Every CPU↔NPU boundary costs a copy and a synchronization. A model split into twenty tiny partitions can run slower than fewer, larger ones; good lowering (and good model design) minimizes boundary crossings, sometimes by leaving a fusible op on CPU-adjacent fallback rather than shattering a partition.
6. At runtime

The ExecuTorch runtime loads the .pte, maps the arena, and walks the bytecode: CPU ops execute via kernels (XNNPACK), delegate blobs execute on the NPU through the backend driver, tensors hand off at the planned boundaries. If a delegate call fails, execution can fall back to CPU — graceful degradation instead of a crash, bought by the partition structure from section 4.

7. A practical checklist
1. torch.export a representative, eval-mode model; confirm it replays numerically.
2. Fuse (conv+bn+relu at minimum) before choosing quantization parameters.
3. PTQ with calibration data that covers deployment inputs; per-channel weights; check accuracy — QAT only if needed.
4. to_edge, inspect the partition report: which ops fell back to CPU, and why?
5. Iterate on fragmentation: unsupported-ops islands are usually fixed by decomposing, replacing, or pre/post-processing them on CPU by design.
6. Measure on hardware with the backend’s profiler — delegate time vs boundary-copy time tells you whether to fuse more or partition less.
Closing

The pipeline is long, but each stage has one job: quantization shrinks the data, fusion erases boundaries, lowering reshapes the program for the hardware, and the runtime just walks the plan. Internalize that division of labor and every accuracy drop or perf cliff maps to exactly one stage — which is what makes on-device deployment debuggable instead of mystical.

Back to Home

1. Mode 1: AR simulate — the hook

2. Mode 2: measure g with a stopwatch — the real lab

3. Mode 3: auto-track — color centroid + least-squares fit

4. Mode 4: hand-hold — the hard vision problem

5. Lab assistant: local model, scoped context

What I’d do differently

1. The harness bug that scored everything 0.000

2. Naive self-debug doesn’t work (I checked)

3. HumanEval: 86.0% → 91.5% (and 8 of 9 fixes generalize)

4. MBPP+: lower floor, stricter ceiling

5. Caveats, stated plainly

6. Reproduce it

The core trick: the browser calls Ollama directly

Not all local models tool-call equally

The key problem, and the loopback proxy

Debugging notes

1. Core Architecture & Execution Loop

2. Model Routing: Local By Default, Cloud on Demand

3. Persistent Two-Tier Memory Across Sessions

4. The 5-Tier Resilient Edit & Patch Engine

5. Safety: Pre-Mutation Checkpoints, /undo, and /diff

6. Takeaways

0. Why this pipeline exists

1. The starting point: an FP32 model and a captured graph

2. Quantization

3. Optimizations during quantization: fusion

4. Lowering, step by step

5. Optimizations during lowering

6. At runtime

7. A practical checklist

Closing