Project Updates

VWM → FRAME: Migration to a Spatial Computation dApp

A deliberately uncomfortable lead-architect review — docs/FRAME_MIGRATION.md. The finding: V26–V30 built a platform kernel, and FRAME is that kernel. Transactions, execution authority, snapshots-as-storage, replication, vector clocks, trust, signatures, federation, capabilities, invariants, and distributed history are now redundant and must be deleted or handed to FRAME. Layered on the ~3,800-line pre-V8 integration sprawl, roughly 45–55% of the codebase should leave.

New architecture

The critical calls

• transactions/ → FRAME canonical mutations. VWM keeps only a pure apply(state, op) → state' — no lock, no revision, no commit. FRAME owns ordering and atomicity.
• replication/, trust/, snapshot/, kernel/ (as authority), invariants/ (as kernel) → deleted. Networking, identity, signatures, capabilities, grants, storage, history, convergence are all FRAME primitives.
• runtime/ event bus → mostly deleted. The reactive wiring survives as a pure deterministic recompute graph; the event log / replay / networking → FRAME history / receipts.
• Identity → spatial entity continuity, not authority → stays as payload computation (renamed knowledge/continuity to avoid confusion with FRAME identity). Trust-identity-as-authority → removed.

What survives — and it's clean

A small, sharp deterministic surface — apply / project / query: voxel data structures, SVO indexing, geometry processing, reconstruction, deterministic simulation, and semantic / identity / reasoning as payload computation (authority, persistence, and human-verification stripped out). Rendering (compiler / Gaussian / neural / visualization) externalizes into one or more Voxel View dApps.

Two honest caveats

• FRAME contract unverified. The migration doc does not have FRAME's actual API in this repo, so every "→ FRAME primitive" mapping (mutation shape, FrameUnit payload limits, receipt/provenance overlap, grant vocabulary) is a contract to verify against real FRAME docs first.
• Determinism is make-or-break — and already proven. FRAME replay needs deterministic dApps; VWM's determinism is guarded by the V30 canonical-world regression harness. The migration is a deletion-led refactor guarded by golden-master checksums, not a redesign — so the hardest part is done; the rest is subtraction and drawing the boundary.

V30, Kernel Finalization & Long-Term Stability

V30 is the final planned kernel milestone. It eliminates the last architectural shortcuts and declares Kernel v1.0 — a deterministic, transactional, evidence-driven, distributed spatial world kernel that stays stable while perception, simulation, rendering, robotics, and applications evolve independently above it.

What shipped (all 10 parts)

• Part 1 — Observation-driven identity — replication exchanges only raw, content-addressed observations; each replica independently recomputes matches/confidence/lifecycle. No object state crosses the wire; old object-state copy path removed.
• Part 2 — Real public-key trust — pure-Python Ed25519 (RFC 8032): sign with private, verify with public; tampered/wrong-key rejected. Pluggable; HMAC remains the fast default for the high-frequency sync path.
• Part 3 — Trust federation — owner/organization/public-key/trust-chain metadata; kernel validates chain terminates at a trusted authority.
• Part 4 — Policy evolution — PolicyRegistry with pluggable migrations + evolution report (compatible / migratable / incompatible).
• Parts 5–6 — scheduled anti-entropy + replication health score (0..1).
• Part 7 (headline) — deterministic regression harness: TinyRoom / Hallway / Apartment / Office / Campus, each proven stable across replay + snapshot/restore + replication.
• Parts 8–10 — docs/GOVERNANCE.md, docs/adr/ (ADR-0001…0005), docs/KERNEL_API.md (Stable / Experimental / Internal / Deprecated), docs/KERNEL_V1.0.md.

After v1.0 — no more V31/V32 kernel milestones

Platform tracks version independently above the stable substrate:

Honest Scope Notes

• Ed25519 is a pure-Python reference (~1 s/op) — genuinely public-key and tested, but slow; HMAC remains the fast default; a native ed25519 backend is the eventual default (Experimental in KERNEL_API.md).
• Regression harness — build() activates sync first so the commit→log hook is installed; snapshots are not in the replication log (documented).
• Content-addressed identity converges replicas; wiring the live identity reactor to content-address end-to-end is a documented adoption step via observe(deterministic_id=…).
• Scheduled anti-entropy is a throttled tick (deterministic + testable); background OS thread + Merkle-range reconciliation are follow-ups.
• Policy migration provides the interface + detection; concrete cross-version migrations are registered by applications.

V29, Distributed Kernel Hardening & Trust

V29 completes the distributed guarantees left simplified in V28. Replicas no longer merge semantic/identity/reasoning conclusions by UUID — they exchange authoritative operations (evidence, facts), and each replica recomputes derived confidence/hypotheses/facts locally. No confidence value is ever copied across the wire.

What shipped — vwm/trust/ (world.trust) + operation replication

• Trust model — kernel-enforced replica capabilities, trust levels, roles (viewer / robot / simulation / annotator / human_admin / peer), and HMAC-SHA256 signatures. Every imported operation is authorized before apply (viewer denied semantic ops; only admin may verify).
• Operation replication — operation, operation_log, operation_codec, and semantic/identity/reasoning deltas capture evidence as operations and apply with local recompute. Reasoning replicates observation facts and re-infers derived facts identically.
• Signatures + Byzantine safety — packages signed and hash-checked; forged/tampered packages rejected before any mutation.
• Provenance — every operation carries origin replica/transaction/revision + signature status + trust policy; distributed invariant asserts nothing loses its origin.
• Versioned policies — differing confidence policy reported incompatible and not merged.
• Scale & repair — log compaction (replay-preserving), partial sync (region/class/revision), subscriptions, anti-entropy (detect + repair), distributed invariants.

End-to-end scenarios (Part 13 — all covered)

• Robot evidence → human verify → sync → confidence recomputed → checksums converge
• Conflicting policies report incompatibility (no silent merge)
• Log compaction preserves replay
• Region sync leaves other rooms unchanged
• Malicious / forged signature rejected, no mutation
• Long-offline + checkpoint + incremental replay + convergence
• Subscription filters updates
• Delete derived reasoning → replay evidence → regenerated identically

Architecture Decision Records

ADR process in docs/adr/: ADR-0001 replicate operations not conclusions · ADR-0002 vector clocks · ADR-0003 transactions are authoritative · ADR-0004 HMAC-now / pluggable public-key.

Honest Scope Notes

• Signatures are HMAC (symmetric) — real integrity given key distribution; ed25519 is the documented drop-in behind the same interface (ADR-0004). No cryptography/nacl in this environment.
• Policy migration is detect-only — incompatibility reported, not transformed.
• Anti-entropy is on-demand (not a background scheduler); Merkle-range reconciliation is a follow-up.
• Identity ops copy objects + match-evidence by uuid and recompute confidence; full per-observation identity streams are a follow-up — the headline evidence-recompute guarantee is fully realized for semantic + reasoning.

V28, Distributed World Replication & Synchronization

V28 turns the single-process V20–V27 kernel into a replicated distributed world: many replicas, one logical world, converging exclusively through validated transactions — never by copying memory. Networking is merely one transport; the kernel depends on none.

What shipped — vwm/replication/ (world.sync)

• Replica & logical-world identity — stable ReplicaID; one shared world_uuid.
• Replication log first-class — a _replication_hook on the TransactionManager logs every commit’s effective changeset + vector clock.
• Vector clocks — causality (before / after / concurrent / equal) without wall-clock ordering; a conflict is flagged only when clocks are concurrent and touch overlapping state.
• Explicit conflicts + pluggable resolution — concurrent-geometry (default: manual, never silent overwrite), concurrent-semantic (default: preserve-alternatives — both classifications kept, evidence retained), schema-mismatch (reject). Strategies: reject / last_writer / preserve_alternatives / merge_if_safe / manual / custom.
• What replicates — geometry via incremental transaction log (replayed on import); semantics / identity / reasoning merged by uuid (globally unique ⇒ no duplicates across replicas).
• Checkpoints — late replica resumes then replays the tail; abstract transport (in-memory + file, both real); schema negotiation; offline workflow.

Eight required end-to-end scenarios (all pass)

• Two cameras — converge, no duplicate entities
• Offline robot — no lost updates
• Concurrent semantic — both hypotheses preserved, evidence retained
• Concurrent geometry — conflict detected (default: manual)
• Checkpoint recovery — checksum matches
• Multi-robot — three replicas converge
• File transfer — no network dependency
• Schema version — graceful rejection, no corruption

Plus sync idempotency / determinism and test_replication_never_violates_invariants.

Honest Scope Notes

• Hallway snapshot gotcha (handled honestly): the server’s loaded hallway was read from a snapshot, not through the replication log — so the live demo uses two fresh transactional replicas to show genuine convergence rather than pretending snapshot geometry replicates.
• Transports: in-memory + file only. HTTP / WebSocket / QUIC / USB share the Transport contract but are not implemented — deliberate; the kernel depends on none of them.
• State-description merge reconciles by uuid through evidence APIs; fine-grained per-transaction semantic deltas are a documented follow-up. Geometry always replicates as incremental transactions.
• Replica discovery is an abstract connect/disconnect interface, not a networking / gossip stack. Cross-process transport and simulation-state conflict are interface-level.

V27, World Services & Query Engine

V27 turns VWM from an internal engine into a platform: applications talk to one interface — world.query — instead of importing semantic, identity, or reasoning internals. This is the cross-layer query package V26 deferred, now delivered.

Query families (all tested)

• Spatial — nearest / inside / contains / intersects / adjacent / line_of_sight (SVO) / visible / reachable (free-space BFS)
• Temporal — changed(since=…) / state_at / timeline / between
• Semantic — entities / find (Django-style confidence__gte) / classes (subclass-aware)
• Identity — objects / track / versions / lifecycle / merge_candidates / moved_more_than
• Reasoning — why / causes / effects / alternatives / predictions / explain_access
• World graph — Entity / Identity / Fact / Evidence / Event / Room nodes with traversal

Honest Scope Notes

• ask() is a transparent keyword parser, not NLP/LLM — patterns only; a learned understander slots behind the same entrypoint.
• Planner is index-vs-scan, not a cost-based optimizer.
• V10 component DSL preserved as world.query.select(...) / world.query_components(qs).
• No package collision — vwm/query/ is distinct from vwm.worldos.query.

V26, Integrated World Kernel

V26 turns V20–V25 into one coherent, event-driven, transactional, persistent world kernel. No decorative feature layer — it tightens, connects, and hardens the existing architecture, and proves it with real end-to-end execution.

What shipped

• vwm/transactions/ — atomic WorldTransaction + single-writer TransactionManager: the only place voxels and revision change. Conflict / nest rejection, complete rollback.
• vwm/core/evidence/ — canonical noisy-OR confidence/hypothesis/review; semantic + reasoning delegate to it.
• vwm/kernel/ — WorldKernel + reactors: V21 is the nervous system for V23–V25 (idempotent, correlation/causation-traced).
• vwm/snapshot/ — whole-world .vwm save/load with manifest, per-section + whole-file checksums, staged validate-then-restore.
• vwm/invariants/ — 7 executable kernel invariants (audit/dev/test/production modes).
• Product test fixed — long-broken test_vwm_product_e2e.py now passes (5 tests); shadowed reconstruction import restored.

Honest Scope Notes

• Incremental snapshots — save_incremental() returns {"supported": false}; full snapshots only.
• Cross-layer query package — deferred here; delivered in V27 as world.query.
• Learned perception — not added; ingested class label is synthetic ground-truth; the wiring is the deliverable.
• No dedicated many-worker concurrency stress suite; single-writer + failure-injection cover correctness.

V25, World Reasoning Engine

The fourth layer — reality → meaning → continuity → inference — produces evidence-backed conclusions without ever mutating the authoritative world. Reasoning never owns truth: it proposes, evidence updates confidence, and confidence yields immutable facts.

The Stack (vwm/reasoning/)

Package: inference · rules · graph · causality · hypotheses · confidence · explanation · planner · persistence · diagnostics · engine · api. Entry: world.reasoning → WorldReasoningEngine.

Definition of Success (all tested)

• Enforced inference chain — observation → rule/AI proposal → evidence → confidence → immutable WorldFact; nothing skips it.
• Classical rules without AI — door→accessible, breaker→outlet→lights, chair→occupancy, vehicle→spot-empty, disappearance→competing explanations.
• Causal graph — chains with confidence weakening down each link.
• Competing hypotheses — kept until resolved; later observation resolves without deleting alternatives.
• Explanations — why / evidence / confidence / assumptions / causal chain / alternatives — no black box.
• Predictions as hypotheses — never overwrite current facts.
• Non-destructive — observe/infer/predict leave voxels, semantics, and identities byte-identical.
• AI via same pipeline — register_reasoner; humans always override.

Honest Scope Notes

• Classical rules only — no AI or learned model bundled; disappearance confidences are labelled priors.
• Prediction is rule-based persistence projection, not a learned forecaster.
• AI reasoner path is an interface (proven with a trivial local reasoner); future hooks are plugins with no model dependency.

V24, Persistent World Identity

VWM shifts from a snapshot to a continuously-evolving world. PersistentObject identities survive across observations and geometry/semantic change — inferred from accumulated evidence, never a single sighting. Immutable append-only history enables replay and change detection without touching the present.

The Stack (vwm/identity/)

Package: object · observation · matcher · tracker · lifecycle · history · versions · confidence · registry · persistence · diagnostics · engine · api. Entry: world.identity → PersistentIdentityEngine.

Definition of Success (all tested)

• Identity survives change — moved chair keeps UUID; repainted wall keeps UUID while attributes update.
• Probabilistic multi-signal matcher — semantic/geometry/colour/relationship/spatial/motion/history → confidence, not certainty. Low spatial weight so moved objects match; size-dominated geometry so a cup ≠ a chair.
• Accumulated confidence — single sighting provisional (~0.5); firms via noisy-OR over observations.
• Ambiguity explicit — competing hypotheses, never a silent pick.
• Immutable history — observations, timestamped versions, lifecycle transitions permanently recorded; deleted objects stay queryable.
• Change detection + world timeline — moved/rotated/resized/recolored/damaged/repaired/removed/added/merged/split/relationship_changed as evidence-backed events.
• Replay without mutation — state_at(t) / replay(t) / compare(t1,t2) reconstruct history from the log.
• Pluggable future models — set_matcher / register_embedder interfaces only; no bundled model.

Honest Scope Notes

• Matcher is a transparent weighted heuristic — real probabilistic matching with per-signal breakdown; not a learned embedding model. Learned identity plugs into the extension interfaces with zero engine change.
• Auto merge/split is conservative — explicit merge()/split() APIs record events; learned automatic merge/split is a follow-up.
• Capture→detection→identity auto-wiring is a natural next integration via observe_entity on V23 entities; the engine and temporal guarantees are complete here.

V23, Semantic World Engine

Meaning lives in a layer fully decoupled from geometry. WorldEntity objects own hypotheses, attributes, and relationships — they only reference authoritative voxels, never store or mutate them. One mutation path: add_evidence → recompute hypothesis. AI and heuristics propose; humans verify or override.

The Stack (vwm/semantic/)

Package: entity · hypothesis · ontology · relationships · confidence · evidence · registry · queries · persistence · diagnostics · engine · api. Entry: world.semantic → SemanticWorldEngine. Distinct from legacy vwm/semantics.py (plural), which is untouched.

Definition of Success (all tested)

• Geometry ↔ semantics decoupled — entities reference voxel_refs / component_ref; no geometry storage or mutation.
• Competing hypotheses — multiple classes per entity with confidence; threshold-parameterised primary class.
• Evidence-only mutation — noisy-OR supporting evidence, refuting attenuation; human verify/override pins confidence (source="human").
• Directional relationships — contains↔inside, parent↔child auto-inverse; queryable and persistent.
• Subclass-aware ontology + queries — find, search (lexical), connected, relationships.
• Independent persistence — reload into a different world instance without geometry recompile.
• Browser inspection — list, search, live diagnostics, per-entity detail, 3D highlight (voxel→scene Y-flip).

Honest Scope Notes

• HeuristicProposer is a small shape-based classifier (thin-tall-planar → Wall, etc.), labelled source="heuristic" — not deep recognition.
• search() is lexical term overlap, not embeddings.
• No perception model bundled. Real detectors attach through the same propose(..., source="ai") path; V22 neural registry is the natural home.
• Every hypothesis is colour-coded proposed vs verified — proposals are never shown as truth without evidence or human decision.

V22, Neural Enhancement Backends

Neural models are interchangeable optimization backends behind one stable interface — never the renderer. The Gaussian Evolution Engine drives them purely through a registry; no engine code names a specific model. The Adaptive Voxel World stays authoritative.

The Stack (vwm/neural/)

Package: backend · registry · manager · models · scheduler · cache · jobs · settings · diagnostics · api. Entry: world.neural → NeuralManager. Evolution holds a NeuralBackendOptimizer that only calls backend.optimize(scene).

Definition of Success (all tested)

• Interchangeable backends — common interface; engine never names FLUX/RealESRGAN/…
• Real GPU path — RealESRGAN RRDBNet x4 on the 2060 SUPER; four real-inference tests ran (not skipped) with weights present.
• VRAM management — estimation, overlap-blended tiling, CPU fallback on CUDA error, idle unload.
• Graceful fallback — OOM / missing weights → classical floor; never crash.
• Background — scheduler thread; render/sim/capture keep consuming the best cached scene.
• World untouched — neural pass produces a new cached Gaussian revision only.

Honest Scope Notes

• classical + realesrgan are real GPU/CPU-executing paths. flux_schnell / swinir / hidream have real code paths but missing weights → available() reports it; optimize raises rather than faking.
• Scene update is a bounded neural-gated colour-detail transfer (new checksum), not full inverse-rendering back-projection onto splats — a documented follow-up. V19 quality_score can stay flat on colour-only changes; neural gain is detail_gain.
• Pre-existing neural.py moved to neural/_legacy.py with re-exports (same class of fix as V21’s runtime).

V21, Reactive World Pipeline

Execution model shifts from orchestration (A calls B calls C) to reaction: subsystems publish immutable RuntimeEvents and any interested peer reacts independently. The Runtime is the backbone; the Adaptive Voxel World stays the single source of truth.

The Stack (vwm/runtime/)

Package modules: event · bus · dispatcher · subscriptions · filters · recorder · replay · bridge · reactors · network · diagnostics · api. Entry: world.events → ReactiveRuntime.

Definition of Success (all tested)

• Immutable, serializable events — frozen RuntimeEvent + 22 canonical types; seq-stamped for replay.
• Event bus — publish/subscribe/unsubscribe, monotonic ordering, subscriber error-isolation, re-entrancy-safe drain, filters.
• Genuine reactivity — LegacyBridge republishes real V11 engine events; V15 auto_wire is the seed; reactors extend Evolution → Viz → Enhancement.
• Independent reactor removal — each reactor is a pure subscriber.
• Record → deterministic replay — .timeline save/load + play/pause/seek/step/windows.
• Networking model — serializer + loopback channel (in-process reference transport).
• World untouched by the reactive layer itself.

Migration Handled Honestly

Pre-V8 runtime.py (imported by many modules) would have been shadowed by the new runtime/ package. It moved to runtime/_legacy.py with re-exports — legacy tests still pass. Documented in docs/V21_REPORT.md.

Honest Scope Notes

• V11 mark_dirty still schedules the compiler directly inside V11. V21 makes cross-subsystem orchestration event-driven and bridges real engine events; it does not rewrite V11’s internal scheduler.
• Reactors do lightweight coordination — heavy work stays in V15/V19/V16/V18, off the synchronous event thread.
• LoopbackChannel is in-process (no socket). Production transport / auth / conflict resolution are follow-ups.
• SimulationTick / Playback* are in the canonical vocabulary but not yet emitted from those sources — incremental migration.

V20, Reality Capture Pipeline

One unified doorway for all reality intake. Any source — phone, drone, robot, LiDAR, webcam, video, image folder, synthetic renderer, or recorded replay — emits a standardized ObservationPacket. The pipeline routes each packet through Registration + Evidence into the voxel world. It observes reality and never writes voxels directly.

The Stack (vwm/capture/)

Modules: observation · device · registry · calibration · synchronization · recorder · playback · stream · formats · pipeline · diagnostics · settings · api. Entry: world.capture.

Definition of Success (all tested)

• One ObservationPacket for every source (rgb/depth/pose/intrinsics/imu/gps/confidence/timestamp/metadata).
• Routing invariant — voxels_added == len(world.coords) from capture path; all via Registration + Evidence.
• Multi-device timestamp sync · recording + identical replay · playback controls · independent calibration · registry extensibility.
• Guarded hardware — webcam/rtsp/lidar/depth_camera/drone report exact blockers; never fake frames.

Honest Scope Notes

• Real hardware drivers are guarded stubs until wired; adding one is subclass CaptureDevice + register.
• RGB-only sources return needs_reconstruction — the pipeline never invents depth.
• Distortion undistortion / extrinsic rig solving are stored but not yet applied in fusion math.

V19, Gaussian Evolution Engine

Clean complement to V18: V18 enhances rendered pixels (the Gaussian image); V19 continuously improves the Gaussian scene data itself — merge / split / opacity / covariance / … — asynchronously, producing new cached scene revisions. The Adaptive Voxel World is untouched throughout.

The Stack (vwm/gaussian_evolution/)

Every module the spec listed: engine · optimizer · scheduler · jobs · queue · cache · metrics · registry · diagnostics · api · settings. Entry: world.gaussian_evolution.

Definition of Success (all tested)

• Optimizers on Gaussian data, not pixels — 8 classical optimizers (noise removal, merge, split, opacity, covariance, colour, overlap-repair, compression) through one GaussianOptimizer.optimize(scene)→scene interface.
• Async, non-blocking — persistent job queue + background worker thread; rendering / simulation never wait.
• Progressive quality → new cached revision per pass — nothing overwritten; history preserved (hallway 0.65 → 1.0).
• Voxel world unchanged throughout (test_world_unchanged_by_evolution).
• Modular / registerable — a new optimizer (brighten) is class + register; engine never changes.
• Scheduler — priorities, cancel, pause, resume; persistence survives restart (cache + queue).
• Browser transparency — V18 path pulls best_arrays() when a better cache exists.
• Heavy AI as optional guarded plugins — FLUX / HiDream registered, report exact blocker, raise rather than fabricate, skipped (recorded) when unavailable.

Honest Scope Notes

• Heavy AI (FLUX / HiDream) needs GPU weights not present here. The tested path is classical splat processing — measurable quality gains, not foundation-model refinement. Plugging a real model in is implement GaussianOptimizer + register.
• Quality metric is a heuristic over splat data (opacity health / overlap / coverage / confidence). A perceptual (rendered) metric — reusable with the V18 rasterizer — is a natural refinement.
• Predictive optimization (camera-motion prefetch) is scaffolded via the scheduler’s trigger / priority model; the motion predictor itself is a documented follow-up.

V18, Gaussian Enhancement Pipeline

Scope correction from “frame enhancement”: V18 makes the Gaussian render itself better — enhancement of rendered Gaussian imagery, conditioned on scene metadata — not full-frame / display post-processing. That later whole-frame stage stays an optional visualization capability. The voxel world is never touched.

The Stack (vwm/gaussian_enhancement/)

Every module the spec listed: packet · metadata · embedding · enhancer · registry · engine · settings · diagnostics · api. Entry: world.gaussian_enhancement.

Definition of Success (all tested)

• GaussianImagePacket — rgb + depth + normals + motion + semantic + material + object-ids + confidence + metadata.
• Packet-only — enhancement never mutates the Adaptive Voxel World.
• Metadata conditioning — visible-region metadata → compact 32-D model-agnostic embedding (never raw voxels / whole world).
• Disabled = passthrough — identical to the raw Gaussian render.
• Modular chain — one GaussianEnhancer.enhance(packet)→packet; profiles scale low → medium → high → cloud.
• Renderer-independent — a packet built from scratch enhances fine.
• Extensible — sepia shown as class + register only.
• Browser-transparent — requests Gaussian renders; receives an enhanced image via the Visualization Engine path.

Profiles

• low — empty pipeline (passthrough)
• medium — upscale → detail → material
• high — + edge_aa → temporal
• cloud — AI stages (guarded / skipped without weights)

Honest Scope Notes

• AI photorealism / super-resolution need GPU weights not present here. The tested path is classical-CV (upscale / detail / material-conditioned / edge-AA / temporal / stylize). AI enhancers (RealESRGAN, photorealistic, reflection) are guarded stubs that report the exact blocker and raise rather than fabricate — plugging a real model in is implement enhance + register, no engine change.
• CPU splat rasterizer is a reference rasterizer, not the browser WebGL splat renderer. It makes the packet + pipeline real and testable without a GPU. Wiring the browser to display the enhanced image (vs live WebGL splats) is a small display-mode follow-up; /api/genh/image already returns the enhanced frame.
• Metadata unprojection is subsampled for speed; denser / GPU pass is future work.

V17, World Simulation

VWM is now alive: simulation evolves the authoritative Adaptive Voxel World through time. Up to V16 the system answered “what is the world?”; V17 answers “what is the world doing?” — and it adds no new foundational layer. It uses the Runtime (V11), Evidence Engine, Voxel World, Compiler (V15), and Visualization (V16) already built.

The Stack (vwm/simulation/)

Every module the spec listed: engine · systems/behaviors · scheduler · timeline · events · triggers · state · diagnostics · api, plus continuous (the preserved pre-V17 WorldClock / EventScheduler API). Entry points: world.simulation and world.tick / pause / resume / step / play.

Behaviors, Triggers, Time

• Behaviors as data — Behavior.update(obj, ctx, dt) attached to objects. Built-ins: door · battery · light · traffic_light · vehicle · robot · tree · river · weather.
• Interactions proven — button→door, battery→light (drains to 0), traffic-light green/yellow/red cycle, proximity→open.
• Triggers — manual / button / time / schedule / distance / region / sensor / signal / collision.
• Time control — pause / resume / replay / fast-forward / slow-motion / single-step via the Runtime clock.
• Events on the V11 bus — DoorOpened / DoorClosed / BatteryEmpty / LightChanged / VehicleMoved / RobotArrived / WeatherStarted-Stopped + SimTick / TriggerFired; DoorOpened and VoxelChanged both fire on a door open.
• Determinism — same world + events + seed → identical timeline (tested).
• Extensibility — a new behavior (sprinkler shown) is just a class attached to an object; the engine never changes.

Migration Handled Honestly

A pre-existing vwm/simulation.py (WorldClock / EventScheduler / CONTINUOUS_TYPES, imported by vwm.components, vwm.voxel_world, vwm.persistence) was shadowed by the new package. It moved to vwm/simulation/continuous.py with re-exports from the package __init__ — every legacy import still works, and all prior tests pass. Documented in docs/V17_REPORT.md.

Honest Scope Notes

• Behavior realism is illustrative — the milestone prioritizes architecture over realism; rich physics / animation is future work on the same Behavior interface.
• Browser play/pause/step strip is a small follow-up (API + endpoints exist).
• Revision on change — simulation bumps the runtime revision per world-changing action so the compiler cache invalidates correctly; world edits should do the same (general improvement surfaced here).

V16, Universal Visualization Engine

VWM now has one Visualization Engine that renders the authoritative world in many ways. Every renderer consumes compiler artifacts (V15), never the voxel world directly; switching views never recompiles; visualization never modifies the world — it is just another observation. This caps the major subsystem stack: one Adaptive Voxel World → one World Compiler → one Visualization Engine → viewer.

The Engine (vwm/visualization/)

Every module the spec listed: engine · renderer · renderers · registry · camera · viewport · overlays · screenshots · recording · settings · diagnostics. Sole entry point: world.visualization / VisualizationEngine.render(...).

17 Renderers, All Consuming Artifacts

• Geometry (10): voxel · gaussian · mesh · wireframe · normals · semantic · materials · lighting · occupancy · bounding_boxes
• Debug (7): chunk_boundaries · dirty_regions · registration_graph · evidence_confidence · compiler_artifacts · component_hierarchy · runtime_events

What Was Verified

• Artifacts-only invariant — Scene gates access; every description carries source: "compiler_artifacts".
• Hot switching — test_hot_switch_never_recompiles: switch + render adds zero compilations.
• Multi-viewport — same scene, different renderers, no recompile.
• Cameras — orbit / first_person / fly / top_down / isometric.
• Selection — select(point) → voxel → component → artifacts → evidence → revision → bbox.
• Screenshots / recording / time — 4K & transparent screenshot descriptions; camera-path / frames / time-lapse; pause / play / step / replay.
• World untouched — rendering leaves world.coords unchanged.
• Extensibility — test_extensibility_register_only: class + register(); engine unchanged.

Honest Scope Notes

• Browser migration: the Python engine produces render descriptions and the endpoints are live, but the interactive Three.js viewer still draws via its own V8–V15 representation code for live modes. Routing the browser’s actual pixel rendering fully through /api/viz/render for every mode is the remaining integration step — descriptions are equivalent and the artifacts-not-voxels invariant already holds engine-side.
• Stylized renderers (photorealistic / LEGO / anime / low-poly / thermal / IR) are the extensible target; built-ins cover geometry + semantic + debug. Each stylization is just a new Renderer over the same artifacts.
• Screenshots / recording produce descriptions/requests; the browser performs the actual pixel capture.

V15 Part 1, World Compiler Framework

VWM now has a single World Compiler Runtime that owns all compilation of derived artifacts. The Adaptive Voxel World remains the sole source of truth; every derived representation — Gaussian, SVO, mesh, semantic cache, lighting, and any future plugin — is an Artifact produced by a Compiler, orchestrated by one runtime. No more standalone compiler subsystems.

The Kernel (vwm/compiler/)

Every module the spec listed: runtime · compiler · registry · scheduler · dependency · artifact · cache · execution · invalidation · diagnostics · persistence · statistics · transactions · exceptions · utilities (+ plugins). Callable API surface: world.compiler.

Five Built-In Plugins + Chained Dependencies

Plugins wrap the real existing logic — gaussian, svo, mesh, semantic_cache — plus lighting, which consumes gaussian_scene. That dependency proves topological ordering (gaussian runs before lighting) and chained dirty propagation: one dirty region resolves to ordered (compiler, chunk) work; no compiler triggers another.

What Was Verified End-to-End

• Incremental — only intersecting chunks’ artifacts recompile (one-artifact-per-dirty-chunk).
• Transactional — prepare → compile → validate → publish → commit; invisible until commit; atomic swap; rollback restores the previous artifact on any failure.
• Cache — key is (compiler, chunk, revision, params); re-invalidating the same key → hits, no recompile.
• Determinism — content-hash dedup; identical revision+params → identical checksum.
• Validation — invalid output never publishes (gaussian: finite + opacity∈[0,1]; svo: node/leaf integrity; mesh: box indices).
• Error isolation — a throwing compiler fails alone; independent compilers still succeed; the world is never modified.
• Events — 10 compiler events on the V11 bus (CompilationStarted/Finished/Failed, ArtifactCreated/Updated/Removed, CacheHit/Miss, TransactionCommitted/RolledBack).
• Extensibility — test_extensibility_no_runtime_change: implement Compiler + register(); runtime untouched.

Honest Scope Notes (Part 1)

• Migration: the framework unifies compilation and plugins wrap real logic, but the viewer’s live Gaussian/SVO rendering still reads the V11 runtime.compilers path (kept so V11/V12 tests pass). Fully routing the viewer through world.compiler artifacts is Part 2; both paths produce equivalent artifacts.
• Concurrency: reference execution is a deterministic single-threaded drain; true parallel multi-worker compilation is a follow-up (the transaction / atomic-swap design already supports it).
• Persistence stores artifact metadata + graph + stats; payloads persist via each compiler’s own serializer (e.g. V12 Gaussian chunk .npz), not a single blob.
• Viewer diagnostics show stats; full compiler-queue / dependency-graph / timeline panels are follow-ups (endpoints exist).

V14, Complete Registration, Mapping & Global World Alignment

VWM now aligns any number of captures, sensors, and sessions into one continuously-improving world through a maintained pose graph — with global SE(3) optimization, loop closure, landmarks, anchors, uncertainty, and incremental corrections via the V11 World Runtime. Registration only improves the alignment of the evidence feeding the Adaptive Voxel World; the voxel world remains the sole source of geometric truth.

The Registration Subsystem (vwm/registration/)

Fifteen modules as specified: transform · uncertainty · graph · optimization · icp · loop_closure · landmarks · anchors · constraints · session · diagnostics · persistence · manager (plus supporting packages). SE(3) exp/log/inverse verified round-trip. The full API sits on world.registration (optimize / loop_closures / constraints / sessions / landmarks / anchors / statistics / rollback / preview / accept / reject / register_capture / add_loop_closure / …). V12’s callable world.registration() still works — the manager is callable.

First-Class Concepts

• Pose graph — nodes (capture / window / keyframe / landmark / anchor / sensor); edges with transform + 6×6 information + fitness / rmse / confidence / method / revision / provider / supporting observations.
• Loop closure — detect → verify → add edge → optimize → correct; drift removed by the added constraint.
• Landmarks — persistent, cross-capture association (not tied to one capture).
• Anchors — local / global / manual / survey / gps / robot / building; fix or prior a node.
• Uncertainty — covariance ellipsoids; information from fitness / rmse.
• Sessions — first-class, queryable mapping sessions.
• Map quality — coverage / density / registration-health / loop-health / scale-consistency / chi².
• Multi-capture fusion — many captures → one graph → one world; pending alignment for low confidence.

Benchmark (synthetic GT — artifacts/registration/benchmark.json)

Honest Scope Notes

• Synthetic GT only — no real sensors/model here; GPS / IMU / wheel-odometry / ROS / LiDAR live adapters are interface-only.
• ICP scale-from-identity and tiny-overlap are unreliable (benchmark shows it) — documented, not hidden; acceptance gate should be raised for low-overlap regimes.
• Numerical Jacobians — correct for small/medium graphs; analytic Jacobians + sparse Cholesky are the scale-up path. Full bundle adjustment over landmark observations is an interface, not a solver, in this milestone.
• Viewer follow-ups — diagnostics panel + graph/loop endpoints are delivered; in-viewport 3D pose-graph / trajectory / covariance-ellipsoid modes and the interactive constraint editor are next (all backend APIs exist and are tested).
• Incremental voxel correction requires per-capture voxel provenance via register_capture(..., voxels=…); older captures without that tracking correct at the pose-graph level but move no voxels.

V12, Streaming Reconstruction, Registration & Persistent Gaussian Scene

V12 wires a ReconstructionProvider stack into the V11 runtime: perception → observations → evidence → authoritative voxels → incremental Gaussian scene. LingBot-Map is vendored and audited (Apache-2.0, commit-pinned); registration closes the V10 cross-capture gap; and a real elliptical Three.js splat renderer visualizes the derived scene. The Adaptive Voxel World remains the sole geometric source of truth. Boundaries are stated honestly throughout — this environment has no CUDA GPU and no LingBot checkpoint, so real-model inference is unsupported rather than faked.

Phase 0 — LingBot Vendored & Audited

Cloned, commit-pinned (7ff6f3ed0913d4d326f8f13bbb429c4ffc0195c2), Apache-2.0 license preserved, 36-module source under third_party/lingbot-map/, full audit in docs/LINGBOT_AUDIT.md, and vwm third-party verify lingbot-map passes with zero drift. Adapter-only integration — no patches to upstream.

Provider Stack (Goals 1–2)

A ReconstructionProvider protocol with typed outputs, registry, and session model (world.reconstruction.providers() / create_session / ingest_frames / ingest_video / ingest_camera_stream). Two providers ship: a deterministic synthetic one that renders a real box-room (depth + RGB + confidence + exact poses, GPU-free) and drives the whole pipeline plus registration ground truth; and the real LingBot adapter that integrates at the GCTStream API level and reports a specific blocker via availability() when torch/CUDA/checkpoint are absent — it never fabricates predictions.

Streaming → Evidence (Goals 3, 7, 8)

Frame-by-frame ingest (video decoded incrementally, never whole-file), depth unprojection with explicit conventions, and positive surface + bounded negative free-space evidence into the V11 engine (never behind the surface). Confidence-weighted, full provenance per observation. Free-space retraction of low-confidence voxels is demonstrated. Nothing bypasses the Evidence Engine.

Registration (Goals 5–6) — Closes the V10 Gap

Real ICP (point-to-point / point-to-plane) + Umeyama scale+rigid, with an absolute-distance acceptance so non-overlapping clouds are rejected. Pending-alignment workflow: world.registration().preview / accept / reject / set_manual_transform. Measured on synthetic ground truth:

• Known 0.4 rad transform recovered to fitness >0.9 / rmse <0.05
• 2.5× scale recovered exactly
• Non-overlapping clouds rejected → pending alignment — never a silent merge

Persistent Gaussian Scene (Goals 12–16)

Chunked, derived from voxels + appearance + evidence; a V11 CompiledRepresentation. Proven incremental: adding one voxel recompiles exactly one chunk; appearance-only edit recompiles one chunk with no geometry change; 23/24 chunks keep identity + content hash. Binary chunk serialization with SHA-256 checksums and corruption detection. And a real elliptical Three.js splat renderer — instanced quads + gaussian-falloff shader, not point sprites — served by /api/gaussian.

Also Delivered

• Trajectory memory — world.trajectories.list / get / pose_at / compare, persisted per capture
• Keyframe policy — fixed_interval / motion / novelty / coverage / hybrid / lingbot_compatible
• Query extensions — provider:…, is_keyframe==true (plus V11 confidence / capture operators)
• 17 docs covering providers, streaming, registration, Gaussians, LingBot audit/integration, and benchmarks

Honest Scope Notes

• No CUDA GPU, no checkpoint here — real LingBot inference is unsupported; the adapter is contract-tested and the full pipeline is validated with the synthetic provider only. No fake real-model benchmarks were produced.
• Designed but not implemented this milestone: dynamic-scene safeguards, loop closure, LOD / frustum streaming, a full VRAM resource manager, and dedicated live-capture viewer panels — each flagged explicitly rather than faked.
• Registration is pairwise / bounded (no global pose-graph); long-sequence drift is not measured without the real model.
• Initial Gaussian compile is one full pass at server start (~7.3 s / 73k voxels); thereafter it is incremental. Background progressive initial compile is future work.
• FlashInfer KV-cache is not installed; SDPA fallback is the documented default.

V11, Persistent World Runtime & Evidence Engine

VWM now has a World Runtime: every modification flows through one service that accumulates evidence, propagates dirtiness, schedules incremental recompiles of only affected chunks, and emits events. Derived systems become incremental consumers, not independent world copies. The Adaptive Voxel World remains the sole source of truth — the architecture was not redesigned. The new package is vwm/worldos/runtime/: evidence · events · dirty · scheduler · depgraph · compiled · runtime.

The Twelve Goals, Delivered

• Evidence Engine — every voxel carries positive/negative evidence mass → confidence, provenance, and estimated error. Observations accumulate: agreement raises confidence, contradiction lowers it without overwriting (history preserved), with weighting, aging/decay, manual override, revision-awareness, and explicit-only garbage collection.
• Dirty-Region Runtime — a change propagates through all six layers (Voxel → Components → SVO → Appearance → Semantic → Chunks), each region carrying bbox, revision, priority, reason, dependencies, and timestamp. Never a global rebuild.
• Scheduler — a priority queue (Immediate / High / Medium / Low / Idle) with visible-first ordering, deterministic drain, and a background worker that never blocks editing.
• Dependency Graph — derived objects declare dependencies; reverse-indexed invalidation touches only dependents. A change to one chunk never touches unrelated ones.
• Event Bus — centralized pub/sub with 10 event types; nothing polls internally.
• Runtime API — mark_dirty / schedule / compile / events / pending_jobs / statistics: the single entry point for change, wired into World.ingest.
• Evidence queries — confidence>0.9, confidence<0.4, evidence_count>5, revision>=35, component==Door, material==Glass, capture==17, observed_by(x), updated_today() — all combinable.
• Runtime Inspector — the viewer panel above: live stats, confidence histogram, streaming event feed.
• World Statistics — computed continuously, polled live by the viewer (~1.5 s).
• Compilation Framework — a CompiledRepresentation interface (compile only dirty chunks → swap) with two working compilers, VoxelMesh and SemanticCache. Future Gaussian, navmesh, physics, lighting, or neural compilers plug in with no new sync logic.
• Incremental audit — docs/INCREMENTAL_MATRIX.md marks every subsystem Incremental / Partial / Global / N-A with justification.

Honest Scope Notes

• Confidence is uniform 1.0 on a fresh single-capture world — the accumulation/decay/conflict mechanism is complete and tested, but it only diverges as re-observations, conflicts, and decay accumulate, so today's histogram is flat.
• Per-voxel dirty/confidence recolor modes are partial — the panel, histogram, and endpoints are delivered; full per-voxel heatmap modes are minimal.
• Appearance-patch recompilation is not yet a runtime compiler (marked Partial in the matrix).
• Runtime persistence to disk is in-memory in this build — the revision/evidence models are serializable; disk persistence is future work.
• Evidence is a per-voxel Python dict — correct and tested; vectorization for very large worlds is future work.

V10, Universal World Engine

VWM now behaves like a world operating system: drag almost any capture or asset in, and the system classifies it, converts it to observations, updates the authoritative Adaptive Voxel World, incrementally rebuilds the SVO, updates Components and revisions, and visualizes it — with every rendered voxel traceable back to its original file. The architecture was not redesigned: the Adaptive Voxel World remains the single source of truth, and the SVO stays a derived, incrementally-maintained index.

Goal 1 — SVO Wired Through the Query Paths

The vwm.worldos.World facade routes spatial queries to the SVO and semantics to Components, so callers never pick an index. docs/SVO_INTEGRATION_MATRIX.md audits every subsystem (VoxelWorld / SVO / both / neither). Measured on the hallway (73 768 voxels): raycast 7 076 q/s vs 1 654 linear — 4.3× faster from empty-node skipping, a win that carries to picking, floor/wall probes, visibility, and line-of-sight. Stated honestly: SVO nearest_occupied is currently slower than a vectorized numpy scan at this scale (83 vs 564 q/s) — a Morton neighbour-search optimization is future work, documented rather than hidden. Edits stay on the authoritative world; the incremental dirty-subtree flush keeps the index in sync.

Goal 2 — Persistent Components

A Component owns metadata and references voxels — never geometry — and is distinct from the pre-existing simulation WorldComponent. Full field set (uuid, type, name, parent/children, voxel+SVO refs, material, appearance, confidence, tags, relationships, observation history, bounds), full editing (create · rename · set-type · merge without duplication · split · reassign · delete), a relationship graph (contains / supports / mounted_on / adjacent / …), and JSON round-trip. Components survive rescans: geometry changes update the referenced voxels while UUID identity persists — tested, not assumed.

Goal 3 — Universal Upload Pipeline

Upload → Detect (extension + magic bytes) → Route → Observations → VoxelWorld. Working end-to-end for glTF/GLB, OBJ, PLY meshes, PLY/XYZ point clouds, Polycam zip, VWM voxel npz, and image heightmaps. LAS/LAZ, video, USD/USDZ, FBX, Gaussian splats, and COLMAP are routed with specific honest errors — they need their runtimes or a heavy offline job, and they say so rather than faking success. Every upload creates a Capture + world revision + incremental SVO update, driven by an async job queue with progress, ETA, and retry.

Query Language, World API, Revisions

• Query language: wall, type:door, material:wood, confidence<0.8, capture:polycam, room:kitchen, uuid:…
• World API: find_components, raycast, nearest_surface, find_by_material, find_room, find_observations, find_all_doors/rooms, find_uncertain, largest_room, components_touching, find_disconnected_regions, trace, ingest.
• Revisions: a delta-based timeline with diff, undo, revert, and replay.

Honest Scope Notes

• Cross-capture registration (ICP-aligning a new upload to the existing world) is not implemented — new captures bin in the world frame and may overlap; the single-asset and voxel-npz paths are exact.
• Six formats are routed but need their runtimes (laspy, OpenUSD, FBX SDK) or a heavy offline job (video); they report specific errors, never fake results.
• The component editor, capture-history browser, and revision browser have working endpoints but minimal front-end surfaces in this build — create/rename/merge/query all work via the API.
• SVO nearest_occupied loses to vectorized numpy at small scale; optimization is future work.

Sparse Voxel Octree Integration, Audited Not Copied

VWM now has a native Sparse Voxel Octree subsystem — a derived acceleration structure for collision, ray queries, and hierarchical spatial reasoning, built directly from the authoritative VoxelWorld. The MIT-licensed otaku690/SparseVoxelOctree is vendored unmodified in third_party/ with its license intact, and docs/SVO_UPSTREAM_AUDIT.md classifies every file as reused, adapted, rewritten, or rejected. The useful algorithms were adapted — the compact flag+pointer node word, dominant-axis conservative rasterization, level-by-level flag/allocate construction, bottom-up attribute mipmap — re-implemented deterministically on the CPU. The legacy OpenGL/VCT architecture was rejected outright: no upstream C++/GLSL is compiled into VWM. The authoritative VoxelWorld is unchanged; the SVO is derived and rebuildable.

The Native Subsystem (vwm/python/vwm/svo/)

Nine modules: svo_types, svo_builder (builds directly from authoritative voxels via Morton order — no mesh round-trip — plus a conservative mesh path), svo_traversal (ray-march with empty-node skipping), svo_collision (point / sphere / AABB / capsule / swept / ray / segment / nearest / floor / wall), svo_queries, svo_incremental (dirty-subtree splice), svo_serializer (CRC-checked, versioned .vwm-svo binary), svo_validation, and svo_gpu (WGSL). Flat structure-of-arrays buffers serve CPU traversal, WebGPU upload, and serialization alike; the CPU is the deterministic reference.

Verified, Not Assumed

The 19 tests in tools/test_vwm_svo.py cover empty / single / full-root / sparse / disconnected / thin-wall / doorway / window / nested / negative-coords / large-bounds / max-depth / round-trip / determinism / CPU-traversal agreement / incremental==full / no-crossable-openings / no-cracks. Serialization round-trips deterministically, hollow interiors stay empty, and the hallway artifact was verified on load.

Honest Benchmark (docs/SVO_BENCHMARK.md)

Stated plainly where the SVO loses: on a small dense surface shell, a 1-byte dense grid is more compact in RAM than the octree. The SVO's wins are sparse-world scaling, empty-space skipping, hierarchical aggregates, and fast incremental maintenance — and the memory advantage grows with sparsity.

Viewer: SVO Representation, 9 Modes

Modes: Off · Leaves · Interior · Depth · Occupancy · Collision · Confidence · Dirty · Traversal-cost, plus a WebGPU ray-march overlay. Click any node → index, depth, bounds, voxel size, child mask, leaf/interior flags, occupancy, confidence, source/svo revision, material and component refs, dirty flag. All prior representations (literal voxels, mesh, feature voxels, error-driven voxels, appearance layer) are retained.

Appearance Separation Carried Forward

The SVO stores geometric occupancy plus a leaf_appearance_ref into v9's independent appearance layer — never pixels. Big flat voxels keep coarse collision geometry while paintings and posters stay visible through the appearance layer. Collision reads geometry, not visual detail.

WebGPU Status (Honest)

The WGSL per-pixel traversal (/api/svo/wgsl) and GPU buffers (/api/svo/gpu) are served, and the viewer's overlay uploads and ray-marches them when navigator.gpu exists. Under the headless test harness (ANGLE/SwiftShader, no WebGPU) it falls back to a documented blocker message — the WebGL node-box visualization and the CPU reference traversal are fully working and authoritative. In a WebGPU-capable Chrome/Edge session the overlay ray-marches the SVO per pixel; that is the one path not fully executable in this environment.

Not Done / Future

• WebGPU compute-shader construction — concept documented; the CPU build is the reference.
• Brick leaves (brick_dim > 1) are scaffolded in the format, but leaves are single-voxel today.
• Background defragmentation of orphaned nodes after many incremental edits — currently reclaimed by the next full rebuild.

V9, Error-Driven Adaptive Voxelization

The philosophy flipped. v8 asked "is there detail?" — a heuristic. v9 asks the only mathematically grounded question: "if this region is one voxel instead of smaller voxels, what measurable error is introduced?" A region stays the largest possible voxel iff a single plane-snapped voxel approximates the measured mesh within a configurable error budget. Subdivision exists only to reduce measurable error, and every voxel stores its error terms and the reason it is the size it is. The new vwm.error_voxelize implements independent error terms — geometry (the master term: robust p95 residual to best-fit plane), normal (p90 angular deviation), curvature (p90 dihedral), topology (never bridge doorways/gaps), thin (trim, frames, handles), silhouette (footprint slivers), and appearance — which is metadata only and never triggers subdivision.

Geometry and Appearance, Fully Independent

Each large voxel stores an 8×8 texture patch — its samples projected onto the voxel's plane in plane-local UV — packed into one atlas, drawn in one draw call. A 24 cm flat wall voxel over a painting keeps its size and displays the painting. This deliberately reverses v8.1, where paintings spawned thousands of tiny voxels: appearance no longer drives geometry. The viewer proves it with a 4-way comparison — Geometry only · Avg color · Appearance layer · Polycam mesh — same geometry, dramatically different visual fidelity. The headless capture confirmed the appearance layer renders real floor and wall colours on big error-bounded voxels.

v8 vs v9, Measured (benchmark_v8_v9.json)

Error-Budget Profiles: Change Tolerances, Not the Algorithm

Six profiles ship (museum · architectural · simulation · robotics · gaming · preview), each just a set of tolerances in error_budget_profiles.json. Measured on this scan:

Two Hard-Won Engineering Lessons (Both Real Bugs, Fixed Honestly)

• Robust percentiles, not max. Using the absolute-max residual let a single LiDAR speckle force full subdivision — 67k voxels. Switching to p95 residual / p90 normals fixed it.
• Budgets must be noise-aware. The mesh's intrinsic noise was measured (24 cm cells: 3.3 mm RMS on flat walls, 9.6 mm at edges) and budgets calibrated to it. Below ~4 mm, fine voxels are forced everywhere — which is correct behaviour, documented rather than hidden: the sensor noise sets the floor.

Explainability: Every Voxel Justifies Itself

Each voxel stores its size, all error values, approximation error, snap status, subdivision-cause bits, and its appearance record. Click any v9 voxel in the viewer → why_this_size ("satisfies the budget" or "parent subdivided due to: geometry, topology"), plus nine error-heatmap modes: geometry error, plane residual, normal variance, curvature, topology risk, thin risk, silhouette error, approximation error, subdivision reason.

Honest Limits

• Sensor noise sets the floor — budgets below ~4 mm force near-uniform fine voxels on this LiDAR mesh, so "museum" here is dense.
• Silhouette is a footprint-fill proxy, not a full multi-view raster; hysteresis is implemented as a budget concept but only matters for incremental re-voxelization (this is a one-shot static scan).
• Sampling dependence: denser mesh sampling reveals more true geometry, so voxel count grows with samples — the error bound holds regardless.
• LOD balancing is measured and reported (0.94–0.97), not forcibly enforced; splitting is always error-safe, so a hard 2:1 tree could be enforced if required.
• The corner solver measures and recovers corners as metadata but does not yet re-mesh sharp edges off the octree lattice — the same open item since v8.

V8, Feature-Preserving Adaptive Voxelization

The voxel world stops looking like a uniform blocky grid and becomes an intentional voxel build of the real hallway: big 24 cm blocks blanketing flat walls, fine 3 cm voxels concentrated only at edges, corners, door frames and trim. A top-down octree voxelizer (vwm.feature_voxelize) is driven by measured per-sample curvature (mesh dihedral angle) and per-cell colour variance — a cell stays coarse when its bulk curvature is low and it is single-material, and subdivides only where curvature, edges, or trim live. Colour is material-aware (per-cell median, no bleeding across surfaces). Evidence-based throughout: every voxel is anchored to a measured surface sample; nothing is invented.

Architectural Fidelity, All Measured

The Critical Tuning Insight

On a noisy phone-LiDAR mesh, dihedral-curvature flatness needed curv_flat ≈ 0.35: at 0.10, 93% of the surface stayed at the finest level (walls never coarsened); at 0.55, the detail itself was over-coarsened away. The threshold is tuned empirically to this mesh's noise, not derived — reported as such.

Viewer: The Slider Request, Delivered

• Representation slider (Surface → Literal Voxels → Feature Voxels), synced with the buttons, plus a new /api/features endpoint that overlays the variable-size voxels on the Polycam mesh exactly.
• Voxel-size slider that live re-bins the literal voxels into 5→40 cm blocks in-browser (mean-color aggregated), with click-inspect working on the binned blocks.
• Three new render modes: adaptive resolution, feature density, and voxel size.

Honest Limits

• The corner solver measures 90° recovery and confirms fine voxels sit at the seams — it does not yet snap voxel faces to reconstruct an idealized sharp edge; voxels remain on the octree lattice. Plane snapping / face smoothing is open rendering-normal work.
• Metrics are top-down/plane-based on an axis-ish room; a fully general scene would need RANSAC plane extraction rather than axis-aligned normal clustering.

The Polycam Scan, Loaded Directly Into the VWM Viewer

Follow-up to the v7 fusion milestone: the actual Polycam capture is now served standalone by the same authoritative-world server and explored live in the Three.js viewer — the recognizable room itself, correctly metric-scaled (12.3 m hallway), not the noisy monocular reconstruction.

What This Is (And Isn't)

• Same engineering client, same architecture. The authoritative-world server (server.py --npz polycam_world/artifacts/occupancy_world.npz) holds the scan; the browser owns no world state. All the v4/v5 machinery — representation toggle, render modes, orbit/fly/walk, inspector, edit with local recompile — works on the Polycam world unchanged.
• Both representations from one field: the extracted surface (the Polycam mesh) and the 73 768-voxel literal view are two derived views of the same imported scan, exactly the v5 representation-≠-storage contract.
• Surface-only, honestly: this direct import has FREE 0 — no free-space carving has been run on it, so Walk mode has nothing to stand on yet and the occupancy view is all-SURFACE. Wiring free-space + a standable spawn for the Polycam world (so you can actually walk the scanned hallway) is the named next step.

Reality-Capture v7, Polycam Multi-Source Fusion

A LiDAR-metric Polycam scan of the same hallway is integrated as another observation source, not a replacement — monocular video and Polycam now converge into the one persistent authoritative VoxelWorld. (The target is the v6 world: an internal incremental-merge pass that gave the world per-voxel observation memory and merged the walk's two keyframe sets as separate scans, confidence 4.1 → 6.84, byte-identical restart reload.)

Polycam Import (Discovered, Not Assumed)

7_16_2026.zip → single 7_16_2026.glb, auto-inspected: 30 974 vertices / 49 475 faces, metric (12.28 × 5.28 × 2.72 m), PBR material with an 8192×8192 base-color texture. GLTF Y-up flipped to VWM Y-down, texture baked to vertex colors, sampled to 120 000 surface observations (position + normal + color). The importer (vwm.polycam_import) is permanent infrastructure — future Polycam scans need no manual work.

Fusion (Accumulate, Never Overwrite)

Polycam samples are voxelized into the world grid and merged as observations, with the highest source quality (1.5) so its 8k-texture color wins where co-observed:

Partial / Not Done (Honest)

• Metric-scale correction is computed and reported but not yet applied — re-voxelizing the world at true metric changes the grid, a deliberate, reversible next step.
• Re-extracting the surface at metric density (Polycam's 49k faces would give flatter walls and sharper corners), semantic re-estimation on the fused geometry, and the live Luanti walkthrough (luanti_validation.json exports to the proven HAV importer, not run live) — next steps, not faked.

Reality-Capture v5, Extracted-Surface Reconstruction

The central move, exactly as VWM's architecture frames it: separate representation from visualization. The VoxelWorld/HAV is unchanged — it stores the TSDF field. The viewer now renders the continuous zero-crossing surface extracted from that field instead of storage cubes, so the hallway looks architectural instead of like a colored voxel blob. Every surface vertex maps back to authoritative voxels, and extraction runs only where the field was observed — no surface over unseen space.

What v5 Does (Evidence-Based)

• Surface extraction (marching cubes) on the TSDF field → a colored triangle mesh with vertex colors sampled from the observed appearance field. 92% of edges are manifold. surface_extraction_benchmark.json records marching cubes against the evaluated alternatives (surface nets, dual contouring for sharp corners).
• Representation ≠ storage: the authoritative voxels are unchanged; the mesh is a derived view. The viewer toggles Extracted Surface ↔ Literal Voxels instantly, both from the same field.
• Conservative field gap-closing: only deeply-enclosed unobserved voxels are bridged (4 gaps), never open space.

Viewer

Default representation is now Extracted Surface; one click returns to Literal Voxels. All 11 render modes are retained (occupancy, adaptive, LOD, inferred fill, …) along with orbit / fly / walk with gravity and collision. Voxels still render at 63 FPS; the mesh runs interactively (24 FPS in headless software rendering — real GPUs are far faster). New endpoint: /api/surface.

Honest Fidelity Assessment

v5 makes the reconstruction look like a surface rather than a voxel blob — the biggest perceptual step toward recognizability — and it is a pure representation change over the same authoritative field, with zero geometry invented. But the underlying geometry is still limited by the single monocular forward-walk: walls are smooth-but-wavy, corners rounded, and fine features (handles, switches, trim) absent because the capture never densely observed them. The representation is now right; the remaining fidelity ceiling is set by the input data, exactly as reported since v2.

Partial / Not Done (And Why)

• Plane-snapped flat walls and a corner solver — the biggest remaining geometry win. Marching cubes rounds corners; truly flat walls and sharp corners need plane-snapping plus dual contouring (QEF on hermite data). The detected planes exist but are not yet applied to vertices, and the plane-residual fidelity sub-metric is disabled (frame misalignment).
• Thin-structure recovery and adaptive microvoxel re-integration (1 cm / 5 mm), plus live Luanti re-import: each needs either denser observation or a next algorithmic step. Not done, because inventing them would be hallucination.

Reality-Capture v4, Recognizable Adaptive Reconstruction

Operating on the same authoritative v3 world, v4 makes the reconstruction recognizable without inventing geometry: continuous surfaces plus adaptive detail levels. Every added voxel is provenance-tagged INFERRED; free space and open unknown space are untouched.

v3 → v4, Measured

What Changed, Conservatively

• Hole-filling → continuous surfaces. An UNKNOWN voxel enclosed by observed matter on ≥4 of 6 faces is a reconstruction gap in a real surface, not open space. 427 such holes were closed iteratively, each tagged INFERRED_hole_fill, with appearance interpolated from neighbouring observed colour (blended, not smeared). Free air is never filled.
• Adaptive detail levels. Every matter voxel is classified from local surface exposure: coarse (buried flat, 9 067), medium (surfaces, 1 893), fine (edges/corners/trim, 402). The viewer renders coarse voxels larger and fine voxels smaller.

Viewer Upgraded

The engineering client gained Walk mode — gravity, collision, and identity height, standing on the reconstructed floor and walking the hallway with WASD, blocked by matter with no clipping — plus three new modes: Adaptive Resolution, LOD Heatmap, and Inferred Fill. Still 63 FPS at 16 212 instanced voxels, single draw call, mode switches with no rebuild. It still owns no world state: it loads the authoritative v4 occupancy via the API, and edits POST back with local 3×3×3 recompile.

Honest Recognizability Assessment

v4 is materially more recognizable than v3 — the hallway now reads as a continuous hollow corridor with wood/stone colouring and a clear floor/wall/ceiling shell rather than a sparse speckled shell. But it is not photo-recognizable at fine scale: the base voxel is 5 cm and the single forward-walk capture never densely observed small features (handles, switches, trim), so those cannot be resolved without either a richer capture or hallucination — and we do not hallucinate. The reconstruction is a faithful, continuous, adaptive voxelized version of exactly what the camera saw.

Partial / Not Done (Honest)

• True sub-base microvoxels (1 cm / 5 mm) are not implemented. Genuine door-handle/keyhole resolution needs capture that observed those features densely, plus region-flagged higher-res TSDF re-integration (documented in microvoxel_report.json).
• Live Luanti re-import walkthrough, appearance-projection-v2 multi-view reweighting, and a progressive-refinement scheduler are next steps.

QA Loop + Official Three.js VWM Viewer

Two deliverables on top of the v3 hollow world: an automated QA loop that found and fixed real import/spawn defects with before/after measurements, and the official browser-based engineering client for the authoritative VoxelWorld — a Three.js viewer/editor that owns no world state of its own.

QA Loop: Real Defects, Measured Fixes

Three autonomous QA iterations over the canonical v3 world, logged in v3/qa/ENGINEERING_LOG.md:

Honest note: broad floor coverage is a capture gap (fixable only by looking down while scanning), not invented by the fix. Remaining defects are capture-data limitations, not code bugs.

The Viewer: One Source of Truth

A Python server (tools/vwm_viewer/server.py) holds the authoritative v3 VoxelWorld and serves and edits it via /api/world, /api/meta, /api/stats, /api/edit. The Three.js client owns no world state — every voxel, class, colour, and metadata comes from the API, and edits POST back to the server, which mutates the authoritative grid. Three.js r128 is vendored locally, no external CDN. Camera movement triggers zero server or reconstruction calls — the zero-regeneration invariant, again, at the viewer level.

Acceptance Criteria (13 / 15)

• ✅ Backend launches, viewer auto-opens in the browser, and loads the authoritative world directly.
• ✅ Displays the hollow hallway; FREE/OCCUPIED/UNKNOWN occupancy view (blue/orange/yellow), plus real-color, hollow (matter only), free-space, confidence heatmap, observation count, semantic, and wireframe — 9 modes with instant switching (in-place instance recolour, no geometry rebuild).
• ✅ Orbit + fly camera (WASD / mouse-look), spawn-in-FREE-space.
• ✅ Click any voxel → inspector with authoritative metadata: class, provenance, confidence, neighbours, metric position, revision.
• ✅ Edit a voxel → the authoritative world changes: revision bumps, class counts change, with local 3×3×3 recompile scope and global_rebuild=false — the local-recompile invariant proven on edit.
• ✅ 63 FPS measured rendering 15 613 instanced voxels (single draw call).
• ✅ Screenshots of all 8 visualization modes captured (v3/viewer_shots/).
• ⚠️ Live Luanti edit-reflection: partial — edits hit the shared authoritative world Luanti reads, but Luanti was not connected this session.

Honest Partial / Not Done

The spec is enormous; these are laid out as extension points in VIEWER_REPORT.md: the full edit suite (component move/rotate/undo/merge), ~21 more debug overlays (TSDF slices, provenance, LOD, trajectory, navmesh), hierarchy panel, search, time controls, measurement tools, MP4 recording, WebXR, and WebSocket live-push (currently a 1 s poll; edits update instantly).

Reality-Capture v3, Hollow Interior + Live World Import

Same canonical phone video, same authoritative pipeline order, now extended with an explicit occupancy stage (FREE / SURFACE / OCCUPIED / UNKNOWN), a free-space connectivity graph, and a working import into the live authoritative VoxelWorld. The central rule of this milestone — a room is free space bounded by voxelized matter — is achieved and proven, not asserted.

Proven (Measured)

• Occupancy classification of the TSDF volume: SURFACE 8 047 + OCCUPIED(inferred) 2 716 = matter, thin shells rather than solid blocks; FREE 4 850 carved air; UNKNOWN 29 387 unobserved and never filled solid.
• Hollow-space validation (hollow_space_validation.json): camera-centreline FREE fraction 1.0, flood-fill reaches 99.2% of free space as one connected region, 0 floating solid voxels.
• Bounded, provenance-tagged wall thickness: 2 716 voxels extruded ≤2 voxels behind surfaces, only into UNKNOWN, never into FREE, and labelled inferred rather than observed. No filled hallways, no giant slabs.
• Free-space connectivity graph (free_space_graph.json): one navigable component of 4 810 voxels — the walkable hallway — with a clearance (distance-to-surface) field computed.
• Live authoritative VoxelWorld import is operational. The previous HAV stall is fixed: 10 763 SURFACE+OCCUPIED voxels → 15 177 HAV leaves in 9.9 s (1 090 fills/s). The fix was structural, not a tweak: TSDF yields a compact shell over a ~31³ span, versus v1's 137k splats over a 1 000-unit span. The world is native VoxelWorld/HAV state now, not just an NPZ.
• Free-space spawn: a spawn voxel in confirmed FREE space is exported, so the identity materializes in air, not inside a wall.

v1 → v2 → v3

Partial / Not Done (Honest)

• Live Luanti client walkthrough — not run. The free-space spawn, flood-fill proof, and walkable free-space graph establish that an identity would spawn in air and traverse connected free space, but the interactive in-client walk with framebuffer capture needs a human session. The world is imported and ready.
• Restart persistence / export-import round-trip — not run this pass.
• Held-out real-view appearance validation, exposure normalisation, view-dependent appearance — not this pass.
• Explicit portals and multi-room segmentation: the free-space graph is built and is a single hallway region here; portal extraction and multi-room splitting are not emitted.
• Absolute metric scale stays best-effort (~2 m scene) from the compressed monocular cloud; relative geometry and hollowness are solid, absolute scale approximate.

Reality-Capture v2, TSDF Volumetric Fusion

Quality-improvement pass on the same canonical phone video as the walkthrough milestone below. Point-splat fusion is replaced with true TSDF volumetric fusion, plus free-space carving, plane-based semantics, and a 42″-door metric anchor. Deterministic run, same authoritative pipeline order: Reality → Capture → Depth → Trajectory → Global Alignment → TSDF Fusion → HAV → Semantics → Components → Appearance → Viz → Export.

v1 → v2, Measured

What Genuinely Improved

• TSDF fusion works. Truncated-signed-distance averaging with weight accumulation over ~15 views per voxel merged 137k noisy splats into 9.6k voxels that reproject 28% more accurately. Duplicate-shell merging and floater removal fall out of the weighting; the 14× voxel reduction at better accuracy is the direct evidence.
• Free-space carving is on. Every voxel is classified OCCUPIED / FREE / UNKNOWN with confidence and provenance: 4 416 carved free, 24 283 correctly left UNKNOWN, never filled solid.
• Watertightness 0.99. 99% of surface voxels are backed by ≥2 occupied neighbours, coherent surfaces rather than floating splats.
• Plane-based semantics: sequential RANSAC found 8 dominant planes labelled wall / floor / ceiling / surface by geometry, each with inlier-count confidence (reported, not forced), plus wall thickness from parallel plane pairs.
• Metric anchoring: the 42″ (1.0668 m) door is measured via an outlier-robust percentile width of a wood-coloured vertical plane (confidence 0.98) and applied as a global scale, guarded by a scene-size sanity gate (2–40 m) that rejects absurd scales.

What Remains Uncertain (Honest)

• Absolute metric scale is only best-effort. The monocular SfM + depth point cloud is geometrically compressed (weak baselines relative to depth), so the door-derived scale (~2 m scene) is almost certainly still too small for a real hallway. The door detector passes its sanity gate with high inlier confidence, but the underlying cloud does not spread into correct hallway proportions. Relative geometry and every other improvement are solid; treat the absolute scale as approximate.
• Render coverage dropped to 0.49 (from 0.99), the direct cost of keeping 9.6k confident voxels instead of 137k noisy splats. The surface is truer but covers less screen when point-rendered; reprojection L1 is still better.
• No loop closure: the capture is a one-way forward walk with no revisit, so there is no loop to close (detected and reported, not invented).
• Scale-consistency CoV 0.576 unchanged: the alignment graph is geometric, not learned; it reduces but does not eliminate per-frame monocular scale drift.

Biggest Next Gains (Data, Not Code)

1. A short back-and-forth / orbit pass over the same hallway → real baselines and multi-view overlap → the cloud spreads to true proportions, fixing both the absolute scale and the coverage.
2. One in-frame metric measurement (ARKit depth on even one frame, or a ruler) → locks scale independent of fragile door detection.
3. ARKit/ARCore device poses → removes monocular-SfM scale drift entirely (CoV → ~0).

Real-Video Reconstruction, Phone Walkthrough

Full pipeline run on a real phone walkthrough of a home (laundry area to hallway): 464×848, 19.4 s, 582 frames. No synthetic data anywhere in the run. The forward-walking phone capture provides the real baselines the webcam test lacked.

Pipeline (All Real)

video → 155 frames @ 8 fps → 24 keyframes → visual SfM (OpenCV SIFT + essential + incremental PnP) → Depth Anything per keyframe → global depth-alignment graph (SIFT correspondences + two-view triangulation → robust per-frame scale/shift) → voxel fusion (occupancy + confidence + provenance) → semantic component proposals → persistent voxel-world data + observed-appearance atlas → all visualisations + reprojection metrics.

Experimental: AI Photoreal Upscale from Voxel Data

As a qualitative experiment, the voxel reconstruction was passed through a generative image model to upscale it back toward a photoreal frame. The result looks convincing, but it is not a measurement and should be read with care: wherever the scan had thin evidence, the model is guessing.

Measured Results

What Succeeded

• This capture works where the webcam didn't. Forward walking gives real baselines: SfM tracked all 24 frames (confidence 0.897) and the depth-alignment graph brought scale drift from CoV ~30 down to 0.576.
• Three real bugs found and fixed along the way: a gauge-normalisation bug that drove aligned depth negative; a NumPy 2.x .ptp() break; and a slow per-voxel render loop (vectorised).
• depth-anything on real imagery is clean (confirmed again): sharp depth maps, hallway receding correctly.
• End-to-end reconstruction with real appearance: colored point cloud, voxel reconstruction, novel views, and a same-pose reprojection that matches the originals (trash can, stone tile floor, wood door, hallway).
• Persistent world materialized: reconstruction written into HAV with 38 284 leaves; full data reopenable via npz.
• All artifacts exported: artifacts/ (keyframes, aligned_depth.npz, fused_cloud.ply, voxel_world.npz, trajectory.jsonl), viz/ (depth, trajectory, point cloud ×3, voxels ×3, components, side-by-side, contact sheet), reconstruction_report.json, RECONSTRUCTION_REPORT.md.

What Failed or Is Weak (Honest)

• Mostly single-view voxels (137 374 provisional vs 311 confirmed, mean 1.0 views/voxel). Forward walking down a hallway means each surface is glimpsed briefly, so little multi-view confirmation: the geometry is thin fan-shells rather than thick confirmed surfaces.
• Scale CoV 0.576 is reduced but still non-trivial. Relative monocular depth + monocular SfM scale still drift; a metric anchor (known door width) or AR poses would tighten it.
• HAV materialization hit a real performance cliff with the dense / large-span voxel set (fills stall across the large SfM span). VoxelWorld.create also deadlocked against OpenCV / depth-anything under detached stdin. Worked around with a separate foreground process (tools/vwm_build_world.py), but the full-density live world is not materialized in-process — it is reopenable from voxel_grid.npz.
• No free-space carving in this run (disabled for speed), so no explicit occupied / free / unknown separation this pass.
• Component labels are weak heuristics (extent-ratio → wall/floor/object at confidence 0.4), not a real semantic segmenter.

Not Done

Live Luanti open of the scanned world; full appearance-compiler compilation (the observed-color atlas is the appearance here); loop closure; exposure normalisation; incremental rescan; held-out real-view LPIPS benchmark. Per prior instruction, v2.4 was not trained.

Next Improvements (Highest Leverage First)

1. Capture motion: a slow orbit / back-and-forth (not just forward) → multi-view confirmation → thick confirmed surfaces, higher IoU.
2. A metric anchor (known door width) or AR poses → resolve residual scale drift → metric world.
3. TSDF-style fusion + re-enabled carving → watertight surfaces and proper free / unknown space. Shipped: see Reality-Capture v2 above.

Full write-up: checkpoints/vwm_scan/real2/full/RECONSTRUCTION_REPORT.md. Source: video_2026-07-16_13-09-43.mp4.

See benchmarks for phone vs webcam comparison tables.

Reality Capture, Real Webcam Bedroom Scan

First end-to-end run on genuine real-world data: 18 s / 425 frames from a Logitech BRIO webcam of a real bedroom (door, closet, dresser, couch, accent wall). Hand-held motion, no phone available. This is a partial delivery of a 36-section milestone; the capture→depth→align→fuse→attach→novel-view core shipped. v2.4 was not trained.

What Genuinely Works (on Real Data)

• Real capture. BRIO webcam bedroom video, not a synthetic render.
• depth-anything on real photos is clean. Single-view backprojection produces a recognisable colored voxel cloud with correct room geometry.
• Real feature geometry. 540 SIFT matches / 516 essential-matrix inliers between real frames (vs zero features on synthetic voxel renders).
• Full pipeline end to end: frames → keyframe selection → visual SfM (VisualSLAMProvider) → depth-anything → DepthAlignmentGraph (56 overlap edges, 16 626 triangulated samples) → voxel fusion → attached real RGB → novel view.
• New modules import clean: webcam ingestion, vwm.camera_trajectory (Known / AR / VisualSLAM providers), vwm.depth_alignment (global scale+shift graph, triangulation, robust IRLS, metric-anchor support). OpenCV 5 installed.

The Honest Limitation

Multi-view fusion is geometrically rough (solved_scale_cov ≈ 30). That is not a pipeline failure — it is the capture: a hand-held webcam moved a few inches gives tiny baselines, so monocular SfM and monocular depth compound into unstable per-frame scale. The single-view result is clean precisely because it sidesteps multi-view scale coupling.

Fixes are already named in the progression; none require redesign:

• Walk around the room (bigger baselines) — the intended phone workflow.
• AR poses (ARKit/ARCore) — metric and stable; ARPoseProvider seam is already there.
• A metric anchor (known door width / LiDAR).

Not Done (Honest)

Loop closure, exposure normalization, view-dependent appearance, held-out real-view benchmark, incremental rescan, dynamic rejection, semantic components, opening the scanned room in the live Luanti client, and more from the 36-section milestone remain open. Phone-native capture + AR pose (webcam substitute only). Per explicit instruction, v2.4 was not trained.

Full write-up: checkpoints/vwm_scan/REALITY_CAPTURE_STATUS.md. Artifacts: real_capture/room_scan.mp4, real/ (SfM + fusion + novel view), single-view voxelization.

See benchmarks for measured real-capture numbers vs synthetic baseline.

Depth Source Comparison, 3-Way Controlled Benchmark

Same controlled VWM room (20-frame trajectory), same known camera poses, same fusion/voxelize/validation pipeline. Only the depth source changes. All monocular runs use one robust pooled calibration (single scale anchor across all frames).

3-Way Comparison

See benchmarks for trend indicators vs sensor ground truth.

Three Honest Findings

1. A methodology bug got caught. First metric run anchored scale on frame 0 (a scale outlier: 8.1 vs typical ~3.9), giving recall 0.39 and error 6.76. Robust pooled calibration across all frames cut error >2.5×. Lesson baked in: never anchor scale on one frame.
2. On synthetic input, DA3 relative slightly beats DA2 metric (recall 0.564 vs 0.449). Metric depth's absolute-scale advantage does not materialize because flat-shaded voxel renders are out of distribution for models trained on real indoor photos. Per-frame correlation is decent (0.76–0.96), but per-frame scale drifts 47–57% (CoV), which caps recall.
3. Neither monocular model reaches sensor recall (~0.45–0.56 vs 0.918). The limiter is cross-frame scale inconsistency plus the synthetic domain gap, not the fusion pipeline. Next fair test: real phone frames + ARKit/ARCore poses where metric models see in-distribution imagery.

Answer: Does Monocular Metric Get Close Without a Sensor?

Not on this synthetic controlled test — monocular tops out around 0.5 recall. But the test is genuinely unfair to monocular models: they are trained on real photos and the input is non-photoreal voxel imagery. The correlations (0.76–0.96) say depth shape is largely right; cross-frame scale is what drifts.

Full write-up: checkpoints/vwm_scan/DEPTH_SOURCE_COMPARISON.md. Artifacts in sensor/, ctrl/ (DA3), metric/ (DA2).

VWM Appearance v2.3, Scale, Rare Geometry, and the 100k Memory Fix

Thesis: the bottleneck is experience (only 1,050 observations). Built the data engine, trained v2.3, and stress-tested the renderer. v2.2 remains production default.

The Bug the 100k Proof Caught

• Root cause: _repair promoted every repaired frame to a keyframe (~1 MB each) with no eviction, while _build correctly capped at max_keyframes=24. Leak rate ~57 KB/frame.
• Symptom: first run RSS climbed 4.6 → 9.7 GB and would have OOM'd before finishing (no swap, 6 GB free).
• Fix: shared _insert_keyframe() helper with while-loop eviction; both paths route through it.
• Validation: 8,000-frame regression → keyframes flat at 24, RSS growth 1.4 MB. Then clean 100k re-run held RSS at 3056 MB for 52 minutes (vs >9.7 GB by 45 min pre-fix).

The Data Engine (what worked)

• Coverage audit measured real gaps: geometry 28/29, materials 21/50, HAV 5/6 UNSEEN.
• 25 rare thin-geometry generators + 17 new material families + compositional material×object de-correlation.
• Active planner (coverage + model-disagreement + error mining) avoids 90% of Cycles work.
• Persistent Cycles factory: 1.79 obs/s, 2.24× speedup, crash-safe/resumable. Dataset 1,050 → 2,757 observations.
• v2.3 improves rare geometry: thin_survival 0.853→0.965, edge-IoU 0.507→0.626, rare gate failures 0.05→0.

Why v2.3 Was Not Promoted

On frozen standard held-out, photographic L1 regressed +45% (0.0178→0.0259; scratch variant +132%). Training pool was 74% rare geometry. VWMScore v3 edged higher (0.7645 vs 0.7596) purely on the structure term; scratch model over-sharpened (edge-IoU 0.872). Structure-for-photography lateral trade, not a clean win.

Standard Held-Out Benchmark (VWMScore v3)

Rare Geometry Held-Out (v2.3 wins here)

See benchmarks for trend indicators.

Honest Status

• PROVEN: data engine, rare-geometry capability, edit/relight proofs, memory leak fix.
• PARTIAL: dataset at 2,757 of 10k (resumable factory); 100k stopped once memory validation achieved.
• NOT DONE: live Luanti client (needs interactive session; v2.3 not promoted anyway).

Artifacts in checkpoints/vwm_appearance_v2_3/ including MILESTONE_STATUS.md, memory_fix_validation.json, and vwmscore_v3_frozen.json.

VWM Appearance Model v2.2, Structural Fidelity Without Sacrificing Photographic Appearance

Promoted to production default. checkpoints/vwm_appearance_v2_2/vwm_appearance_v22.pt (version vwm-structural-unet-2.2), auto-resolved at runtime and integrated into combined.py.

Forensic Root Cause (measured, not assumed)

Controlled ablation (identical init/seed/split/steps) showed that neither edit-locality nor light-invariance alone softens edges, only their combination does. Gradient-cosine analysis on the shared decoder trunk: intrinsic ~ light-invariance = +0.89 (nearly collinear). Both objectives pull the trunk toward the flat, lighting-free albedo manifold; v2's final photographic head is a thin residual on that trunk, so activating both roughly doubles the flatten pull and the residual head inherits a smoother base. Amplification imbalance, not gradient cancellation.

The Fix (architecture, from the diagnosis)

• Decoupled intrinsic decoder so intrinsic/light-invariance cannot flatten the final head's features.
• Structure-guided final decoder fed an authoritative G-buffer boundary map (depth/normal/component discontinuities, never Cycles RGB).
• Structural losses: anti-blur band (edges only), cross-boundary bleed loss, structure-aware locality that never flattens boundary pixels.

Final Benchmark (VWMScore v3)

See benchmarks for trend indicators vs prior versions.

Structural Torture House

Thin chair legs, keyboard keys, vent slats, and similar geometry: thin_survival 0.841 vs v2's 0.543, without hallucinating edges (boundary precision held at 0.95). Candidate ladder of eight configs; promoted checkpoint = gentle anti-blur-weighted candidate G (3500 steps), not the longest run.

Invariant Proof

10,000 camera-only frames: neural_model_invocations = 0, compiler_invocations = 0, 48.6 FPS median. The compiler invariant holds.

Honest Limitations

• Dataset scale: still 1050 observations (not 5k/10k). Cycles ~0.8 obs/s on RTX 2060 SUPER; gains are architectural, measured on the same held-out split.
• In-Luanti capture: not re-run; Python integration wired, invariant proof stands in for runtime behavior.
• 85 modules import clean; full forensic package in checkpoints/vwm_appearance_v2_2/.

Production default; evaluated against v2.3 (not promoted). Memory leak fix from v2.3 applies to the fused renderer used by v2.2. See benchmarks.

VWM Appearance Model v2.1, Executed, Measured, Honestly Judged

Four named weaknesses attacked with real data, real training, and real measurement. Two resolved as measured capabilities; two partial with honest reasons. v2 remains production default.

Resolved: Edit-Locality Supervision (#1 weakness)

Built authoritative edit-pair generation (tools/vwm_dataset_v21.py): render World A, apply an authoritative edit, render the same cameras of World B, derive dependency-aware influence masks (UNCHANGED / INTRINSIC / GEOMETRY / TRANSFORM / LIGHT) from the world itself, not image diff. All 5 influence classes appear across 270 edit observations.

The trainer runs the model on both before/after states and penalizes drift over UNCHANGED pixels on both heads. Measured: locality loss EMA 0.0005; sequential-edit torture shows 0 unchanged-component drift across 8 edits (move mug, rotate chair, lamp off, oak→aluminum, HAV refine, move chair, lamp on, remove mug). Material/remove invalidates exactly 1 component each.

Resolved: Intrinsic Relighting Measured (#2 weakness)

Built relighting groups (vwm_blender_batch.py renders warm/cool/off states over identical geometry with lighting-free albedo targets). Forensic test: intrinsic-head variance across lamp-on/off drops to 8×10⁻⁶, the intrinsic head genuinely ignores lighting.

3-Way Benchmark (16 held-out scenes, VWMScore v2)

Partial: Dataset Scale

600 → 1050 observations (600 standard + 270 edit pairs + 180 relight), audited 0 rejected. A real 3.5× multi-task scale-up, but not the 10k stretch target. Cycles throughput (~0.8 obs/s) caps a responsible session.

Partial: In-Luanti v2.1 Capture

Integration plugs into the identical compiler→renderer→bridge path proven in prior milestones. Live client capture not re-run to avoid disrupting the active desktop/game.

What This Proves

• Appearance behaves like a world property: a wall move leaves 0 unchanged drift; wood grain stays surface-attached; lamp off does not destroy material identity.
• Camera only observes: 1,000 camera-only frames → 0 compiler calls, 0 model calls at 89.5 FPS median (default v2).
• Honest promotion: when the checkpoint did not beat v2 on aggregate score and gate failures, v2 stayed default and v2.1 was retained with full documentation.

Superseded as production default by Appearance Model v2.2. v2.1 available as vwm_appearance_v21.pt.

VWM Appearance Model v2, Scaled, Consistency-Trained, Benchmarked, Promoted

The v1 proof became a real training system, and v2 won on evidence.

v1 Audit, What Drove v2

Channel ablation was decisive: only deferred RGB contributes (removing it costs ΔL1 +0.010); albedo, normal, and depth added nothing. v1 was learning a tone/GI transform. That proved v2 must change the task to demand structure, not just enlarge the network.

Scaled, Diverse Dataset

• tools/vwm_blender_batch.py, one bpy process per scene renders all views (beauty + lighting-free albedo/intrinsic pass + component masks), amortizing startup.
• Material library expanded to 40+ families.
• tools/vwm_appearance_dataset_v2.py generated 600 aligned VWM→Cycles pairs across 100 diverse procedural scenes (10 scene classes, 8 component types).
• 0 rejected under silhouette-parity validation, with component-local multi-view correspondence.

v2 Model & Losses

vwm/appearance_model_v2.py is a dual-head U-Net (~1.45M params): an intrinsic albedo head (relightable, multi-view-consistent) and a final RGB head (photographic).

Trained via tools/vwm_train_appearance_v2.py with photographic L1 + intrinsic L1 + edge-preservation + explicit multi-view surface consistency loss (scatter-mean intrinsic over component+local-cell keys across two views). 3,000 steps, 105s, 215 MB VRAM, mixed precision.

Honest Iteration, v2 Lost, Then Won

The first v2 attempt (intrinsic-only) lost the benchmark: VWMScore 0.584 vs v1's 0.792, because it re-lit intrinsic albedo through a crude approximation, hurting the final photographic match (L1 0.135). Rather than ship a worse model, the architecture was improved with a dual head and re-benchmarked.

Benchmark, 16 Held-Out Scenes, 96 Views

v2 is 42% better photographically than v1 on unseen diverse scenes (better generalization from 12× more diverse data), has perfect cross-view surface identity, passes the geometry gate, and its intrinsic head is relightable, which v1 is not.

Promoted & Integrated, Invariant Intact

• Compiler auto-selects the v1/v2 adapter by checkpoint version; component_local conditioning added for v2.
• Compiling a furnished house ran the model 17× (once per view), then 1,000 camera-only frames ran it 0× at 209 FPS median (p95 5.5 ms).
• All suites green; 83 modules import clean.
• Model card, benchmark JSON, and both checkpoints persist in checkpoints/vwm_appearance/.

Honest Boundaries

• Scale. 600 pairs / 100 scenes / 3,000 steps is a real generalization win, but not the 10k-100k stretch target. Pipelines are built to scale further.
• Edit-locality loss. Edit locality is guaranteed architecturally (per-component compilation, dependency fingerprints, proven in the house test), but edit-pair data primitives are scaffolded; the loss term is not wired.
• In-Luanti v2 capture. v2 plugs into the compiler→fused-renderer→photoreal-bridge path proven in-client previously; live capture was not re-run this milestone (GPU in use). Integration is wired and headlessly proven.
• Intrinsic relighting. Works architecturally via the renderer's dynamic lighting; lamp-off does not invalidate material identity. No Blender/Cycles relighting-target comparison for v2 specifically yet.

Former production default; superseded by v2.2. See benchmarks for 3-way v2/v2.1/v2.2 comparison.

VWM-Native Photographic Appearance Compiler v1, Trained, Integrated, Proven

The first genuinely trained VWM-native appearance compiler from mathematically matched VoxelWorld→Blender/Cycles data, integrated behind the compiler invariant.

What Shipped

• Blender/Cycles pipeline, bpy 5.0.1 in dedicated .venv-blender; Cycles headless with CUDA+OPTIX on RTX 2060 SUPER. tools/vwm_blender_render.py + vwm/blender_export.py.
• Camera parity, exact raycaster basis; silhouette IoU 0.9705 between Cycles and VWM G-buffer.
• Dataset, 50 aligned (deferred + G-buffer → Cycles) pairs, 10 scenes × 5 views, scene-split held-out (1008/1009, no leakage).
• Model, compact residual U-Net (11 input channels → RGB residual). Checkpoint vwm_appearance_v1.pt: 600 steps, 13s, 394 MB VRAM.
• Held-out result, L1 to Cycles target dropped 0.343 → 0.016 (21×), improving 100% of held-out views; edge-IoU 0.72.
• Integration, AppearanceCompiler(model_checkpoint=...) runs per view during compilation, geometry-gated. 6 views compiled, then 500 camera frames at 242 FPS with 0 model runs.
• Verification, 46 suites green; 82 modules import clean.

Milestone Checklist

Honest Boundaries

• Scale. First real checkpoint: 50 pairs, 600 steps, 160² procedural distribution. Decisively beats deferred baseline on held-out L1, but narrow domain.
• SD-Turbo head-to-head. Not run on the same held-out set.
• In-Luanti capture. Render path proven from prior milestone; live capture not relaunched (GPU in use).

Superseded by Appearance Model v2 for production default. v1 checkpoint remains for compatibility.