Voxel World Model: An Authoritative Voxel Architecture for Reality Capture, Persistent Appearance, and Zero-Regeneration Neural Observation

Abstract

1. Introduction

Modern neural rendering systems often conflate observation with simulation: moving a camera triggers expensive generative inference, world state lives inside a neural field, and geometry is implicit. VWM inverts this relationship.

The voxel world is the product. Renderers, depth estimators, language models, robots, sensors, replay systems, AR clients, and VR clients are tools that read or update that world under explicit contracts. An identity digging a block, a door sensor firing, a robot moving through a room, or a scan fusing depth all interact with the same authoritative structure. Photorealism is a view mode, not the simulation.

1.1 Contributions

• Authoritative adaptive voxels (HAV) with fractional sub-meter leaves, surface evidence, and hierarchy-preserving export.
• Hierarchical Adaptive Components binding geometry, semantics, behavior, and neural artifacts into recursive assemblies.
• Reality capture pipeline that turns real observations into persistent HAV surface evidence through Depth Anything, sensor depth, AR/known/SfM poses, and a global alignment graph.
• Appearance compiler producing persistent component-local bakes; geometry-gated neural inference only on dependency change.
• Zero-regeneration invariant with formal dependency fingerprints and multi-thousand-frame proofs.
• Fused reprojection renderer separating compilation from presentation (keyframe depth reprojection + partial G-buffer repair).
• Interchangeable visualization layer spanning native voxel, photorealistic, stylized, analytical, AR, and VR views.
• Honest benchmark suite including VWMScore v3, StructuralIdentityScore, controlled depth comparison, and real capture measurements.

2. Design Philosophy

VWM rests on five rules enforced in code and review:

1. Reality inputs edit voxels and semantics, never the renderer.
2. VolSplat / Gaussian splats are importers into voxels, not runtime scenes.
3. Neural rendering is a view plus persistent artifacts; work is scoped to dirty AABBs and HAV hits only.
4. Visualization never mutates simulation state.
5. Export always leaves from the voxel world; hierarchy is preserved when possible.

Why voxels? Voxel worlds are deterministic, editable, network-efficient, persistent, and naturally semantic. A chair is addressable as a chair. Physics, pathfinding, and AI context compile from the same structure humans edit.

Combined product. In the Luanti integration, VoxelWorld is the mind and Luanti is the body: identity dig/place events journal into VoxelWorld; HAV sync drives Luanti nodes and sub-meter entities. Native voxel, photorealistic, stylized, analytical, AR, and VR views can replace one another without changing authority.

3. System Architecture

The canonical pipeline is a single platform stage map, documented here as the completed VWM framework rather than a renderer-specific subsystem description:

3.1 End-to-end system process

1. Reality. A room, vehicle, factory, game world, robot environment, or digital twin emits observations and events.
2. Capture. RGB, depth, LiDAR, AR poses, smart-device events, telemetry, and user edits enter as timestamped observations.
3. Geometric alignment. Poses and depth are aligned into a shared coordinate frame through known rigs, ARKit/ARCore, visual SfM, metric anchors, and the global depth-alignment graph.
4. HAV authority. Aligned evidence is fused into the Hierarchical Adaptive Voxel world. Cells store occupancy, free space, confidence, provenance, and surface evidence state.
5. Semantic world. Geometry is grouped into hierarchical components with identity, state, behavior, capabilities, relationships, and hard facts / soft hints.
6. Persistent appearance. Real observations and AI compilation fill missing or improved appearance into local persistent representations keyed by authoritative dependencies.
7. Visualization. Native voxel, photorealistic, stylized, analytical, AR, and VR renderers observe the compiled world. Camera movement only changes the observer pose.
8. Simulation and editing. Identities, robots, sensors, AI agents, and automation mutate authoritative voxels, components, state, and events.
9. Localized recompilation. Edits invalidate only affected components or HAV regions; unchanged appearance, semantics, and surface evidence persist.
10. Export and replay. Worlds export from authority (HAV, components, timeline, optional splats/meshes) and can be replayed from recorded events.

3.2 Core modules

3.3 Behavioral loop

Continuous simulation uses WorldClock and EventScheduler. Components may sleep until woken by events, edits, or dependencies. Autonomous maintenance analyzes read-only state, schedules prioritized work, and executes within a per-tick budget. Robotics, sensors, digital twins, replay, and AI all use the same event and state layer, so there is no separate “robot world”, “render world”, or “AI world”.

4. Hierarchical Adaptive Voxels (HAV)

The authoritative world is not a fixed-resolution grid. VoxelWorld.hav is an adaptive octree:

• Coarse empty terrain remains large voxels.
• Semantics, user edits, import density, and camera proximity trigger local refinement.
• Past 1 m, leaves use fractional sub-cells (frac_origin, frac_size); Luanti draws them as scaled cube entities.
• Dirty regions and neural artifacts are keyed by hierarchy node and depth.
• Export writes hav.json and LOD; flattening is opt-in for consumers that require it.

Hierarchy: World → Regions → Chunks → Cells → Subcells → …

4.1 Surface evidence

Each occupied cell carries fusion status from multiview evidence:

Free-space carving marks interior as empty; unseen space is never filled solid. This prevents monocular hallucination from inventing walls behind furniture.

5. Hierarchical Adaptive Components

VoxelWorld.components owns recursive semantic assemblies:

Each component bundles: adaptive voxels, child components, semantics (hard facts + soft hints), state, behaviors, capabilities, interfaces, events, timeline entries, and optional neural artifacts scoped to that subtree.

• Move or delete cascades through the ownership tree.
• Dirty tracking prefers the smallest affected component (ComponentDirtyTracker, HierarchicalDirtyTracker).
• Opening a cabinet dirties only that cabinet's artifacts, not the house.

6. Reality Capture Pipeline

Reality capture ingests RGB (and optionally depth) streams, estimates or receives camera poses, aligns monocular depth globally, and fuses evidence into HAV voxels with attached real RGB.

6.1 Camera trajectory providers

6.2 Scan session

ScanSession provides atomic writes and a resumable journal so capture can survive interruption. CLI entry points: tools/vwm_scan_pipeline.py, tools/vwm_real_scan.py.

6.3 Real webcam capture (measured)

July 2026: 18 s / 425 frames from a Logitech BRIO of a real bedroom. End to end in 10.3 s, fusing 6,154 surface voxels (3,200 confirmed, 2,954 provisional). SIFT: 540 matches, 516 essential-matrix inliers. Alignment graph: 56 edges, 16,626 triangulated samples.

Single-view depth on real photos produces a coherent colored voxel cloud (closet recess, dresser, couch, wall). Multi-view fusion is rough (solved_scale_cov ≈ 30) because hand-held motion over inches yields tiny baselines. This is a capture-quality limitation, not an architecture failure.

7. Depth Anything Integration

Monocular depth is provided by depth-anything.cpp via vwm.depth_anything_adapter. We evaluate:

• DA3-SMALL (relative depth)
• DA2 Metric Hypersim Small (metric prior)
• RGB-D / LiDAR sensor depth (controlled ground truth)

7.1 Domain-gap finding

On synthetic flat-shaded voxel renders, SIFT finds zero features; on real BRIO frames, 540 matches. Depth Anything produces poor fusion on synthetic input (surface recall ~0.45–0.56) but coherent geometry on real photographs. The model is trained on real indoor imagery; non-photoreal voxel renders are out of distribution. Per-frame depth correlation with ground truth remains decent (0.76–0.96), but per-frame scale drifts 47–57% (coefficient of variation), capping multiview recall.

7.2 Controlled 3-way benchmark

Same room, same poses, same fusion; only depth source changes. All monocular runs use one robust pooled calibration (single scale anchor across all frames).

Backprojection convention verified to 0.02 world-unit error against ground truth. With reliable depth, the fusion pipeline reaches 0.918 surface recall.

8. Multi-View Depth Alignment

Monocular depth predicts good per-frame shape but scale drifts frame to frame. DepthAlignmentGraph treats each depth map as noisy geometric evidence and solves per-frame calibration jointly:

where $D_i$ is raw monocular depth for frame $i$, and $(s_i, b_i)$ are solved globally.

8.1 Algorithm

1. SIFT correspondences between overlapping frames (pose from any CameraTrajectoryProvider).
2. Two-view triangulation of matched pixels → true inter-camera distances.
3. Robust global IRLS solve linking predicted depth rays to triangulated distances.
4. Metric models: strong prior $s_i \approx 1$, $b_i \approx 0$. Relative models: scale+shift freedom.

8.2 Calibration methodology lesson

An early bug anchored scale on frame 0 alone. Frame 0 was a scale outlier (per-frame scale 8.1 vs typical ~3.9), producing backprojection error 6.76 and recall 0.39. Robust pooled calibration across all frames cut error to 2.37 (>2.5× improvement). Rule baked into pipeline: never anchor scale on one frame.

9. Surface Evidence and Multiview Fusion

multiview_fusion.py backprojects aligned depth into world space, votes occupancy per HAV cell, and applies free-space carving along rays. Cells accumulate toward CONFIRMED or CONFLICTED based on consistent observations.

Validation on held-out views reports:

• Occupancy IoU vs ground truth
• Surface precision and recall
• Mean backprojection error in world units
• Novel-view render for qualitative audit

Sensor-depth controlled run: precision 0.674, recall 0.918, 4,513 confirmed voxels, backprojection error 0.17.

10. Appearance Compilation

Photorealistic appearance is compiled, not generated per frame. The trained U-Net runs only inside AppearanceCompiler, geometry-gated, when dependency fingerprints change or a quality tier is requested.

10.1 CompiledAppearance

Output is a component-local surface bake: quantized local cells (0.25 m) storing RGB (and quality tiers UNSEEN → FINAL). Lighting is applied at render time, not compile time, so lamp toggles never recompile intrinsic material.

10.2 Dependency fingerprint

SHA-1 digest over: world_id, scope, geometry signature, material signature, semantic signature, compiler version, model identity. Properties:

• Geometry signature is rotation-invariant (minimum over four Y-rotations): rotate a desk → no recompile.
• Material and semantic signatures exclude positional voxel coordinates: move a chair → no recompile; appearance follows the component transform.
• Camera pose is not in the fingerprint.

10.3 v2.x milestone arc

10.4 v2.2 architectural fix

Forensics on v2.1 found intrinsic and light-invariance gradients nearly collinear on the shared decoder trunk (cosine +0.89), flattening features the thin residual final head reads. v2.2 fixes:

1. Decoupled intrinsic decoder off the shared encoder.
2. Structure-guided final decoder fed an authoritative G-buffer boundary map (depth / normal / component discontinuities).
3. Structural losses: boundary presence, anti-blur band, cross-boundary bleed, structure-aware locality.

10.5 FusedAppearanceRenderer

Runtime presentation uses depth reprojection of cached keyframes plus partial G-buffer repair (~10–20 ms), not full recompilation. Keyframes refresh on scene signature change or large camera rotation (keyframe_angle_deg ≈ 24°). Median 56 FPS fused path; earlier fused benchmark cited 89 FPS. Reprojection is presentation work, not compilation.

11. The Zero-Regeneration Invariant

11.1 Why this holds

1. Appearance is derived and persistent, stored per component in AppearanceStore.
2. Dependency fingerprints exclude camera pose.
3. FusedAppearanceRenderer.scene_signature() caches on world.generation and store size; it does not change when only the camera moves.
4. The U-Net executes only inside AppearanceCompiler.compile() on fingerprint mismatch or tier upgrade.
5. Runtime path reprojects keyframes and repairs holes; it never calls the compiler.

11.2 Measured proofs

11.3 What triggers regeneration

Demo house: desk material change recompiles desk only (18 components unchanged); chair move → 0 recompiles; six lighting states → 0 intrinsic recompiles.

12. Neural Rendering

The primary neural view is structure-conditioned deferred PBR, not an egocentric video generative model. Waypoint (2.3B autoregressive) was audited and rejected: wrong task, ~4 s/frame, noisy output. It remains an experimental backend only.

Camera movement reprojects existing appearance (~15–21 ms) instead of rebuilding (~55–355 ms). Blender Cycles targets provide training supervision; silhouette IoU between VWM G-buffer and Cycles is 0.97 across v2+ milestones.

13. Gaussian Representations

Appearance is unified behind AppearanceRepresentation:

gaussian_appearance.py fits Gaussians from compiled surface bakes. Gaussians follow component transforms without refit. Depth is constrained against authoritative voxel depth to prevent wall leakage.

13.1 Planned observation path: raster → local AI → global AI

The intended production Gaussian view (roadmap V13) keeps the world and raster deterministic, then applies optional enhancers:

• Local enhancement — conditioned on per-object / per-voxel metadata (semantic, material, moisture, season, confidence). Improves surface detail, materials, edges; never invents occupancy.
• Global enhancement — conditioned on scene metadata (weather, fog, sun elevation, exposure, time of day). HDR, grading, atmosphere, bloom, DOF, temporal, stylization.
• Independent toggles — either or both enhancers may be off (pure Gaussian) or on; camera motion still never regenerates world authority.

This separation matches VWM’s modular philosophy: narrowly defined stages that can improve independently without rewriting the Adaptive Voxel World. See also the homepage section Frame Enhancement.

13.1 VolSplat import path

VolSplat never becomes the runtime scene. vwm.splat_voxelize and tools/vwm_splat_to_voxels.py convert splats into voxel evidence with provenance.

14. Dirty-Region Regeneration

Three granularities cooperate:

• DirtyRegionTracker: axis-aligned bounds on flat edits.
• HierarchicalDirtyTracker: octree node hits in HAV.
• ComponentDirtyTracker: smallest semantic component affected.

Progressive neural quality (INTERACTIVE → HIGH → FINAL) compiles dirty regions first under budget. Dormant regions sleep; camera focus wakes nearby components.

15. Simulation and Semantics

15.1 Semantics

SemanticNode hierarchy: World → Region → Building → Room → Object → Component. Each node carries:

• hard_facts: deterministic (material, dimensions, state)
• soft_hints: renderer conditioning (wear, style)
• relationships, history, and live state

context_compiler builds hierarchical, budgeted prompts for AI agents from semantic trees, not raw voxels.

15.2 Simulation

VoxelWorld.simulate_tick(dt) advances continuous state: doors auto-close, batteries drain, plants grow, weather evolves. invoke_behavior and ai_act("open door") route through component capabilities. Visualization observes state; it never originates behavior.

15.3 Digital twins

Sensors → event bus → semantic state (vwm.sync, vwm.twin_bridge.TwinBridge). Timeline replay (vwm.timeline) records every change for debugging, training datasets, and forensic analysis. Robotics interfaces share the same world model (vwm.robotics, vwm.robot_nav).

16. Export and Import

16.1 Export (from authority only)

export_io.py writes:

• hav.json hierarchy and LOD
• components.json semantic assemblies
• blocks JSONL, world packs, timeline datasets, optional Gaussian exports

Hierarchy is preserved unless a downstream consumer explicitly requires flattening.

16.2 Import paths

• Scan pipeline → HAV voxels with surface evidence
• VolSplat / Gaussian → voxel occupancy (vwm.reconstruction)
• Blender authoring targets (vwm.blender_export)

All imports attach provenance and confidence; reconstruction module supports incremental updates.

17. Development Methodology

VWM development is evidence-driven and promotion-gated:

1. State the invariant (e.g., zero regeneration on camera move) before optimizing aesthetics.
2. Build measurable proofs (test_vwm_1000frame_proof.py, test_vwm_10kframe_proof_v22.py) that count model and compiler invocations.
3. Freeze scoring formulas (VWMScore v3) before ranking candidates.
4. Run candidate ladders isolating one axis at a time (v2.2 candidates A–G).
5. Promote only on Pareto improvement against the production default; v2.1 and v2.3 were completed but not promoted when standard metrics regressed.
6. Document failures honestly in MODEL_CARD, MILESTONE_STATUS, FAILURE_ATLAS, and scan comparison markdown.
7. Separate compilation from presentation so FPS optimizations cannot accidentally reintroduce per-frame inference.

17.1 Data engine (v2.3)

Bottleneck identified as experience, not architecture: dataset grew 1,050 → 2,757 observations via rare-geometry factory (1.79 obs/s, crash-safe journal, active planner avoiding ~90% of Cycles work). v2.4 was explicitly not trained under the milestone rule; appearance promotion stays tied to frozen metrics, not chronological version number.

18. Experiments and Benchmarks

18.1 VWMScore v3 (frozen)

where liv penalizes intrinsic light-variance and loc penalizes unchanged-region drift under edits.

18.2 StructuralIdentityScore

Harmonic-mean aggregate over four axes measured against the G-buffer boundary map (never Cycles RGB):

• boundary_recall: authoritative edges that survive as luminance edges
• boundary_precision: render edges on real structure (anti-hallucination)
• thin_survival: thin structures keep local contrast
• bleed_free: materials do not smear across seams

18.3 Appearance benchmark (standard held-out, 16 scenes)

18.4 Structural torture house

18.5 v2.3 vs v2.2 (not promoted)

Full tables: benchmarks page. Narratives: project updates.

19. Validation Boundaries and Documented Failures

19.1 Failed iterations (measured)

19.2 Validation boundaries in current artifacts

• Dataset scale: 2,757 / 10,000 target observations; Cycles factory ~0.8–1.79 obs/s on RTX 2060 SUPER.
• Monocular depth on synthetic renders: recall caps ~0.45–0.56 vs sensor 0.918.
• Real multi-view fusion with small baselines: scale CoV ≈ 30 in the BRIO test; metric-quality runs use walk-around baselines, AR poses, or metric anchors.
• Scan artifact coverage: the delivered real run validates capture→depth→align→fuse→attach→novel-view; loop closure, incremental rescan, semantic discovery, and live Luanti scan opening are separate acceptance surfaces.
• Appearance on scan artifacts: current real scan stores voxel-mean RGB; the complete architecture routes scan-derived components through the same persistent appearance compiler used by authored worlds.
• In-Luanti live client acceptance not re-run (requires human walkthrough).
• v2.4 not trained by explicit milestone rule.

20. Complete System Operation

This section states the finished VWM framework as an operational system. The implementation evidence above measures individual surfaces of that system; the architecture itself is a single closed loop.

20.1 Reality to authoritative world

1. Capture enters as evidence. Phone cameras, webcams, RGB-D sensors, LiDAR, ARKit/ARCore poses, robot cameras, IoT sensors, game edits, and imported reconstructions produce timestamped observations. They do not become the world directly.
2. Geometry is aligned. Camera trajectories and depth are put into a common frame using known poses, AR poses, visual SfM, metric anchors, and the depth-alignment graph. The result is metric or anchored evidence, not renderer state.
3. HAV stores authority. Aligned rays and surfaces update Hierarchical Adaptive Voxels with occupancy, free-space carving, provenance, confidence, and surface evidence classes. The authoritative world is the HAV plus component graph, not the captured video or depth map.
4. Real observations persist. Real RGB and depth-derived surface measurements are attached to local cells/components as durable evidence. Later observations refine, confirm, conflict with, or supersede earlier evidence.

20.2 Authority to semantic simulation

1. Semantic components give identity. HAV regions become rooms, doors, chairs, desks, machines, shelves, vehicles, or arbitrary user-defined components. Each component owns geometry, hierarchy, state, behavior, capabilities, interfaces, and history.
2. All actors share state. Simulation, AI, robotics, sensors, digital twins, replay, editor tools, and clients read/write through the same VoxelWorld contracts. There is no secondary state graph for a robot, renderer, or AI agent.
3. Events are replayable. Identity edits, sensor changes, robot actions, scans, AI decisions, and scheduled behaviors append to a timeline. Replay reconstructs the world by applying authoritative events, not by replaying pixels.

20.3 Semantic world to persistent appearance

1. Observed appearance is reused. Real captured colors and materials initialize local appearance where evidence exists.
2. AI compiles missing or improved appearance. The compiler fills gaps, upgrades quality tiers, and improves material detail only when authoritative dependencies require it. The compiled result is stored as component-local appearance, not a transient frame.
3. Representations are local and swappable. Surface bakes, Gaussian components, hybrid depth-composited layers, and neural residuals all attach to authoritative components. They can be exported or replaced without becoming the source of truth.

20.4 Observation, editing, and localized recompilation

1. Camera movement observes. Moving the camera changes only observer pose. Native voxel, photorealistic, stylized, analytical, AR, and VR views render the compiled world without invoking the appearance compiler.
2. World edits invalidate locally. A material edit dirties that component; a geometry edit dirties affected HAV nodes; a component move carries its local appearance with the transform. Unchanged components and regions keep their compiled artifacts.
3. Recompilation is scoped. Dirty-region tracking chooses the smallest affected component or HAV region, rebuilds only required artifacts, and leaves the rest of the world interactive.

20.5 Export, import, and replay

1. Export leaves from authority. HAV, components, timeline, semantics, and optional appearance representations are exported from VoxelWorld.
2. Reconstruction formats are boundary representations. Gaussian splats, VolSplat outputs, meshes, point clouds, blocks JSONL, and world packs are import/export formats. They do not own simulation.
3. Replay is first-class. A recorded VWM timeline can reproduce state for debugging, dataset generation, robotics simulation, digital twin audits, or historical visualization.

21. Conclusion

Voxel World Model demonstrates that a persistent, authoritative voxel simulation can host reality capture, semantic behavior, and photorealistic observation without collapsing them into a single neural field. The zero-regeneration invariant is not aspirational: it is enforced by dependency fingerprints, proven to 10,000+ frames, and compatible with production-quality appearance (v2.2, VWMScore v3 0.773).

On capture, the pipeline reaches 0.918 surface recall when depth is reliable, runs end to end on real bedroom footage in 10.3 seconds, and cleanly separates domain-gap effects (synthetic vs real imagery) from architectural limits. Metric walkable replicas use the same framework with capture modes that provide stable scale: wide baselines, AR poses, sensor depth, or anchors.

Every layer is replaceable. The voxel world endures.