The complete intended path from real-world input to display — with the Adaptive Voxel World as the single source of truth, and AI only improving observations and caches.
Background improvement can update the Gaussian cache without rebuilding the voxel world, re-running simulation, re-importing data, or interrupting rendering.
Raw captures become time-aligned, calibrated ObservationPackets ready for registration — never silent merges of incompatible frames. See V20.
Observations accumulate into evidence. Agreement raises confidence; contradiction lowers it without overwriting history. Pose graphs align multi-capture sessions into one world frame.
Authoritative world state — the single source of truth.
Everything below this line is derived or observational. Gaussians, meshes, neural frames, and stylizations never replace this layer.
Simulation advances world state over time through the Runtime. Compiler and visualization respond automatically (e.g. door open → dirty chunk → gaussian recompile) with no special orchestration in the sim engine.
Outputs (artifacts): Gaussian Scene · Mesh Scene · Voxel Scene · Point Cloud · future render formats. Compilers never write the authoritative world — they publish committed artifacts only.
A derived, chunked representation of the world for real-time observation — not authority.
Runs whenever compute is available — not every frame. Improves the Gaussian representation itself; results are cached as versioned revisions. See V19.
FLUX · HiDream · future foundation models — improve the Gaussian representation offline or on spare capacity. They do not run in the real-time display path. Registered as guarded plugins until GPU weights exist.
Enhanced Gaussian Cache (V19) — optimized splat revisions, nothing overwritten, history preserved. Browser / V18 path transparently prefer best_arrays(). Only recompute when necessary.
Runs when enabled. Packet-only — never touches the voxel world. Profiles scale low (passthrough) → medium → high → cloud. Live classical-CV enhancers; AI stages (RealESRGAN / photorealistic / reflection) are registered but guarded until GPU weights exist.
→ Enhanced Frame — a photorealism-ward Gaussian image, still an observation. See V18 report and architecture status.
Operates on the final image only — entirely optional (still goal):
See also Frame Enhancement for the local vs global separation and independent local_ai / global_ai toggles.
Invariant: heavier models improve the cached Gaussian representation. They never become the simulator, never invent occupancy into authority, and never block the real-time path.
| Pipeline stage | Status | Milestone |
|---|---|---|
| Ingest / reality capture | Shipped | V20 · report |
| Evidence / confidence | Shipped | V11 Runtime |
| Registration / pose graph | Shipped | V14 Mapping |
| Adaptive Voxel World | Authority | Core (capture era → present) |
| Reactive Runtime (event bus) | Shipped | V21 · report |
| Simulation | Shipped | V17 |
| World Compiler | Shipped (Part 1) | V15 |
| Gaussian artifact + raster | Foundation | V12 · V16 |
| Background Gaussian opt + cache | Shipped (classical) | V19 · report |
| Local Gaussian enhancers | Shipped (classical-CV) | V18 · report |
| Neural enhancement backends | Shipped (RealESRGAN CUDA) | V22 · report |
| Semantic world engine | Shipped (evidence + hypotheses) | V23 · report |
| Persistent world identity | Shipped (timeline + replay) | V24 · report |
| World reasoning engine | Shipped (facts + causal graph) | V25 · report |
| Integrated world kernel | Shipped (tx · events · snapshot) | V26 · report |
| World services / query | Shipped (cross-layer explain) | V27 · report |
| Distributed replication / sync | Shipped (vector clocks · conflicts) | V28 · report |
| Distributed trust / operation replication | Shipped (ops · HMAC · recompute) | V29 · report |
| Kernel finalization / v1.0 | Shipped (regression · Ed25519 · governance) | V30 · report |
| Optional full-frame / display AI | Goal | Later · enhancement layers |
Architecture status → Multimodal World I/O → Roadmap → Updates → Frame enhancement → Paper →