SOMNIA: Reconstructing Renderer State From AI Video
An operational experiment in recovering temporal rendering guidance from synthetic footage that never had a renderer
https://youtu.be/PGwVhuuKol0
On October 3, 2026, Black Sail Studio completed the first end-to-end run of SOMNIA, an experimental rendering subsystem being developed inside EVA Destiny. In that run, an AI-generated video was processed through depth reconstruction, dense optical-flow reconstruction, temporal analysis, renderer-state adaptation, and a downstream neural-rendering stage conditioned on reconstructed motion state, producing a visible processed video output. Depth participated as an internal geometric prior for temporal reasoning and history control, while adapted motion data was the runtime state handed into the neural-rendering path. The result is significant because the source clip never originated from a conventional 3D renderer. There was no native Z-buffer, no object ID buffer, no authoritative geometry, no persistent scene graph, and no engine-generated motion-vector field. There were only pixels, and SOMNIA’s purpose is to determine how much useful renderer-like state can be reconstructed from those pixels after the fact.
That distinction defines the project more precisely than calling it a video-enhancement system. A conventional real-time renderer begins with structured scene information and eventually produces an image. It knows where objects are, how the camera moved, which surfaces occlude others, and how a point that existed in one frame projects into the next. Modern temporal and neural rendering methods can exploit that information because the renderer owns the state that produced the pixels. Generative video reverses the relationship. It can produce an image sequence that appears to describe a coherent three-dimensional world without exposing any of the conventional renderer state associated with that world. A dog can run across a field, a face can rotate toward the camera, or an entire room can appear to move through perspective while downstream software receives only RGB frames and timestamps. SOMNIA begins from that finished appearance and attempts to recover a useful approximation of the missing spatial and temporal structure.
The project is therefore best understood as a form of renderer-state reconstruction from synthetic video. It is not full 3D reconstruction in the classical sense, and it is not attempting to recover ground-truth geometry that must necessarily exist somewhere behind the image. In many generative systems, such a ground-truth scene may never have existed in a conventional representation at all. The video model may synthesize coherent-looking motion in latent space without maintaining a persistent polygonal world, a stable camera model, or semantically consistent object identities. SOMNIA instead reconstructs surrogate state: estimated depth, apparent motion, temporal confidence, occlusion cues, and reset decisions that can support downstream temporal processing.
The research direction emerged from the intersection of several independent technical developments. NVIDIA’s DLSS 5 work provided the clearest example of why modern neural rendering requires more than a color image. NVIDIA describes DLSS 5 as a real-time generative rendering stage that uses the game engine’s rendered frame as its grounding input while also relying on engine motion vectors and carried temporal state to maintain deterministic temporal behavior. NVIDIA’s September 2026 developer material explicitly describes a strict one-frame-in, one-frame-out model grounded by the engine frame and its motion vectors, while the ADLR research page presents DLSS 5 as a generative neural-rendering stage for interactive graphics. NVIDIA Developer
That interface correction is important because it changes how SOMNIA’s own reconstructed signals should be described. Depth is valuable to SOMNIA, but it is not being treated as a drop-in runtime substitute for an engine buffer consumed directly by DLSS 5. The current published DLSS 5 runtime description centers on the rendered color frame, engine motion vectors, carried temporal state, and artistic controls. SOMNIA therefore uses depth primarily as an internal prior for its own temporal reasoning: estimating whether a region is likely foreground or background, identifying possible occlusion and disocclusion, checking whether warped history remains geometrically plausible, and deciding when prior temporal information should be rejected. The more direct renderer-facing problem is motion.
That problem is harder than it appears because optical flow and renderer motion vectors are not the same thing. Optical flow estimates apparent image-space correspondence between frames. If a visual feature that occupied \((x,y)\) in frame \(t\) appears at approximately \((x+u,y+v)\) in frame \(t+1\), a dense flow estimator can describe that apparent displacement as \(F_t(x,y)=(u,v)\). A game engine’s motion-vector buffer, by contrast, is derived from known scene geometry, object transforms, and camera transforms. It therefore represents temporal movement according to a precise rendering convention rather than merely inferring that two patches of pixels resemble each other.
This difference matters because downstream temporal systems may expect motion in a very specific convention. A renderer may consume backward reprojection vectors rather than forward flow, normalized coordinates rather than pixel displacement, pre-jitter or post-jitter positions, or vectors scaled to a particular render resolution. A video-derived flow field cannot simply be copied into such an interface and assumed to be correct. SOMNIA therefore treats optical flow as an intermediate representation from which renderer-facing motion guidance must be adapted.
One of the primary research influences on that stage is SEA-RAFT, introduced by Yihan Wang, Lahav Lipson, and Jia Deng of Princeton University at ECCV 2024. SEA-RAFT simplifies and accelerates the RAFT optical-flow family using a mixture-of-Laplace loss, direct initial-flow regression, and rigid-motion pretraining. The authors reported state-of-the-art Spring benchmark performance at publication and at least a 2.3× speed improvement over existing approaches while maintaining competitive accuracy. European Computer Vision Association That efficiency is particularly useful in SOMNIA because optical flow is only one stage in a larger local pipeline. A motion estimator that consumes most of the system’s runtime or VRAM would make the overall architecture substantially less practical.
In a reproducible implementation, the optical-flow stage should ideally estimate both forward and backward motion. Let \(F_{t\rightarrow t+1}\) map a point from the current frame into the next frame and \(F_{t+1\rightarrow t}\) map the predicted destination back again. For a region with consistent correspondence, the two should approximately cancel:\[ F_{t\rightarrow t+1}(p) + F_{t+1\rightarrow t} \left( p + F_{t\rightarrow t+1}(p) \right) \approx 0 \]
The magnitude of the disagreement,\[ E_{FB}(p)= \left\| F_{t\rightarrow t+1}(p) + F_{t+1\rightarrow t} \left( p + F_{t\rightarrow t+1}(p) \right) \right\|, \]
becomes a useful temporal confidence signal. Large disagreement can indicate occlusion, disocclusion, ambiguous correspondence, generative morphing, newly created detail, or outright flow failure. It does not prove which of those occurred, but it provides evidence that prior-frame history should be trusted less.
Depth supplies a complementary form of evidence. SOMNIA’s geometry path is based on temporally consistent monocular depth estimation rather than treating each frame independently. The strongest research fit for that problem is Video Depth Anything, presented at CVPR 2025 by Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang, Jiashi Feng, and Bingyi Kang at ByteDance. The model builds on Depth Anything V2 but replaces its image-only head with an efficient spatial-temporal head, introduces a temporal consistency loss based on temporal depth gradients, and uses a key-frame-based strategy for long-video inference. The authors explicitly designed the system to reduce the temporal inconsistency that arises when monocular depth is predicted independently for each frame. CVF Open Access
The published Video Depth Anything model sizes are approximately 28.4 million parameters for Small, 113.1 million for Base, and 381.8 million for Large. The repository provides both relative-depth and metric-depth variants at those scales. GitHub For a staged local pipeline targeting a 16 GB-class GPU, the Small and Base variants are particularly relevant because SOMNIA does not need the depth model to win a standalone image-quality contest. It needs the depth field to remain temporally coherent enough to support history gating, occlusion reasoning, and geometric consistency tests.
That temporal requirement is more important than it may first appear. A single-frame monocular depth model can produce individually convincing depth maps while allowing the scene to “breathe” through time. An object that should remain stationary might receive slightly different relative depth on each frame. If a downstream temporal system treats those predictions as ground truth, the result can be ghosting, incorrect disocclusion, geometry drift, or unnecessary history rejection. Video Depth Anything is attractive to SOMNIA precisely because it treats depth as a temporal signal rather than as a stack of independent still-image predictions.
Even then, relative depth must be handled carefully. Monocular relative depth can drift in scale and offset, so two temporally adjacent predictions may preserve correct scene ordering while differing by an affine transform. A useful model is\[ D’_t \approx a_t D_t + b_t, \]
where \(a_t\) and \(b_t\) may vary with time. SOMNIA should therefore estimate alignment only over high-confidence correspondences rather than over the entire frame, because low-confidence regions are exactly where generative morphing, occlusion, or flow failure are likely to contaminate the fit. Once aligned, a warped previous depth estimate can be compared with the current depth to derive a geometric consistency signal.
A simple depth-consistency term can be defined by warping previous depth into the current frame:\[ \hat{D}_t = \mathrm{warp}(D_{t-1}), \]
then computing\[ E_D(x,y)=|D_t(x,y)-\hat{D}_t(x,y)|. \]
Because relative depth scale can vary, this residual should ideally be evaluated after the affine alignment described above. Large disagreement does not automatically mean the current frame is wrong; it may represent true scene change, disocclusion, generator morphing, or imperfect motion estimation. What matters is that the disagreement reduces confidence in reusing history.
This brings SOMNIA to the part of the system that may ultimately matter more than the choice of either the depth model or the flow model: temporal history control. A temporal renderer becomes powerful because it remembers previous frames, but the same memory becomes destructive when prior information ceases to describe the current scene. Generated video makes this particularly dangerous because the source model can introduce new detail without physical cause, alter object topology, or transition between locally plausible but globally inconsistent states.
SOMNIA therefore treats history as conditional rather than permanent. A stable region should retain previous temporal information. An uncertain region should attenuate it. A scene cut or catastrophic morph should invalidate it. A simple history update can be written as\[ H_t = \alpha_t \, \mathrm{warp}(H_{t-1}) + (1-\alpha_t)\,I_t, \]
with\[ \alpha_t \in [0,1]. \]
Here \(I_t\) is the current frame, \(H_{t-1}\) is prior accumulated state, and \(\alpha_t\) is a confidence-derived history weight based on some combination of flow consistency, depth agreement, photometric residual, and scene continuity. When \(\alpha_t\) is high, trusted history contributes strongly. When it approaches zero, the current frame dominates. At a hard cut or catastrophic discontinuity, the history can be reset entirely.
Before applying that logic to generative footage, the renderer-state adapter should be validated against analytically controlled scenes. This is one of the most important parts of making the system reproducible because it allows implementation errors to be separated from generator errors. A simple synthetic square translating exactly sixteen pixels per frame provides a known motion field. A controlled camera pan tests global motion. A moving object combined with a moving camera tests whether local and global motion remain distinguishable. A foreground occluder moving across a background exposes disocclusion handling. A hard cut verifies that temporal history is cleared rather than carried into an unrelated frame.
Those tests should be run before treating a complex AI-generated clip as evidence about the adapter itself. Otherwise, a visible artifact could have originated in the video generator, the flow estimator, the depth model, coordinate conversion, history logic, resolution scaling, or the neural renderer, and there would be no clean way to tell which component failed. A deterministic 16-pixel translation is not glamorous, but it is a far stronger diagnostic tool than an aesthetically complex dream sequence.
Once those deterministic tests are passed, generative video becomes the real stress case. One of EVA Destiny’s first official dream sequences provides a useful example. The generated video depicted a golden retriever in an outdoor scene, and the model introduced water droplets around the dog’s nose. The droplets looked plausible, but there was no guarantee that they had a causal predecessor in the previous frame. A depth estimator will still assign them apparent spatial structure. An optical-flow model will still attempt to infer motion. If the resulting state is accepted without sufficient confidence checks, the temporal system may promote those invented droplets into persistent scene state.
This example illustrates a fundamental limitation of renderer-state reconstruction: rendering cannot determine semantic truth simply by improving temporal coherence. A hallucinated object can be reconstructed very well. SOMNIA can determine whether a visual feature appears temporally consistent, whether it aligns with depth, and whether it can be reprojected coherently. It cannot, by those mechanisms alone, determine whether that feature should exist. Semantic verification would require an additional layer of reasoning beyond the rendering problem itself.
The renderer-facing adaptation stage is therefore the point at which SOMNIA stops being ordinary computer vision and becomes a rendering experiment. Optical flow must be transformed into the conventions required by the downstream neural stage. That means documenting direction, scale, coordinate space, resolution, jitter treatment, frame timing, invalid-vector semantics, and reset behavior. If a source flow field is estimated in pixel units at resolution \(W_f\times H_f\) but consumed at \(W_r\times H_r\), the displacement must be scaled appropriately:\[ u_r=u_f\frac{W_r}{W_f}, \qquad v_r=v_f\frac{H_r}{H_f}. \]
If the downstream system expects backward motion rather than forward displacement, the field must be converted accordingly rather than merely negated without considering reprojection semantics. If temporal jitter is present in the target pipeline, the adapter must either remove or reproduce it in the same coordinate convention. These details are not implementation trivia. They determine whether the neural renderer sees meaningful motion state or corrupted guidance.
The second major engineering influence on SOMNIA came from the experimental OptiScaler DLSSNR Pre-SR Multipass work. This community branch exposes several behaviors relevant to SOMNIA’s renderer architecture. Its RunBeforeSR=true option runs Neural Rendering on the color input before Super Resolution, and its Passes setting allows one to three sequential neural-rendering layers, each with independent persistent history. The project’s design documentation explicitly describes these passes as owning persistent NGX features and temporal history across frames. GitHub
The importance of this work is architectural rather than turnkey. OptiScaler’s experimental neural-rendering branch operates in the context of a game rendering pipeline and expects renderer semantics and buffers that an arbitrary MP4 does not provide. It is therefore not a tool that can simply be pointed at AI-generated frames to produce SOMNIA automatically. Instead, it demonstrates useful behavior around pass placement, pre-Super-Resolution processing, persistent temporal feature lifetimes, multipass sequencing, and the practical integration of DLSS Neural Rendering into an existing graphics path. The project itself is explicit that it is an experimental community implementation rather than an NVIDIA-supported production integration. GitHub
The RunBeforeSR behavior is especially relevant because it allows Neural Rendering to operate closer to the native render resolution before final upscaling. In a synthetic-video context, this suggests an ordering in which generated imagery is first converted into renderer-like temporal state, then passed through neural rendering, then reconstructed to final display resolution. Whether this ordering produces a measurable improvement over post-SR processing is an empirical question, not an assumption, but OptiScaler demonstrates that such ordering is technically meaningful and that temporal history can persist across those neural passes. GitHub
Inside EVA Destiny, SOMNIA sits downstream from a broader local generative chain. Conversation and persistent memory are first converted into a dream or visualization concept by a local language model. That concept drives image generation. The generated image becomes input to a local video model. Once a synthetic clip exists, SOMNIA takes over. Frames are decoded, temporally consistent depth is estimated, dense motion is reconstructed, history confidence is calculated, reset logic decides which temporal information remains valid, and the motion path is adapted into the convention required by the downstream neural stage. In the October 3 run, the neural stage consumed adapted reconstructed motion state while depth served as an internal prior for temporal gating and history logic. This distinction matters because depth and motion are coupled to the pipeline differently.
That is also the reason the system should not be described merely as “AI video passed through DLSS.” The load-bearing claim is that reconstructed state participates in the rendering process. A chain that computes depth and flow for visualization and then independently runs a neural enhancement pass on RGB would be an interesting analysis pipeline, but it would not demonstrate the same thing. SOMNIA’s end-to-end milestone is specifically that reconstructed temporal state was used inside the downstream path that produced the processed result.
The entire architecture is designed around local execution, which introduces a second class of constraints beyond the rendering problem itself. EVA Destiny’s visual stack includes multiple large models that cannot all remain resident simultaneously on a 16 GB consumer GPU. Instead of treating that as a reason to move the workload into the cloud, the system schedules VRAM in time. The language model loads, generates the dream script, and unloads. The image model loads, generates the keyframe, and unloads. The video model takes its turn, followed later by the depth and optical-flow stages. Neural rendering and final composition occur after those resources are released.
The consequence is that aggregate capability can exceed simultaneous VRAM capacity. The trade-off is latency. SOMNIA accepts that trade because EVA’s dream system is an asynchronous workload. A dream can render during idle periods, after a conversation, or while the user is doing something else inside the application. The architecture therefore treats time itself as a schedulable resource. A pipeline that would be impractical if every stage had to execute within a real-time frame budget can still be entirely reasonable if stages are serialized and the result is presented later.
This local scheduling philosophy also makes experimental reproducibility easier because every stage can be instrumented independently. A research-mode SOMNIA run should save not just the final video but the intermediate buffers that explain how the output was produced. At minimum, that means source RGB frames, depth predictions, forward flow, backward flow, flow-confidence maps, aligned depth, depth-consistency error, photometric residual, history weight, reset masks, adapted renderer motion, neural-rendering output, and final composed output. Debugging views should include flow visualizations, depth maps, warped previous frames, confidence heatmaps, and raw-versus-processed differences.
Those diagnostics are essential because the likely failure modes sit at the boundaries between systems. Relative depth can drift even while looking visually plausible. Optical flow can become meaningless when a video model changes object topology. A newly exposed region can inherit history that never belonged to it. A forward flow field can be handed to a renderer that expects backward reprojection. Motion vectors can be scaled incorrectly between inference resolution and render resolution. A scene cut can be misclassified as extreme motion. Many of those mistakes produce superficially similar artifacts in the final video, so a reproducible system must make the intermediate state observable.
The first end-to-end SOMNIA run establishes functional validation of the chain, but it does not yet establish how much each component improves the result. That measurement gap should remain explicit. The next experimental phase is an ablation study in which the same source clip is processed through controlled configurations: raw generator output, neural rendering without reconstructed state as a control, depth-assisted temporal logic, temporally stabilized depth, flow-assisted temporal processing, combined depth and flow, combined depth and flow with reset logic, and the full SOMNIA path. These are planned comparison conditions; the existence of the control list should not be read as a claim that every variant has already been run.
Each configuration should record the same experimental metadata: input resolution, output resolution, frame count, frame rate, model versions, inference resolution, per-stage runtime, peak VRAM, CPU memory, load and unload overhead, number of history resets, mean flow consistency, mean depth consistency, total processing time, and final encoding parameters. Without that information, visual comparisons become difficult to interpret because a quality difference could come from model state, scaling, encoding, or an unrelated processing change.
Temporal evaluation also requires different metrics from still-image evaluation. PSNR or single-frame perceptual similarity can report that individual frames look clean while missing severe temporal flicker, texture crawling, geometry drift, or unstable identity. A more meaningful SOMNIA evaluation should include motion-compensated residuals, forward-backward flow consistency, temporal perceptual error, depth reprojection consistency, flicker energy in nominally static regions, geometry drift, and reset frequency. A system that appears stable only because it throws away history every frame is fundamentally different from one that successfully preserves useful temporal state for long sequences, so the measurements should reveal that distinction.
A motion-compensated residual provides one useful baseline. If the previous frame is warped into the current frame using estimated backward motion,\[ \hat{I}_t = \mathrm{warp}(I_{t-1}), \]
then a residual field\[ E_I(x,y)= \left\| I_t(x,y)-\hat{I}_t(x,y) \right\| \]
can indicate where the current observation disagrees with temporal prediction. Large residuals may result from true lighting changes, disocclusion, flow error, object morphing, or newly hallucinated detail. The residual therefore should not be interpreted as an error label by itself, but it is highly useful as an uncertainty term.
The same philosophy applies to depth. After aligning depth scale on high-confidence correspondences and warping the previous depth into the current frame, the resulting geometric residual can be incorporated into the confidence model rather than used as an absolute truth test. A conceptual per-pixel confidence can be written as\[ C_t(x,y)= f\!\left( C_{\mathrm{flow}}, C_{\mathrm{depth}}, C_{\mathrm{photo}}, C_{\mathrm{scene}} \right), \]
where \(f\) may initially be a hand-designed weighting rule and could later become a learned fusion model. The important point is that SOMNIA does not need to pretend any one reconstructed buffer is perfect. It needs to combine imperfect signals in a way that allows reliable history to survive while unreliable history is rejected.
This probabilistic treatment is especially important because generated video is not bound by the physical assumptions under which many computer-vision methods were originally developed. A real-world camera records objects that occupy physical space and persist causally through time. A generative model can create a plausible object in one frame that has no true predecessor in the previous frame. It can alter surface texture while preserving silhouette, change silhouette while preserving texture, or introduce lighting changes that do not correspond to any coherent source. SOMNIA is therefore not reconstructing a hidden physical reality. It is reconstructing the most useful temporal state consistent with the visible evidence.
That makes the project conceptually unusual. A generative model first invents a world as pixels. Computer vision then attempts to infer geometry and motion from that invented world. A temporal system decides which parts of those inferences remain trustworthy. A renderer-facing adapter translates the resulting motion state into the conventions expected by a neural graphics pipeline. The neural renderer then reinterprets the same synthetic scene using information reconstructed from the scene itself.
In a conventional renderer, scene state precedes the image.
In SOMNIA, the image precedes the scene state.
That reversal is the core experiment.
The October 3 run demonstrates that the chain can operate end to end. It does not mean the architecture is finished, and it does not mean every reconstructed signal has already been shown to improve output quality independently. The precise milestone is narrower and more useful: a generated video can enter SOMNIA, depth and motion can be reconstructed, depth can participate in temporal confidence and history logic, adapted motion state can drive the downstream neural-rendering path, and a processed video can emerge for inspection. That is functional validation of the architecture on a real synthetic-video input.
The next phase is characterization. The system now needs deterministic motion tests, controlled ablations, temporal metrics, VRAM traces, stage timings, and side-by-side output comparisons. Those measurements will establish how much benefit comes from motion reconstruction, how much comes from depth-assisted history control, where reset logic prevents error accumulation, and where generative inconsistency exceeds what reconstruction can recover.
Longer term, the most elegant solution may be for generative video models to expose structured auxiliary state directly. A future model could produce not only RGB frames but also temporally coherent depth, motion, semantic masks, camera estimates, surface normals, object correspondence, and confidence fields. If such outputs become standard, SOMNIA’s reconstruction stages could become smaller or disappear entirely. The renderer would no longer have to infer the implied scene from finished pixels because the generator itself could provide the latent state required for downstream temporal graphics.
Until then, SOMNIA explores a different possibility: that useful renderer state can be reconstructed after generation well enough to bridge synthetic video into systems designed around temporal rendering.
That is the milestone.
Not that a video was upscaled.
Not that depth and flow can be visualized.
Not that several models can be chained together.
The milestone is that footage which never possessed a conventional renderer can be given an approximate renderer state afterward, and that reconstructed state can be used inside a real downstream neural-rendering path.
Example Video: Locally Generated 720p
https://youtu.be/78_EP8bJdnI
Research References
NVIDIA Applied Deep Learning Research. DLSS 5: Generative Neural Rendering. Published September 1, 2026. NVIDIA describes DLSS 5 as a real-time generative rendering stage for interactive graphics and presents the broader technical basis for the system. NVIDIA
NVIDIA Developer Technical Blog. What’s New for Game Developers: DLSS 5 with 3D-Guided Neural Rendering, NVIDIA ACE Updates, and New RTX Kit Capabilities. September 22, 2026. NVIDIA describes DLSS 5 as using the game engine’s rendered frame as its foundation, operating one frame in / one frame out, and relying on engine motion vectors for temporally stable output. NVIDIA Developer
Wang, Yihan; Lipson, Lahav; Deng, Jia. SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow. ECCV 2024, Princeton University. The authors report strong Spring benchmark results and at least 2.3× higher speed than previous approaches while maintaining competitive performance. European Computer Vision Association
Chen, Sili; Guo, Hengkai; Zhu, Shengnan; Zhang, Feihu; Huang, Zilong; Feng, Jiashi; Kang, Bingyi. Video Depth Anything: Consistent Depth Estimation for Super-Long Videos. CVPR 2025, ByteDance. The work extends Depth Anything V2 with a spatial-temporal head, temporal-gradient consistency loss, and a key-frame-based strategy for long-video inference. CVF Open Access
Depth Anything. Video Depth Anything implementation and pretrained models. The project publishes Small, Base, and Large relative- and metric-depth variants at approximately 28.4M, 113.1M, and 381.8M parameters. GitHub
wilsjo2. OptiScaler-DLSSNR-PreSR-Multipass. Experimental community fork documenting pre-Super-Resolution Neural Rendering, one-to-three sequential passes, persistent temporal history, and renderer integration behavior. This is community experimental work, not an NVIDIA-supported SOMNIA implementation. GitHub
-Sean
#ai #artificialIntelligence #dlss #nvidia
