RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors
Anonymous Authors
The framework
From observation to interaction.
RECAST combines vehicle generation, scene registration, and planner-in-the-loop rendering.
Enlarge figure ↗Use a segmented vehicle image from a driving log.
Adapt the image-to-3D prior in two stages.
Align and insert the actor into the reconstructed scene.
Render, plan, update, and repeat.
The dataset
RECAR Dataset
Background-free, multi-view imagery connects an image-to-3D prior with the diversity of real vehicles.
Vehicle color and type distributions
Enlarge figure ↗Compare RECAR with existing vehicle datasets
| Dataset | Vehicles | Type | Views | Resolution | BG-free | Regular |
|---|---|---|---|---|---|---|
| SRN-Car | 2.1K | Synthetic | 250 | 128 × 128 | Yes | Yes |
| Objaverse-XL-Car | 11.4K | Synthetic | — | — | Yes | Yes |
| MVMC | 576 | Real | 10 | 600 × 450 | No | No |
| 3DRealCar | 2.5K | Real | 200 | 1920 × 1440 | No | No |
| MAD-CARS | 70K | Real | 85 | 1920 × 1080 | No | No |
| RECAR | 20K | Real | 30 | 1200 × 800 | Yes | Yes |
Views: average number of views or video frames per vehicle. BG-free: background-free images are provided or can be rendered from supplied assets. Regular: viewpoints follow a predefined sampling pattern. —: not applicable.
Closed-loop evaluation
Closed-loop planner evaluation.
Planner-in-the-loop demonstrations with GTRS-Dense under controlled ego–actor interactions beyond log replay.
Scene 002 · Stationary-facing
Compare closed-loop results
| Method | NC (%) ↑ | DAC (%) ↑ | TTC (s) ↑ | Rc (%) ↑ | COM (%) ↑ |
|---|---|---|---|---|---|
| StreetGS | 12/54(22.2%) | 53/54(98.1%) | 0.798 | 16.82 | 91.20 |
| StreetGS w/ TRELLIS | 33/54(61.1%) | 53/54(98.1%) | 2.115 | 16.81 | 94.68 |
| StreetGS w/ AH | 30/54(55.6%) | 54/54(100%) | 2.070 | 17.11 | 94.91 |
| OmniRe | 19/54(35.2%) | 53/54(98.1%) | 1.178 | 16.67 | 89.12 |
| OmniRe w/ TRELLIS | 32/54(59.3%) | 52/54(96.3%) | 2.139 | 16.72 | 93.87 |
| OmniRe w/ AH | 33/54(61.1%) | 53/54(98.1%) | 2.119 | 16.44 | 94.44 |
| RECAST (Ours) | 34/54(63.0%) | 54/54(100%) | 2.150 | 17.57 | 95.89 |
54 deterministic rollouts per method: 9 scenes × 3 motions × 2 distance–speed settings. NC: no-collision rate; DAC: drivable-area compliance; TTC: mean minimum predicted time-to-collision; Rc: route completion; COM: driving comfort. w/: native target actor replaced by a generated actor. AH: Asset Harvester.
Best
Scene-level evaluation
Scene-level rendering comparison.
Compare native actors, Asset Harvester, TRELLIS, and RECAST as actor motion exposes previously unseen views.
Scene 002 · Oncoming · StreetGS
Compare scene-level results
| Method | FDincep ↓ | CLIPmargin × 1000 ↑ | VQAmargin × 1000 ↑ |
|---|---|---|---|
| StreetGS | 129.35 | 0.14 | -294.91 |
| StreetGS w/ TRELLIS | 112.84 | 2.86 | -172.36 |
| StreetGS w/ AH | 110.00 | 1.95 | -156.42 |
| OmniRe | 138.52 | 0.04 | -314.18 |
| OmniRe w/ TRELLIS | 121.47 | 2.93 | -216.82 |
| OmniRe w/ AH | 119.10 | 1.96 | -197.11 |
| RECAST (Ours) | 112.10 | 3.47 | -131.04 |
| Method | FDincep ↓ | CLIPimg × 100 ↑ | CLIPveh × 100 ↑ | LPIPS ↓ |
|---|---|---|---|---|
| StreetGS | 193.56 | 65.43 | 20.62 | 0.6920 |
| StreetGS w/ TRELLIS | 126.14 | 69.54 | 21.28 | 0.6809 |
| StreetGS w/ AH | 124.61 | 70.17 | 21.03 | 0.6818 |
| OmniRe | 197.41 | 65.72 | 20.87 | 0.6931 |
| OmniRe w/ TRELLIS | 133.04 | 69.99 | 21.32 | 0.6824 |
| OmniRe w/ AH | 129.11 | 70.23 | 21.11 | 0.6823 |
| RECAST (Ours) | 124.74 | 70.67 | 21.43 | 0.6800 |
Controlled interactions on Waymo. Margin: mean positive-prompt score minus mean negative-prompt score. CLIPveh uses the prompt “vehicle.” Scaling is shown in the column headers. w/: native target actor replaced by a generated actor. AH: Asset Harvester.
Best · Second-best
Actor-level evaluation
One observation. A view-complete actor.
Compare generated vehicles across input observations and viewpoints, then inspect the complete held-out RECAR results.
Enlarge figure ↗Compare actor-level results on RECAR
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | CLIPimg × 100 ↑ | FDincep ↓ | FDdinov2 ↓ |
|---|---|---|---|---|---|---|
| Real3D | 13.19 | 0.5691 | 0.3223 | 68.17 | 91.196 | 1282.01 |
| LN3Diff | 18.95 | 0.8843 | 0.1105 | 63.03 | 93.288 | 1455.05 |
| 3DTopia-XL | 15.54 | 0.8085 | 0.1817 | 59.90 | 166.294 | 1299.59 |
| SAR3D | 15.39 | 0.8231 | 0.1588 | 66.55 | 64.042 | 746.99 |
| AH | 19.22 | 0.8792 | 0.1044 | 78.29 | 38.368 | 183.08 |
| TRELLIS | 21.02 | 0.9077 | 0.0603 | 78.22 | 9.788 | 43.89 |
| Ours w/o II | 22.90 | 0.9178 | 0.0534 | 78.30 | 8.176 | 36.94 |
| RECAST (Ours) | 22.90 | 0.9179 | 0.0532 | 78.38 | 7.992 | 36.47 |
AH: Asset Harvester. Ours w/o II uses vehicle-domain adaptation only, without driving-log-robust adaptation. All values are reported in the paper.
The paper
Abstract
Closed-loop driving simulation requires rendered observations to remain reliable as the ego vehicle and surrounding actors move beyond their recorded trajectories, exposing views absent from the source log. Existing data-driven simulators reconstruct dynamic actors from sparse observations, which can result in rendering artifacts under these viewpoint changes. We introduce RECAST (REconstructing Controllable Actors for Simulation and Testing), a 3D Gaussian Splatting framework that generates a view-complete actor from a single segmented vehicle observation in a driving log and registers the generated actor in the reconstructed scene. RECAST supports planner-in-the-loop rendering under controlled ego–actor motion. To adapt an image-to-3D prior to real vehicles, we further introduce RECAR, a dataset of approximately 20K real vehicles with 600K background-free RGBA images spanning diverse vehicle colors and types. We use two-stage adaptation to improve vehicle generation from real driving-log observations. At the actor level, RECAST reduces FDincep from 9.788 to 7.992 relative to unadapted TRELLIS. At the scene level, under actor motion beyond logged trajectories, RECAST reduces FDincep from 129.35 to 112.10 and increases CLIPmargin (×1000) from 0.14 to 3.47 relative to Street Gaussians. We demonstrate planner-in-the-loop simulation with the image-conditioned planner GTRS-Dense. Compared with native Street Gaussians actors, RECAST increases the no-collision (NC) rate from 22.2% (12/54) to 63.0% (34/54) and the mean minimum predicted time-to-collision (TTC) from 0.798 s to 2.150 s. These experiments show that RECAST supports closed-loop planner evaluation under controlled ego–actor interactions beyond log replay.