RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors

Anonymous Authors

The framework

From observation to interaction.

RECAST combines vehicle generation, scene registration, and planner-in-the-loop rendering.

RECAST framework: actor generation, scene registration and closed-loop simulationEnlarge figure ↗
RECAST framework. Two-stage adaptation produces view-complete vehicle actors; registration places them in a reconstructed scene for planner-in-the-loop simulation.
01 · Observe

Use a segmented vehicle image from a driving log.

02 · Generate

Adapt the image-to-3D prior in two stages.

03 · Register

Align and insert the actor into the reconstructed scene.

04 · Simulate

Render, plan, update, and repeat.

The dataset

RECAR Dataset

Background-free, multi-view imagery connects an image-to-3D prior with the diversity of real vehicles.

~20Kreal vehicles
~600KRGBA images
30views per vehicle on average
1200 × 800image resolution

Vehicle color and type distributions

Vehicle color and body-type distribution in RECAREnlarge figure ↗
Compare RECAR with existing vehicle datasets
Dataset comparison reported in the paper.
DatasetVehiclesTypeViewsResolutionBG-freeRegular
SRN-Car2.1KSynthetic250128 × 128YesYes
Objaverse-XL-Car11.4KSynthetic——YesYes
MVMC576Real10600 × 450NoNo
3DRealCar2.5KReal2001920 × 1440NoNo
MAD-CARS70KReal851920 × 1080NoNo
RECAR20KReal301200 × 800YesYes

Views: average number of views or video frames per vehicle. BG-free: background-free images are provided or can be rendered from supplied assets. Regular: viewpoints follow a predefined sampling pattern. —: not applicable.

Closed-loop evaluation

Closed-loop planner evaluation.

Planner-in-the-loop demonstrations with GTRS-Dense under controlled ego–actor interactions beyond log replay.

Scene 002 · Stationary-facing

22.2% → 63.0%
12/54 → 34/54 evaluated runs
0.798 s → 2.150 s
Predicted time-to-collision
Compare closed-loop results
Table II · Image-conditioned closed-loop evaluation with GTRS-Dense
MethodNC (%) ↑DAC (%) ↑TTC (s) ↑Rc (%) ↑COM (%) ↑
StreetGS12/54(22.2%)53/54(98.1%)0.79816.8291.20
StreetGS w/ TRELLIS33/54(61.1%)53/54(98.1%)2.11516.8194.68
StreetGS w/ AH30/54(55.6%)54/54(100%)2.07017.1194.91
OmniRe19/54(35.2%)53/54(98.1%)1.17816.6789.12
OmniRe w/ TRELLIS32/54(59.3%)52/54(96.3%)2.13916.7293.87
OmniRe w/ AH33/54(61.1%)53/54(98.1%)2.11916.4494.44
RECAST (Ours)34/54(63.0%)54/54(100%)2.15017.5795.89

54 deterministic rollouts per method: 9 scenes × 3 motions × 2 distance–speed settings. NC: no-collision rate; DAC: drivable-area compliance; TTC: mean minimum predicted time-to-collision; Rc: route completion; COM: driving comfort. w/: native target actor replaced by a generated actor. AH: Asset Harvester.

Best

Scene-level evaluation

Scene-level rendering comparison.

Compare native actors, Asset Harvester, TRELLIS, and RECAST as actor motion exposes previously unseen views.

Scene 002 · Oncoming · StreetGS

Compare scene-level results
Table III · Scene quality
MethodFDincep ↓CLIPmargin × 1000 ↑VQAmargin × 1000 ↑
StreetGS129.350.14-294.91
StreetGS w/ TRELLIS112.842.86-172.36
StreetGS w/ AH110.001.95-156.42
OmniRe138.520.04-314.18
OmniRe w/ TRELLIS121.472.93-216.82
OmniRe w/ AH119.101.96-197.11
RECAST (Ours)112.103.47-131.04
Table III · Actor-crop quality
MethodFDincep ↓CLIPimg × 100 ↑CLIPveh × 100 ↑LPIPS ↓
StreetGS193.5665.4320.620.6920
StreetGS w/ TRELLIS126.1469.5421.280.6809
StreetGS w/ AH124.6170.1721.030.6818
OmniRe197.4165.7220.870.6931
OmniRe w/ TRELLIS133.0469.9921.320.6824
OmniRe w/ AH129.1170.2321.110.6823
RECAST (Ours)124.7470.6721.430.6800

Controlled interactions on Waymo. Margin: mean positive-prompt score minus mean negative-prompt score. CLIPveh uses the prompt “vehicle.” Scaling is shown in the column headers. w/: native target actor replaced by a generated actor. AH: Asset Harvester.

Best · Second-best

Actor-level evaluation

One observation. A view-complete actor.

Compare generated vehicles across input observations and viewpoints, then inspect the complete held-out RECAR results.

Vehicle generation comparison: input, LN3Diff, Real3D, 3DTopia-XL, SAR3D, AH, TRELLIS and RECAST variantsEnlarge figure ↗
Qualitative actor comparisons from the paper. Each column starts with one input vehicle observation; the rows compare generated views across methods. Enlarge to inspect details.
Compare actor-level results on RECAR
Actor-level evaluation on held-out RECAR. ↑ Higher is better; ↓ lower is better.
MethodPSNR ↑SSIM ↑LPIPS ↓CLIPimg × 100 ↑FDincep ↓FDdinov2 ↓
Real3D13.190.56910.322368.1791.1961282.01
LN3Diff18.950.88430.110563.0393.2881455.05
3DTopia-XL15.540.80850.181759.90166.2941299.59
SAR3D15.390.82310.158866.5564.042746.99
AH19.220.87920.104478.2938.368183.08
TRELLIS21.020.90770.060378.229.78843.89
Ours w/o II22.900.91780.053478.308.17636.94
RECAST (Ours)22.900.91790.053278.387.99236.47

AH: Asset Harvester. Ours w/o II uses vehicle-domain adaptation only, without driving-log-robust adaptation. All values are reported in the paper.

The paper

Abstract

Closed-loop driving simulation requires rendered observations to remain reliable as the ego vehicle and surrounding actors move beyond their recorded trajectories, exposing views absent from the source log. Existing data-driven simulators reconstruct dynamic actors from sparse observations, which can result in rendering artifacts under these viewpoint changes. We introduce RECAST (REconstructing Controllable Actors for Simulation and Testing), a 3D Gaussian Splatting framework that generates a view-complete actor from a single segmented vehicle observation in a driving log and registers the generated actor in the reconstructed scene. RECAST supports planner-in-the-loop rendering under controlled ego–actor motion. To adapt an image-to-3D prior to real vehicles, we further introduce RECAR, a dataset of approximately 20K real vehicles with 600K background-free RGBA images spanning diverse vehicle colors and types. We use two-stage adaptation to improve vehicle generation from real driving-log observations. At the actor level, RECAST reduces FDincep from 9.788 to 7.992 relative to unadapted TRELLIS. At the scene level, under actor motion beyond logged trajectories, RECAST reduces FDincep from 129.35 to 112.10 and increases CLIPmargin (×1000) from 0.14 to 3.47 relative to Street Gaussians. We demonstrate planner-in-the-loop simulation with the image-conditioned planner GTRS-Dense. Compared with native Street Gaussians actors, RECAST increases the no-collision (NC) rate from 22.2% (12/54) to 63.0% (34/54) and the mean minimum predicted time-to-collision (TTC) from 0.798 s to 2.150 s. These experiments show that RECAST supports closed-loop planner evaluation under controlled ego–actor interactions beyond log replay.

Full-resolution figure · Scroll to inspect