sceneforge results

/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap
← all runs

lab_colmap

session /ihub/homedirs/svs_ald/sudhir/real2sim/captures/session_20260901_180746
work /ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap
Lab scene, 8-frame test session (2026-09-01). COLMAP --thorough poses. This session is a plumbing test, not a real capture: the 8 frames were shot as separate viewpoints rather than a continuous sweep. COLMAP still registered all 8 into one connected model -- they share the stool and the objects on it. What is real here: * metric scale cross-checks against VGGT to 4.3% (1.787 m vs 1.867 m baseline) * SAM3 segmentation is fully automatic: robot 0.69-0.76, stool 0.98-0.99 * the scene compiles and simulates in MuJoCo * at the capture poses the reconstruction is near-photographic (PSNR ~26) What is not: * held-out PSNR is 8.4 -- between the capture poses the splat has nothing. See lab_colmap_path.mp4 for what that looks like. * object reconstruction is pose-limited. The green cup first came out 146 cm across; keeping the largest DBSCAN cluster brought it to 6.3 cm, which is a workaround for pose error, not a fix. * ground plane is suspect: objects land at z = 1.24-1.37 m, which does not square with a 0.34 m stool. Re-check against a hand-measured height.

Video

lab_colmap_compare.jpg0.2 MB ↓
lab_colmap_compare.mp42.0 MB ↓
lab_colmap_holds.jpg0.1 MB ↓
lab_colmap_holds.mp41.0 MB ↓
lab_colmap_path.jpg0.1 MB ↓
lab_colmap_path.mp43.9 MB ↓
lab_colmap_robot_masks.png1.2 MB ↓
lab_colmap_sim_only.jpg0.0 MB ↓
lab_colmap_sim_only.mp40.1 MB ↓
lab_colmap_splat_only.jpg0.1 MB ↓
lab_colmap_splat_only.mp40.9 MB ↓
lab_colmap_verify_frame003.png1.1 MB ↓

Stages

10_pose — Pose

Camera poses from unposed images (VGGT / COLMAP / GLOMAP).

How to read it. `registered` should equal `requested`. `n_models` > 1 means the capture split into disconnected pieces -- not enough overlap. A near-zero baseline means the camera barely moved.
backendcolmap
views8
baseline_recon_units8.035
points862
has_depthFalse
seconds9.200
matcherexhaustive
registered8
requested8
n_models1
split_warningFalse
sparse_points862
raw json
{
  "backend": "colmap",
  "views": 8,
  "baseline_recon_units": 8.0349,
  "points": 862,
  "has_depth": false,
  "seconds": 9.2,
  "matcher": "exhaustive",
  "registered": 8,
  "requested": 8,
  "n_models": 1,
  "split_warning": false,
  "sparse_points": 862
}

20_scale — Metric scale

Converts the scale-free reconstruction to metres using the D455 depth map. This is what replaces Re3Sim's ArUco marker.

How to read it. `spread` is agreement between independent per-frame estimates: <5% consistent, 5-15% loose, >15% do not trust. `baseline_m` should match the real extent of the capture.
backendcolmap
methodsparse_points
scale_m_per_unit0.222
spread0.022
verdictconsistent
frames_used1
frames_total1
baseline_m1.787
raw json
{
  "backend": "colmap",
  "method": "sparse_points",
  "scale_m_per_unit": 0.22240982592029473,
  "spread": 0.022467145446276666,
  "verdict": "consistent",
  "frames_used": 1,
  "frames_total": 1,
  "baseline_m": 1.7870425486009325,
  "per_frame": [
    {
      "scale": 0.22240982592029473,
      "median_scale": 0.22287464803591286,
      "n": 3438,
      "rel_mad": 0.022467145446276666,
      "trusted": true,
      "method": "sparse_points"
    }
  ]
}

30_ground — Ground plane / world frame

Fits the table plane and builds a world frame with +Z up and the surface at z=0, so MuJoCo gravity and object placement mean something.

How to read it. `plane.rms_m` under ~10 mm is a clean fit. Check `camera_height_range_m` against reality -- if the cameras are not plausibly above the table, the fitted plane is not the table.
scale_m_per_unit0.222
plane
offset-1.333
inlier_frac0.121
rms_m0.008
raw json
{
  "T_world_recon": [
    [
      0.0,
      0.14803555823500386,
      -0.16598675900187515,
      0.0
    ],
    [
      0.22194283461616937,
      0.010750725279419799,
      0.00958805164785523,
      0.0
    ],
    [
      0.01440516672709814,
      -0.16563823854989834,
      -0.14772472970883135,
      1.3331918243190537
    ],
    [
      0.0,
      0.0,
      0.0,
      1.0
    ]
  ],
  "T_rot_only": [
    [
      0.0,
      0.6655981030624768,
      -0.7463103678762827,
      0.0
    ],
    [
      0.9979003117232207,
      0.04833745647223585,
      0.04310983837238967,
      0.0
    ],
    [
      0.06476857156599068,
      -0.7447433487460141,
      -0.6642005545284301,
      1.3331918243190537
    ],
    [
      0.0,
      0.0,
      0.0,
      1.0
    ]
  ],
  "scale_m_per_unit": 0.22240982592029473,
  "plane": {
    "normal": [
      -0.06476857156599068,
      0.7447433487460141,
      0.6642005545284301
    ],
    "offset": -1.3331918243190537,
    "inlier_frac": 0.12093624473570823,
    "rms_m": 0.008349812442451915
  },
  "camera_height_range_m": [
    1.267853035438392,
    1.518344919894524
  ],
  "warnings": []
}

40_mask — Robot mask

SAM3 text-prompted segmentation of the robot, which must be kept out of the background splat (MuJoCo renders it from the URDF).

How to read it. `empty_masks` should be 0. `median_coverage` above ~50% means the prompt matched the whole scene rather than the robot.
promptrobotic arm, robot
frames8
median_coverage0.058
empty_masks0
raw json
{
  "prompt": "robotic arm, robot",
  "frames": 8,
  "median_coverage": 0.05776513671875,
  "empty_masks": 0,
  "warnings": []
}
000000.png0.0 MB ↓
000001.png0.0 MB ↓
000002.png0.0 MB ↓
000003.png0.0 MB ↓
000004.png0.0 MB ↓
000005.png0.0 MB ↓
000006.png0.0 MB ↓
000007.png0.0 MB ↓

50_splat — Background splat

Gaussian splat trained on the masked images, initialised from the measured depth cloud rather than sparse SfM points.

How to read it. `eval` is on HELD-OUT views -- the honest number. Training-view PSNR only measures overfitting. `eval.depth.bias_mm` is the early warning for a wrong scale factor.
backendgsplat
ply/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/50_splat/background.ply
n_gaussians600,000
train_seconds347.100
peak_vram_gb1.100
iters7,000
cap_max600,000
sh_degree3
init_points300,000
train_views7
maskedTrue
eval
n_views1
lpips0.783
psnr8.599
ssim0.294
eval.depth
bias_mm-634.476
inlier_frac_10mm0.005
mean_abs_mm741.475
median_abs_mm571.079
n182,403.000
p95_abs_mm1,866.799
rms_mm963.233
valid_frac0.713
raw json
{
  "backend": "gsplat",
  "ply": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/50_splat/background.ply",
  "n_gaussians": 600000,
  "train_seconds": 347.1,
  "peak_vram_gb": 1.1,
  "eval": {
    "n_views": 1,
    "lpips": 0.782911479473114,
    "psnr": 8.599191358385454,
    "ssim": 0.2944691777229309,
    "depth": {
      "bias_mm": -634.4758885846951,
      "inlier_frac_10mm": 0.005389165748370367,
      "mean_abs_mm": 741.4746807016857,
      "median_abs_mm": 571.078896522522,
      "n": 182403.0,
      "p95_abs_mm": 1866.7991518974302,
      "rms_mm": 963.233157013922,
      "valid_frac": 0.71251171875
    }
  },
  "iters": 7000,
  "cap_max": 600000,
  "sh_degree": 3,
  "init_points": 300000,
  "train_views": 7,
  "masked": true
}

60_mesh — Static geometry

TSDF fusion of the depth maps into a mesh with real measured thickness, plus convex parts for collision.

How to read it. `extent_m` must match the real object. MuJoCo hulls a mesh whole, so the convex parts are what make concave shapes collide right.
visual_mesh/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/static.obj
views_fused8
promptwooden stool, wooden table
voxel_m0.005
vertices2,462
faces2,570
raw json
{
  "visual_mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/static.obj",
  "views_fused": 8,
  "prompt": "wooden stool, wooden table",
  "voxel_m": 0.005,
  "vertices": 2462,
  "faces": 2570,
  "extent_m": [
    1.409764253259149,
    1.1572487849828283,
    0.33672944038106034
  ],
  "collision_parts": [
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_000.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_001.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_002.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_003.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_004.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_005.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_006.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_007.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_008.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_009.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_010.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_011.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_012.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_013.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_014.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_015.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_016.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_017.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_018.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_019.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_020.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_021.obj",
    "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/60_mesh/collision/part_022.obj"
  ]
}
000000.png0.0 MB ↓
000001.png0.0 MB ↓
000002.png0.0 MB ↓
000003.png0.0 MB ↓
000004.png0.0 MB ↓
000005.png0.0 MB ↓
000006.png0.0 MB ↓
000007.png0.0 MB ↓

70_objects — Objects

Per-object mesh and physics. Mass is hand-tuned; inertia follows from the mesh at that mass.

How to read it. `extents_m` is the reality check. A 4 mm slab means one-sided observation (the RoboSnap failure); anything over ~60 cm means the views are not aligning and the poses are too imprecise.
objectsize (cm)masspartswatertight
green cup145.9 x 145.9 x 92.90.15 kg16True
small bottle143.0 x 139.5 x 135.50.3 kg16True
raw json
{
  "objects": [
    {
      "name": "green cup",
      "slug": "green_cup",
      "mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/mesh.obj",
      "collision_parts": [
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_000.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_001.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_002.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_003.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_004.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_005.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_006.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_007.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_008.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_009.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_010.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_011.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_012.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_013.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_014.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/green_cup/collision/part_015.obj"
      ],
      "position": [
        -0.1133618750830095,
        0.10450628750081967,
        1.1326153704390265
      ],
      "mass_kg": 0.15,
      "friction": 1.0,
      "com": [
        -0.081256882964619,
        0.0012264559794668258,
        0.059600058657337716
      ],
      "inertia": [
        [
          0.028674448002942284,
          0.001589896355387967,
          -0.0020612609676132513
        ],
        [
          0.001589896355387967,
          0.029426769568551406,
          0.001622694868281238
        ],
        [
          -0.0020612609676132513,
          0.001622694868281238,
          0.04562839181565874
        ]
      ],
      "extents_m": [
        1.45913,
        1.459131,
        0.928842
      ],
      "watertight": true,
      "n_points": 40306,
      "mode": "in_scene"
    },
    {
      "name": "small bottle",
      "slug": "small_bottle",
      "mesh": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/mesh.obj",
      "collision_parts": [
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_000.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_001.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_002.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_003.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_004.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_005.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_006.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_007.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_008.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_009.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_010.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_011.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_012.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_013.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_014.obj",
        "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/70_objects/small_bottle/collision/part_015.obj"
      ],
      "position": [
        -0.09752691947383645,
        0.11892121780883465,
        1.1845283782204157
      ],
      "mass_kg": 0.3,
      "friction": 1.0,
      "com": [
        -0.07115752676084439,
        0.0029275889937453287,
        -0.04112076495845971
      ],
      "inertia": [
        [
          0.062393293284257086,
          0.0035591868349627858,
          0.005802523472895785
        ],
        [
          0.0035591868349627858,
          0.0705222759625172,
          -0.005830588301923358
        ],
        [
          0.005802523472895785,
          -0.005830588301923358,
          0.0652283177069003
        ]
      ],
      "extents_m": [
        1.429616,
        1.395416,
        1.35504
      ],
      "watertight": true,
      "n_points": 10623,
      "mode": "in_scene"
    }
  ]
}
000000.png0.0 MB ↓
000000.png0.0 MB ↓
000001.png0.0 MB ↓
000001.png0.0 MB ↓
000002.png0.0 MB ↓
000002.png0.0 MB ↓
000003.png0.0 MB ↓
000003.png0.0 MB ↓
000004.png0.0 MB ↓
000004.png0.0 MB ↓
000005.png0.0 MB ↓
000005.png0.0 MB ↓

+4 more files in 70_objects/

80_scene — MuJoCo scene

Assembles the MJCF: static geometry, objects, cameras at the real capture poses, robot.

How to read it. `compiles` must be true. The splat is deliberately absent -- MuJoCo cannot render gaussians; it is composited in stage 90.
xml/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/80_scene/scene.xml
bodies4
geoms59
nq14
cameras8
objects2
collision_parts23
compilesTrue
raw json
{
  "xml": "/ihub/homedirs/svs_ald/sudhir/real2sim/work/lab_colmap/80_scene/scene.xml",
  "bodies": 4,
  "geoms": 59,
  "nq": 14,
  "cameras": 8,
  "objects": 2,
  "collision_parts": 23,
  "compiles": true
}

90_verify — Verification vs real photographs

Renders the sim, composites the splat behind it, and diffs against the actual photograph from that pose. The end-to-end check.

How to read it. This catches the silent failures: wrong scale, rotated background, objects floating. Compare held-out views against training views -- a large gap means the capture was too sparse.
views3
splatTrue
summary
frame3.000
lpips0.333
psnr20.044
sim_pixel_frac0.026
ssim0.720
raw json
{
  "summary": {
    "frame": 3.0,
    "lpips": 0.33285081883271533,
    "psnr": 20.04391981195853,
    "sim_pixel_frac": 0.02561197916666667,
    "ssim": 0.7203467090924581
  },
  "per_view": [
    {
      "psnr": 8.427669210273294,
      "ssim": 0.25233685970306396,
      "lpips": 0.7842886447906494,
      "frame": 0,
      "sim_pixel_frac": 0.02056640625
    },
    {
      "psnr": 26.34230117570805,
      "ssim": 0.9556958675384521,
      "lpips": 0.05215443670749664,
      "frame": 3,
      "sim_pixel_frac": 0.02694921875
    },
    {
      "psnr": 25.361789049894252,
      "ssim": 0.9530074000358582,
      "lpips": 0.162109375,
      "frame": 6,
      "sim_pixel_frac": 0.0293203125
    }
  ],
  "views": 3,
  "splat": true
}
000000.png1.2 MB ↓
000003.png1.1 MB ↓
000006.png0.7 MB ↓