Human-hand WAM datasets

See the data

Each video: rgb, depth, and two hand images below, with the caption bar at the bottom. Synthetic = all sim. Real-paired = real video with sim replays.

How it was made

Code: onticlabs/real2sim, branch human-wam/prod. The README there maps each step to its code.

1. 1

   Replay the recording in MuJoCo

   Real hands (MANO) and objects become a physics twin. Below: real video beside sim replays. Preview round.

2. 2

   Make failures through physics

   Same start, same instruction. Friction, grip or mass changes make it fail. Left: failure. Right: the success it branched from.

3. 3

   Recover from the failure

   Some failures continue: the hand re-grasps and finishes the task.

4. 4

   Render a synthetic room

   Random room and light; the frame-0 scene matches the success twin. Preview round.

5. 5

   Draw the action images

   act = hands + intended target. act_fk = hands only. Bottom panels.

6. 6

   Depth for real-paired video

   The sim has no arm or body. Sim depth where it matches, MoGe-2 on person and room. Green panel = loss weight.

7. 7

   Encode short and long

   Wan2.2 VAE latents. Shorts are 81-frame windows. Long points follow a whole episode (10 s shown).

Then five independent validators checked the frozen data: all PASS, two after fixes, 133 items dropped. After the depth fix, all 2,633 real-paired items changed only in depth and its weight.

Pointmaps

A pointmap stores the 3D position of every pixel, so a model can read geometry directly instead of combining depth and camera.

Drag to orbit, scroll to zoom. Exact sim depth lifted to 3D, coloured by the decoded video, every 2nd frame. Shown in world coordinates so the scene stays still; the pointmap stream stores XYZ per camera.

Pilot result: after the VAE round trip, pointmaps are about 2× worse than depth + camera on hand pixels in synthetic groups (median 4–37 mm vs 2–17 mm). Real-paired groups are about equal. Details in docs/PTS_PILOT_REPORT.md on the code branch.

Which to use

Use it

# Project world-models, tag human-wam. On Prometheus, prefix ontic with
# PYTHON_KEYRING_BACKEND=keyring.backends.fail.Keyring or it hangs.
J=de0238f4-cc54-493f-9517-6c212c2ece45     # human_ship_v1, 74.7 GiB
ontic pull $J --path data/_index           # the index (107 MB)
ontic pull $J --path data/FLAGS.txt        # trainer flags for this tree
ontic pull $J --path data/<group>/<episode>/<file>.npz   # one window
ontic pull $J --path data/_text            # 6.2 GB caption embeddings, for training only
git clone -b won/wam-mix5-v7-human git@github.com:onticlabs/experiments.git

# python snippet.py   (needs numpy + torch; run where you pulled the files)
import json, os, sys

sys.path.insert(0, "experiments/projects/world-models/2026-11-09_wam-mix5-v7-human/wam_src/mix5")
import data_mix5 as D

root = "output-pulls/de0238f4-cc54-493f-9517-6c212c2ece45/data"
rows = [json.loads(l) for l in open(f"{root}/_index/index.jsonl")]

for r in rows:
    files = [f"{root}/{r['group']}/{s['file']}" for s in r["slots"].values()]
    if not all(os.path.exists(f) for f in files):
        continue                               # only the windows you pulled
    w = D.load_window(root, r)                 # picks sim_rand0 or sim_rand1 at random
    lat = w["lat"][next(iter(w["lat"]))]       # slot "single" (static exo) or "moving" (ego)
    print(r["group"], r["kind"], r["outcome"], r["view"], r["captions_plain"][0])
    print({k: tuple(v.shape) for k, v in lat.items()})
    # real-paired: {'rgb': (48, 21, 24, 32), 'sim': (48, 21, 12, 16), 'depth': (48, 21, 12, 16),
    #               'act': ..., 'cam': (48, 21, 6, 8), 'act_fk': ..., 'depth_w': (1, 21, 12, 16)}

Run on five windows, one per group, pulled from the store on 2026-11-09. In a launch, add {job: de0238f4-cc54-493f-9517-6c212c2ece45, source: output/data} to r3.yaml. Names human-ship-v1 and human-long-v1 work once PR #390 merges.

Train

• Synth groups draw c1, c3, c4 and c5 at 25 % each. Real-paired groups draw all five at 20 %.
• act vs act_fk: the current run (issue #387) uses hands only with --act-fk-groups, both given and generated.
• depth_w: applied per cell for real-paired groups; no extra real-depth factor (real_depth_w: 1.0).
• Start: full state of v7-wan step 15,000. emb5 grows one row for human hands, starting at zero.

Limits

• About 40 % of HOT3D real frames count as bad, so person and room depth has weight 0 there. The values are correct; the loss ignores them. Mean depth_w: HOT3D 0.33, HO-Cap 0.68.
• Short windows that reuse a long point match it in every stream except act, by design: each window draws only its own captioned moves.
• 20 reuse shorts, all hot3d_real_h, have their long point in _dropped (job b2b153a9).
• Thin depth specks at the sim / MoGe seam on some real-paired samples (right).
• The long set is 2.49 M frames in 261,266 small files, so reads are slow. Pull only the points you need.
• No human pointmap dataset yet: pilot only. Robots have one.

Licences

Commercial use is unresolved. MANO applies to all five groups and is non-commercial. Ask Won before any use beyond internal research.

Sources read 2026-11-09; this is not legal advice.

Other human data