The sam3dbody backend reads its options from the request's prompt string
(`hands`, `detect`, `persons=N`): `hands` runs the full mode and the
packet carries which hands were fused and their boxes; `detect` finds up
to N persons with SAM 3.1 (an optional native-segment role on the body
entry, the same artifact the segment entry pins) and runs one body pass
per person with its box and mask, so the packet's people array grows.
The body crate shares one body pass between the packet, mask and hands
entry points, and infer_full takes the mask prompt too.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
BodyModel::infer_masked takes a provided person box and its full-frame 0/1
mask: the mask is warped with the crop's own affine (bilinear, rounded back
to 0/1 like the reference's uint8 warp), encoded by the prompt encoder's
mask CNN (four stride-2 convolutions with channel LayerNorm and erf GELU,
then a 1x1 to 1280) and added to the backbone tokens before the ray
conditioning, replacing the folded no-mask term. Oracle parity on the
mask fixture: warp exact, every CNN stage within f32 accumulation noise,
conditioned context 8e-4, end to end kp3d 2.2 mm / kp2d 0.25 px. 41 tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
BodyModel::infer_full runs the reference's full mode natively: the body
pass's hand boxes become two 512 crops (the left one cut from the mirrored
image, padding 0.9), each goes through the backbone, its own ray
conditioning and the hand decoder (the body decoder with the *_hand
tensors), the hand-mode rig pre-transform (local_to_world_wrist, wrist and
root offsets, non-hand parameters zeroed) and the hand camera head's
scale factor of 10; the left result is un-mirrored. The fusion gates each
hand (wrist angle, box size, keypoint spread, wrist distance), re-prompts
the body decoder with the trusted wrists and elbows as point prompts (the
decoder now takes N prompt tokens and a previous estimate), and writes the
fused wrist angles, hand parameters and hand scale/shape back before the
final rig pass. pose.rs gains the roma xyz (extrinsic) and XZY (intrinsic)
euler pairs; mhr.rs returns joint global rotations.
Oracle parity on both full-mode fixtures: every hand-decoder stage per
step (tokens 7e-3, heads 4e-4, rig params 1e-4), fusion reports as the
reference (one hand trusted on the standing photo, both on the crop), and
end to end kp3d 1.6 mm / kp2d 0.85 px. 36 tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One backend call per SAM-style decoder layer (PE norms, token
self-attention, token-to-image cross-attention, erf-GELU feed-forward,
final norm) inside one command buffer, with the layer's f32 weights cached
on the device under their content identity and pooled transients.
gpu_two_way_layer_resident in the common backend; CUDA declines and the
per-op path stays. Body decoder loop on the M3 Max: layers 100 -> 9 ms,
frame 405 -> 266 ms, oracle parity unchanged (kp3d 1.5 mm).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Metal tensor backend is host-Vec based: every op copied its inputs up
and its result back, which put the DINOv3 backbone at 2.1 s a frame on an
M3 Max. try_vit_backbone_resident_f32 runs a whole pre-norm ViT stack
(LayerNorm, q/k/v, rotate-half rope from tables, flash attention, out,
LayerNorm, SwiGLU, down; residuals in place) inside one command buffer
against cached bf16 weights, with a new kernel_makepad_rope_half_tables_f32
kernel for the rope. The precompiled metallib now carries the bf16 GEMM
kernels the runtime source compile already enabled on bfloat devices.
The common backend exposes it as gpu_vit_backbone_resident (CUDA declines;
its per-op path is already resident) and the body backbone tries it first:
233 ms a frame on the M3 Max, same oracle parity (kp3d 1.5 mm).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The FP8 backbone mode added each linear's bias by downloading, tiling and
re-uploading it per call, which cost more than the FP8 GEMM saved. A
gpu_add_cols_broadcast op (CUDA kernel; host loop on the Metal tensor
backend) adds a cols-wide bias to every row on the device, and each linear
keeps its bias resident after the first upload.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
BodyModel::set_backbone_fp8 quantises every backbone weight to E4M3 with
a per-tensor absmax scale and runs the tensor-core FP8 GEMM (bias
broadcast after). A backend without FP8 turns it off per layer on the
first refusal, so Metal keeps bf16. The oracle test reports its
accuracy and timing next to bf16; the default stays bf16 until the
numbers say otherwise.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The backbone's linears now take the cuBLASLt bias-epilogue / bf16 mm
paths (bf16 operands, f32 accumulation, bf16 output — the reference's
precision) with the f32-accumulating GEMM as the fallback. The rig's
pose correctives run on the final refinement step only by default: the
intermediate steps only feed keypoints back into the decoder, and the
oracle shows the difference is 0.4 mm (2.1 vs 1.7 mm against the
reference) for half the loop's rig time; `correctives_every_step` keeps
the exact mode.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The backbone, conditioning and decoder now take any square crop whose
side is a multiple of 16 (512 is the trained size); BodyModel::set_crop_size
selects it and caches the dense positional grid per size. Measured on the
oracle image against the reference: 512 gives 1.7 mm mean keypoint error,
384 and 256 about 2 cm mean (6 to 7 cm worst joint), 192 falls apart —
the knob is a real accuracy trade, not free speed. The decoder loop
reports its split (layer chain, heads, rig+camera, refinement) so the
next optimisation is chosen on numbers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The backbone (20 heads) and the decoder (8 heads) both have 64-wide
heads, which the stack's FA2 kernel covers: f16 operands with f32
softmax and accumulation, the reference's own precision class, instead
of the composite path that materialises the 1029 x 1029 scores per head.
The composite path stays as the fallback where a backend lacks the
kernel. Oracle parity unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The ray-conditioning conv used to download the 1024 x 1280 embedding,
concatenate the 99 ray features on the host and upload the result; it is
now two resident linears over the two column blocks with the no-mask
term folded into the bias, plus an add (32 ms -> a few on the 4090).
The decoder loop downloaded the whole normalised token block every
layer for its one pose row; it now slices that row and fetches the
block once at the end (or per layer under a trace). Same numbers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On the 4090 the frame was 885 ms of which 840 were host scalar loops in
the decoder loop: the three 70-row refinement FFNs (about 240M
multiply-adds a step), the two heads, and the rig's 55k x 3000
corrective output layer plus its identity blendshape sum. Those are now
GPU-resident linears (heads.rs GpuStepHeads, MhrRig::prepare_gpu), the
skinning computes one transform per joint instead of one per influence,
and the results are identical (all 29 oracle tests unchanged). The rig
still runs entirely on the host when no GPU side was prepared.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
model.rs closes the loop: crop -> backbone -> ray-conditioned context ->
six decoder steps, each through the pose head, the rig, the camera and
the projection, then the packet the sandbox already reads (kp3d/kp2d in
camera axes, the 204 rig parameters, global rotation, camera translation,
joint positions). Against the reference on the oracle image, on Metal:
3D keypoints within 1.7 mm, 2D within 0.4 px, rig parameters, camera
and rotation within 2e-3. packet.rs writes the JSON by hand with the
reference worker's rounding and field order.
The hub gains the `body-native` feature (default on): registry entry
`sam3dbody` pinned to the Comfy-Org repack by revision, size and sha,
body_native_backend.rs beside the subprocess reference backend with the
same live_step contract, the `body` capability advertised when the
feature is compiled, and a stubbed test double for the CPU-only tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
decoder.rs builds the 145-row token set (pose, previous, prompt, two
hand-box rows, 70 keypoint rows, 70 3D-keypoint rows), runs the six
layers with the SAM-style repeated positional encoding (none on the
first layer's self-attention), cross-attends against the ray-conditioned
context, and after each step hands the normalised pose token to the
heads and updates the keypoint rows from the caller's feedback: the
2D-keypoint positional FFN and the bilinearly sampled context features
on the valid rows, the pelvis-centred 3D positional FFN on the rest.
heads.rs holds the host-side ReLU FFN heads, the refinement FFNs, the
hand-box MLP and the hand classifier. Against the oracle on Metal, every
layer's residual stream is within 6e-3, every step's pose head within
8e-4 and camera head within 2e-5; token assembly and the host heads
match to 1e-4.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
dino.rs is the 32-block ViT-H+/16 on the gpu_* surface after the TRELLIS
conditioner: bf16-resident linears with f32 accumulation, layer scale
folded into the output projections, rotate-half rope, SwiGLU, the
model's own final norm. preprocess.rs is the crop (box -> 1.25 pad ->
3:4 -> square -> 512 bilinear warp, ImageNet normalisation), the CLIFF
condition vector and the patch rays; condition.rs is the dense
positional encoding and the ray-conditioned decoder context. Against the
reference oracle on Metal: backbone 1.1% mean relative (bf16 noise),
ray-conditioned context 8e-5, crop within one u8 rounding step, rays and
dense PE 1e-7. One reference detail the paper does not state: its shrink
of the ray field is an antialiased filter whose taps clip at the image
edge, so the two edge patches sample inside their block centre (9.03 and
501.97 rather than 7.5 and 503.5); block centres left 0.1 of error.
The Metal tensor backend was missing rope_half (it aliased the
interleaved layout) and layer_norm_mul_add; both now exist with the
CUDA contract, which is what lets this backbone run on Apple silicon.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mhr.rs is the Momentum Human Rig on the CPU: blendshapes, the 889x249
parameter transform, parents-first similarity kinematics (x,y,z,w
quaternions, Rz Ry Rx eulers, 2^s scales), the sparse-then-dense pose
corrective MLP over joints 2.., linear blend skinning and the 308-row
keypoint regression. pose.rs decodes the 519-wide head output: 6d
rotations, the 23 ball / 58 hinge / 6 translation body layout, the
mixed-dof hand layout, scale and hand component bases, the camera
translation and the perspective projection. Against the reference
oracle: vertices within 1e-4 cm with correctives on, keypoints within
1e-6 m, rig parameters within 1e-7. Two conventions the spec could not
settle on paper are now settled by the fixture: the head's global
rotation triple arrives Z,Y,X-ordered, and the corrective features start
at joint 2 (750 wide). Fixture tests skip cleanly without the oracle
directory or the weights.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
makepad-ai-body joins the AI model workspace: the constants of the
architecture (DINOv3 ViT-H+/16 at 512, the 6-layer promptable decoder,
the 519-wide pose head, the 127-joint / 18439-vertex MHR rig) and the
single-file safetensors reader for the Comfy-Org repack, which fails
closed on a Meta checkpoint-style header and checks the shapes the port
is written for at load. The backbone, decoder and rig modules follow
in their own lanes against the spec under local/agent_state/sam3dbody.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
