NeurIPS 2026

Action-conditioned surgical video generation

From Articulated Kinematics to Routed Visual Control

Bohan Li1,2,4 Shuojue Yang2 Baorui Peng4 Xianda Guo5 Erli Zhang2 Youqi Tao6 Junfeng Duan1 Daguang Xu7 Qi Dou8 Xin Jin4 Wenjun Zeng4 Hao Zhao3 Yueming Jin2
1SJTU 2NUS 3THU 4EIT 5WHU 6Harvard 7NVIDIA 8CUHK

KVLR turns sparse surgical robot actions into image-aligned visual control fields, then routes conditioning across physical modalities and motion scales for faithful, realistic, and efficient video synthesis.

Paper PDF Demo Video Code Citation
5 image-aligned control modalities
26.6% Chamfer Distance decrease over strong baselines
38.4% FID decrease with structured visual routing
47.1% lower latency for KVLR-fast at 288 x 512
KVLR teaser showing kinematic-to-visual lifting, hierarchical routing, performance gains, and KASA annotation examples.
KVLR lifts articulated kinematics into visual control and uses hierarchical routing to activate the most relevant modality and motion-scale pathways.

Abstract

Kinematic control that lands in image space

Action-conditioned surgical video generation asks a model to synthesize realistic visual outcomes from desired robotic actions. The hard part is the representational gap: a compact articulated control vector must govern detailed tool appearance, occlusion, tissue interaction, and temporal motion in image space.

KVLR introduces a kinematic-to-visual lifting paradigm that converts articulated actions into five interpretable, pixel-aligned control modalities: semantics, depth, rotation, velocity, and acceleration. A hierarchical routing framework then selects useful modalities and motion scales, while kinematic-prior losses keep routing physically meaningful, temporally stable, and efficient.

Representation

KVA-Field

A projected action field encodes where the tool is, how it is oriented, and how it is moving, using control signals that can be derived from articulated actions.

Routing

Two-tier control

Modality routing selects physical cues, while motion-scale routing specializes computation for fine manipulation, transport motion, and stable regions.

Efficiency

Action-adaptive execution

Routing significance exposes conditional sparsity, enabling KVLR-fast to skip or reuse low-significance pathways without discarding action alignment.

Benchmark

KASA

A curated benchmark pairs real robotic surgical videos with articulated action annotations for knotting, needle grasping, and needle puncture.

Demo

Action-conditioned synthesis in motion

The supplementary video highlights the central behavior of KVLR: generated frames remain visually plausible while following the supplied articulated surgical action trajectory.

Input
Reference frame, prompt, and 9D articulated action sequence.
Control basis
Part semantics, depth, local rotation, velocity, and acceleration.
Output
Action-faithful surgical video with adaptive control computation.

Method

Lifting first, routing second

Architecture overview of KVLR with KVA-Field lifting, modality routing, motion-scale routing, and adaptive efficient generation.
Articulated kinematics are lifted into a KVA-Field before hierarchical routing allocates control capacity across modalities and motion scales.
01

Project articulated kinematics

The shaft, wrist, and grippers are reconstructed as part-aware geometry and projected into the camera view.

02

Build image-aligned controls

Pixel-aligned channels encode part semantics, rendered depth, orientation, velocity, and acceleration.

03

Route conditional computation

Tier-1 routing chooses useful physical modalities; Tier-2 routing specializes updates by motion scale.

04

Exploit routing sparsity

KVLR-fast combines few-step distillation, spatial adaptive execution, and temporal cache reuse for low-latency synthesis.

KASA Benchmark

KASA action supervision

KASA data construction pipeline from surgical videos to semantic annotation and articulated pose tracking.
KASA combines human-in-the-loop semantic annotation with differentiable pose tracking to obtain lightweight, visually grounded action supervision.
1,047 surgical scenes
105,175 video frames
3 representative actions
55-235 frames per sequence

Results

Faithful control, stronger fidelity, faster variant

Action faithfulness CD 86.89

Full model in the architecture ablation, compared with 111.11 from raw-action direct conditioning.

Visual fidelity FID 14.37

Structured lifting plus two-tier routing improves over KVA-Field direct conditioning at 45.20.

Efficient generation 0.37 s

KVLR-fast latency per 17-frame clip at 288 x 512, with 0.55 PFLOPs per clip.

Architecture Ablation

Method CD TI FID PSNR Latency
Text-only baseline 138.88 0.57 93.31 12.80 4.59
Raw action direct condition 111.11 0.69 79.56 15.08 4.61
KVA-Field direct condition 96.45 0.78 45.20 17.65 4.68
KVA-Field with Tier-1 routing 91.30 0.84 26.85 19.34 4.66
KVA-Field with Tier-1 and Tier-2 routing 86.89 0.89 14.37 20.82 4.64

Efficiency

Method Latency FLOPs CD FID
Teacher, 50 steps 4.64 6.94 86.89 14.37
Generic distillation 0.52 0.78 98.45 19.82
Action-aware student 0.52 0.78 87.50 15.60
Spatial adaptive execution 0.42 0.63 88.65 16.55
Temporal cache, final efficient 0.37 0.55 89.87 17.67

Resources

Paper, code, and video

GitHub KVLR Code

Training, inference, distillation, and evaluation.

 PDF Paper

From Articulated Kinematics to Routed Visual Control. Accepted at NeurIPS 2026.

 MP4 Supplementary Demo

Video demonstration packaged with the supplementary material.

Citation

Reference this work

@article{li2026articulated,
  title={From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation},
  author={Li, Bohan and Yang, Shuojue and Peng, Baorui and Guo, Xianda and Zhang, Erli and Tao, Youqi and Duan, Junfeng and Xu, Daguang and Dou, Qi and Jin, Xin and others},
  journal={arXiv preprint arXiv:2605.08712},
  year={2026}
}