1<!DOCTYPE html> 2<html> 3<head> 4 <meta charset="utf-8"> 5 <!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" --> 6 <!-- Replace the content tag with appropriate information --> 7 <meta name="description" content="Project page for 'ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation'."> 8 <meta property="og:title" content="ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation"/> 9 <meta property="og:description" content="Human-scene interaction (HSI) generation is crucial for applications in embodied AI, virtual reality, and robotics. While existing methods can synthesize realistic human motions in 3D scenes and generate plausible human-object interactions, they heavily rely on datasets containing paired 3D scene and motion capture data, which are expensive and time-consuming to collect across diverse environments and interactions. We present ZeroHSI, a novel approach that enables zero-shot 4D human-scene interaction synthesis by integrating video generation and neural human rendering. Our key insight is to leverage the rich motion priors learned by state-of-the-art video generation models, which have been trained on vast amounts of natural human movements and interactions, and use differentiable rendering to reconstruct human-scene interactions. ZeroHSI can synthesize realistic human motions in both static scenes and environments with dynamic objects, without requiring any ground-truth motion data. We evaluate ZeroHSI on a curated dataset of different types of various indoor and outdoor scenes with different interaction prompts, demonstrating its ability to generate diverse and contextually appropriate human-scene interactions."/> 10 <meta property="og:url" content="https://awfuact.github.io/zerohsi/"/> 11 <!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630--> 12 <meta property="og:image" content="static/images/banner_image.jpg" /> 13 <meta property="og:image:width" content="1200"/> 14 <meta property="og:image:height" content="630"/> 15 16 17 <meta name="twitter:title" content="ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation"> 18 <meta name="twitter:description" content="Human-scene interaction (HSI) generation is crucial for applications in embodied AI, virtual reality, and robotics. While existing methods can synthesize realistic human motions in 3D scenes and generate plausible human-object interactions, they heavily rely on datasets containing paired 3D scene and motion capture data, which are expensive and time-consuming to collect across diverse environments and interactions. We present ZeroHSI, a novel approach that enables zero-shot 4D human-scene interaction synthesis by integrating video generation and neural human rendering. Our key insight is to leverage the rich motion priors learned by state-of-the-art video generation models, which have been trained on vast amounts of natural human movements and interactions, and use differentiable rendering to reconstruct human-scene interactions. ZeroHSI can synthesize realistic human motions in both static scenes and environments with dynamic objects, without requiring any ground-truth motion data. We evaluate ZeroHSI on a curated dataset of different types of various indoor and outdoor scenes with different interaction prompts, demonstrating its ability to generate diverse and contextually appropriate human-scene interactions."> 19 <!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600--> 20 <meta name="twitter:image" content="static/images/twitter_banner_image.jpg"> 21 <meta name="twitter:card" content="summary_large_image"> 22 <!-- Keywords for your paper to be indexed by--> 23 <meta name="keywords" content="zero-shot, human-scene interaction, video generation, 4D"> 24 <meta name="viewport" content="width=device-width, initial-scale=1"> 25 26 27 <title>ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation</title> 28 <!-- <link rel="icon" type="image/x-icon" href="static/images/favicon.ico"> --> 29 30 <link rel="stylesheet" href="static/css/bulma.min.css"> 31 <link rel="stylesheet" href="static/css/bulma-carousel.min.css"> 32 <link rel="stylesheet" href="static/css/bulma-slider.min.css"> 33 <link rel="stylesheet" href="static/css/fontawesome.all.min.css"> 34 <link rel="stylesheet" href="static/css/index.css"> 35 <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 36 <link rel="stylesheet" href="static/css/video-gallery.css"> 37 <link rel="stylesheet" href="static/css/grid.css"> 38 39
39<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
39 40
40<script src="https://documentcloud.adobe.com/view-sdk/main.js"></script>
40 41
41<script defer src="static/js/fontawesome.all.min.js"></script>
41 42
42<script src="https://kit.fontawesome.com/d383128d10.js" crossorigin="anonymous"></script>
42 43
43<script src="static/js/bulma-carousel.min.js"></script>
43 44
44<script src="static/js/bulma-slider.min.js"></script>
44 45
45<script src="static/js/index.js"></script>
45 46
46<script src="static/js/video-gallery.js"></script>
46 47
47<script src="static/js/grid.js"></script>
47 48
48<script type="text/javascript" src="https://www.statcounter.com/counter/counter.js" async></script>
48 49</head> 50<body> 51 52 53<section class="hero"> 54 <div class="hero-body"> 55 <div class="container is-max-desktop"> 56 <div class="columns is-centered"> 57 <div class="column has-text-centered"> 58 <h1 class="title is-1 publication-title">ZeroHSI: Zero-Shot 4D Human-Scene Interaction<br> by Video Generation</h1> 59 <div class="is-size-5 publication-authors"> 60 <!-- Paper authors --> 61 <span class="author-block"> 62 <a href="https://awfuact.github.io/" target="_blank">Hongjie Li</a><sup>*</sup>     63 </span> 64 <span class="author-block"> 65 <a href="https://kovenyu.com/" target="_blank">Hong-Xing (Koven) Yu</a><sup>*</sup>     66 </span> 67 <span class="author-block"> 68 <a href="https://lijiaman.github.io/" target="_blank">Jiaman Li</a>     69 </span> 70 <span class="author-block"> 71 <a href="https://jiajunwu.com/" target="_blank">Jiajun Wu</a>     72 </span> 73 </div> 74 <div class="is-size-5 publication-authors"> 75 <!-- Affiliations --> 76 <span class="author-block"> 77 Stanford University    <sup>*</sup>Contributed Equally 78 </span> 79 </div> 80 <div class="is-size-4 publication-authors" style="padding:1.5%;"> 81 <!-- Conference --> 82 <span class="author-block"> 83 <font color="#008FFF">3DV 2026</font> 84 </span> 85 </div> 86 <div class="column has-text-centered" style="padding-top:0;"> 87 <div class="publication-links"> 88 <!-- Arxiv PDF link --> 89 <span class="link-block"> 90 <a href="https://arxiv.org/pdf/2412.18600" class="external-link button is-normal is-rounded is-dark"> 91 <span class="icon"><i class="fas fa-file-pdf"></i></span> 92 <span>Paper</span> 93 </a> 94 </span> 95 96 <!-- ArXiv Abstract Link --> 97 <span class="link-block"> 98 <a href="https://arxiv.org/abs/2412.18600" class="external-link button is-normal is-rounded is-dark"> 99 <span class="icon"><i class="ai ai-arxiv"></i></span> 100 <span>arXiv</span> 101 </a> 102 </span> 103 104 <!-- Code link --> 105 <!-- <span class="link-block"> 106 <a href="" class="external-link button is-normal is-rounded is-dark"> 107 <span class="icon"><i class="fab fa-github"></i></span> 108 <span>Code (coming soon)</span> 109 </a> 110 </span> --> 111 112 <!-- Data link --> 113 <span class="link-block"> 114 <a href="https://drive.google.com/file/d/1BN2etWLa9WQPf9lnNrh6HpeLKnKVcmdk/view?usp=sharing" class="external-link button is-normal is-rounded is-dark"> 115 <span class="icon"><i class="fa-solid fa-download"></i></span> 116 <span>Data</span> 117 </a> 118 </span> 119 120 <!-- Twitter link --> 121 <span class="link-block"> 122 <a href="https://x.com/Koven_Yu/status/1872927109724946920" class="external-link button is-normal is-rounded is-dark">
123 <span class="icon"><i class="fa-brands fa-x-twitter"></i></span> 124 <span>TL;DR</span> 125 </a> 126 </span> 127 </div> 128 </div> 129 </div> 130 </div> 131 </div> 132 </div> 133</section> 134 135 136<!-- teaser --> 137<section class="hero teaser"> 138 <div class="hero-body"> 139 <div class="container"> 140 <figure class="image" style="margin-bottom: 0;"> 141 <img src="static/images/teaser.jpg" alt="Research Poster"> 142 <figcaption class="mt-3"> 143 ZeroHSI generates natural 3D human motion to interact with various environments (indoor/outdoor, real reconstructed/synthesized) and dynamic objects by leveraging video generation models and neural rendering, eliminating the need for paired motion-scene training data. 144 </figcaption> 145 </figure> 146 </div> 147 </div> 148</section> 149<!--End teaser --> 150 151 152<!-- Real scene demos --> 153<section class="hero is-small"> 154 <div class="hero-body"> 155 <div class="container"> 156 <h2 class="title is-2" style="margin-bottom: 0; margin-top: 3rem;">Generated Interactions on Real Reconstructed Scenes</h2> 157 <div class="video-caption">Input Reconstructed 3D Scene</div> 158 159 <div class="video-gallery-buttons" id="gallery-buttons-real-scene"> 160 <button class="button is-primary" data-video-index="0"> 161 <img src="static/images/garden_1.png" alt="Garden 1"> 162 </button> 163 <button class="button" data-video-index="1"> 164 <img src="static/images/bicycle.png" alt="Bicycle"> 165 </button> 166 <button class="button" data-video-index="2"> 167 <img src="static/images/garden_0.png" alt="Garden 0"> 168 </button> 169 <button class="button" data-video-index="3"> 170 <img src="static/images/room.png" alt="Room"> 171 </button> 172 <button class="button" data-video-index="4"> 173 <img src="static/images/garden_2.png" alt="Garden 2"> 174 </button> 175 <button class="button" data-video-index="5"> 176 <img src="static/images/truck.png" alt="Truck"> 177 </button> 178 </div> 179 180 <div class="video-gallery-carousel" id="gallery-carousel-real-scene"> 181 <div class="video-pair active"> 182 <div class="video-container"> 183 <div class="video-caption">Generated HSI Video</div> 184 <video preload="auto" controls muted loop playsinline height="100%"> 185 <source src="static/videos/garden_1_kling.mp4" type="video/mp4"> 186 </video> 187 </div> 188 <div class="video-container"> 189 <div class="video-caption">4D HSI Generated by ZeroHSI</div> 190 <video preload="auto" controls muted loop playsinline height="100%"> 191 <source src="static/videos/garden_1_ours.mp4" type="video/mp4"> 192 </video> 193 </div> 194 <div class="pair-caption">Text Prompt: "The person is lifting a vase."</div> 195 </div> 196 197 <div class="video-pair"> 198 <div class="video-container"> 199 <div class="video-caption">Generated HSI Video</div> 200 <video preload="auto" controls muted loop playsinline height="100%"> 201 <source src="static/videos/bicycle_kling.mp4" type="video/mp4"> 202 </video> 203 </div> 204 <div class="video-container"> 205 <div class="video-caption">4D HSI Generated by ZeroHSI</div> 206 <video preload="auto" controls muted loop playsinline height="100%"> 207 <source src="static/videos/bicycle_ours.mp4" type="video/mp4"> 208 </video> 209 </div> 210 <div class="pair-caption">Text Prompt: "The person is sitting on the bench."</div> 211 </div> 212 213 <div class="video-pair"> 214 <div class="video-container"> 215 <div class="video-caption">Generated HSI Video</div> 216 <video preload="auto" controls muted loop playsinline height="100%"> 217 <source src="static/videos/garden_0_kling.mp4" type="video/mp4"> 218 </video> 219 </div> 220 <div class="video-container"> 221 <div class="video-caption">4D HSI Generated by ZeroHSI</div> 222 <video preload="auto" controls muted loop playsinline height="100%"> 223 <source src="static/videos/garden_0_ours.mp4" type="video/mp4"> 224 </video> 225 </div> 226 <div class="pair-caption">Text Prompt: "The person is watering flowers with a watering can."</div> 227 </div> 228 229 <div class="video-pair"> 230 <div class="video-container"> 231 <div class="video-caption">Generated HSI Video</div> 232 <video preload="auto" controls muted loop playsinline height="100%"> 233 <source src="static/videos/room_kling.mp4" type="video/mp4"> 234 </video> 235 </div> 236 <div class="video-container"> 237 <div class="video-caption">4D HSI Generated by ZeroHSI</div> 238 <video preload="auto" controls muted loop playsinline height="100%"> 239 <source src="static/videos/room_ours.mp4" type="video/mp4"> 240 </video> 241 </div> 242 <div class="pair-caption">Text Prompt: "The person is playing guitar while sitting on the sofa."</div> 243 </div> 244 245 <div class="video-pair"> 246 <div class="video-container"> 247 <div class="video-caption">Generated HSI Video</div> 248 <video preload="auto" controls muted loop playsinline height="100%"> 249 <source src="static/videos/garden_2_kling.mp4" type="video/mp4"> 250 </video> 251 </div> 252 <div class="video-container"> 253 <div class="video-caption">4D HSI Generated by ZeroHSI</div> 254 <video preload="auto" controls muted loop playsinline height="100%"> 255 <source src="static/videos/garden_2_ours.mp4" type="video/mp4"> 256 </video> 257 </div> 258 <div class="pair-caption">Text Prompt: "The person is operating a lawn mower."</div> 259 </div> 260 261 <div class="video-pair"> 262 <div class="video-container"> 263 <div class="video-caption">Generated HSI Video</div> 264 <video preload="auto" controls muted loop playsinline height="100%"> 265 <source src="static/videos/truck_kling.mp4" type="video/mp4"> 266 </video> 267 </div> 268 <div class="video-container"> 269 <div class="video-caption">4D HSI Generated by ZeroHSI</div> 270 <video preload="auto" controls muted loop playsinline height="100%"> 271 <source src="static/videos/truck_ours.mp4" type="video/mp4"> 272 </video> 273 </div> 274 <div class="pair-caption">Text Prompt: "The person is cleaning the car."</div> 275 </div> 276 </div> 277 </div> 278 </div> 279</section> 280<!--End real scene demos--> 281 282 283<!-- Long-term demos --> 284<section class="hero is-small"> 285 <div class="hero-body"> 286 <div class="container"> 287 <h2 class="title is-2" style="margin-bottom: 0; margin-top: 3rem;">Generated Long-term Interactions</h2> 288 <div class="video-caption">Input Reconstructed 3D Scene</div> 289 290 <div class="video-gallery-buttons" id="gallery-buttons-long-term"> 291 <button class="button is-primary" data-video-index="0"> 292 <img src="static/images/garden_0.png" alt="Garden Long 1"> 293 </button> 294 <button class="button" data-video-index="1"> 295 <img src="static/images/bicycle_1.png" alt="Bicycle Long 1"> 296 </button> 297 <button class="button" data-video-index="2"> 298 <img src="static/images/bicycle_2.png" alt="Bicycle Long 2"> 299 </button> 300 <button class="button" data-video-index="3"> 301 <img src="static/images/bicycle_3.png" alt="Bicycle Long 3"> 302 </button> 303 </div> 304 305 <div class="video-gallery-carousel" id="gallery-carousel-long-term"> 306 <div class="video-single active"> 307 <div class="video-container"> 308 <div class="video-caption">Long-term HSI Generated by ZeroHSI</div> 309 <video preload="auto" controls muted loop playsinline height="100%"> 310 <source src="static/videos/garden_long_1.mp4" type="video/mp4"> 311 </video> 312 </div> 313 <div class="pair-caption"> 314 <div class="prompt-container"> 315 <div class="prompt-line">
316 <span class="prompt-label">Text Prompt 1:</span> 317 <span class="prompt-text">"The person is walking forward to the table."</span> 318 </div> 319 <div class="prompt-line"> 320 <span class="prompt-label">Text Prompt 2:</span> 321 <span class="prompt-text">"The person is watering flowers with a watering can."</span> 322 </div> 323 <div class="prompt-line"> 324 <span class="prompt-label">Text Prompt 3:</span> 325 <span class="prompt-text">"The person is putting down the watering can on the table."</span> 326 </div> 327 <div class="prompt-line"> 328 <span class="prompt-label">Text Prompt 4:</span> 329 <span class="prompt-text">"The person is leaning on the table."</span> 330 </div> 331 </div> 332 </div> 333 </div> 334 335 <div class="video-single"> 336 <div class="video-container"> 337 <div class="video-caption">Long-term HSI Generated by ZeroHSI</div> 338 <video preload="auto" controls muted loop playsinline height="100%"> 339 <source src="static/videos/bicycle_long_1.mp4" type="video/mp4"> 340 </video> 341 </div> 342 <div class="pair-caption"> 343 <div class="prompt-container"> 344 <div class="prompt-line"> 345 <span class="prompt-label">Text Prompt 1:</span> 346 <span class="prompt-text">"The person is sitting on the bench."</span> 347 </div> 348 <div class="prompt-line"> 349 <span class="prompt-label">Text Prompt 2:</span> 350 <span class="prompt-text">"The person is standing up."</span> 351 </div> 352 <div class="prompt-line"> 353 <span class="prompt-label">Text Prompt 3:</span> 354 <span class="prompt-text">"The person is practice Tai Chi."</span> 355 </div> 356 </div> 357 </div> 358 </div> 359 360 <div class="video-single"> 361 <div class="video-container"> 362 <div class="video-caption">Long-term HSI Generated by ZeroHSI</div> 363 <video preload="auto" controls muted loop playsinline height="100%"> 364 <source src="static/videos/bicycle_long_2.mp4" type="video/mp4"> 365 </video> 366 </div> 367 <div class="pair-caption"> 368 <div class="prompt-container"> 369 <div class="prompt-line"> 370 <span class="prompt-label">Text Prompt 1:</span> 371 <span class="prompt-text">"The person is sitting on the bench."</span> 372 </div> 373 <div class="prompt-line"> 374 <span class="prompt-label">Text Prompt 2:</span> 375 <span class="prompt-text">"The person is standing up."</span> 376 </div> 377 <div class="prompt-line"> 378 <span class="prompt-label">Text Prompt 3:</span> 379 <span class="prompt-text">"The person is practice Tai Chi."</span> 380 </div> 381 </div> 382 </div> 383 </div> 384 385 <div class="video-single"> 386 <div class="video-container"> 387 <div class="video-caption">Long-term HSI Generated by ZeroHSI</div> 388 <video preload="auto" controls muted loop playsinline height="100%"> 389 <source src="static/videos/bicycle_long_3.mp4" type="video/mp4"> 390 </video> 391 </div> 392 <div class="pair-caption"> 393 <div class="prompt-container"> 394 <div class="prompt-line">
395 <span class="prompt-label">Text Prompt 1:</span> 396 <span class="prompt-text">"The person is sitting on the bench."</span> 397 </div> 398 <div class="prompt-line"> 399 <span class="prompt-label">Text Prompt 2:</span> 400 <span class="prompt-text">"The person is standing up."</span> 401 </div> 402 <div class="prompt-line"> 403 <span class="prompt-label">Text Prompt 3:</span> 404 <span class="prompt-text">"The person is practice Tai Chi."</span> 405 </div> 406 </div> 407 </div> 408 </div> 409 </div> 410 </div> 411 </div> 412</section> 413<!-- End long-term demos --> 414 415 416<!-- Comparison of dynamic scenario --> 417<section class="hero is-small"> 418 <div class="hero-body"> 419 <div class="container"> 420 <h2 class="title is-2" style="margin-bottom: 0; margin-top: 3rem;">Generated Interactions on Synthetic Scenes with Objects</h2> 421 <div class="video-caption">Input Synthetic 3D Scene</div> 422 423 <div class="video-gallery-buttons" id="gallery-buttons-dynamic"> 424 <button class="button is-primary" data-video-index="0"> 425 <img src="static/images/1_living_room_dynamic.png" alt="Living Room Dynamic"> 426 </button> 427 <button class="button" data-video-index="1"> 428 <img src="static/images/2_gym_dynamic.png" alt="Gym Dynamic"> 429 </button> 430 <button class="button" data-video-index="2"> 431 <img src="static/images/7_store_dynamic_0.png" alt="Store Dynamic 0"> 432 </button> 433 <button class="button" data-video-index="3"> 434 <img src="static/images/7_store_dynamic_1.png" alt="Store Dynamic 1"> 435 </button> 436 </div> 437 438 <div class="video-gallery-carousel" id="gallery-carousel-dynamic"> 439 <div class="video-triple active"> 440 <div class="video-container"> 441 <div class="video-caption">ZeroHSI (Ours)</div> 442 <video preload="auto" controls muted loop playsinline height="100%"> 443 <source src="static/videos/1_living_room_dynamic_ours.mp4" type="video/mp4"> 444 </video> 445 </div> 446 <div class="video-container"> 447 <div class="video-caption">LINGO</div> 448 <video preload="auto" controls muted loop playsinline height="100%"> 449 <source src="static/videos/1_living_room_dynamic_lingo.mp4" type="video/mp4"> 450 </video> 451 </div> 452 <div class="video-container"> 453 <div class="video-caption">CHOIS</div> 454 <video preload="auto" controls muted loop playsinline height="100%"> 455 <source src="static/videos/1_living_room_dynamic_chois.mp4" type="video/mp4"> 456 </video> 457 </div> 458 <div class="pair-caption">Text Prompt: "The person is playing guitar while sitting on the sofa."</div> 459 </div> 460 461 <div class="video-triple"> 462 <div class="video-container"> 463 <div class="video-caption">ZeroHSI (Ours)</div> 464 <video preload="auto" controls muted loop playsinline height="100%"> 465 <source src="static/videos/2_gym_dynamic_ours.mp4" type="video/mp4"> 466 </video> 467 </div> 468 <div class="video-container"> 469 <div class="video-caption">LINGO</div> 470 <video preload="auto" controls muted loop playsinline height="100%"> 471 <source src="static/videos/2_gym_dynamic_lingo.mp4" type="video/mp4"> 472 </video> 473 </div> 474 <div class="video-container"> 475 <div class="video-caption">CHOIS</div> 476 <video preload="auto" controls muted loop playsinline height="100%"> 477 <source src="static/videos/2_gym_dynamic_chois.mp4" type="video/mp4"> 478 </video> 479 </div> 480 <div class="pair-caption">Text Prompt: "The person is lifting weights."</div> 481 </div> 482 483 <div class="video-triple"> 484 <div class="video-container"> 485 <div class="video-caption">ZeroHSI (Ours)</div> 486 <video preload="auto" controls muted loop playsinline height="100%"> 487 <source src="static/videos/7_store_dynamic_0_ours.mp4" type="video/mp4"> 488 </video> 489 </div> 490 <div class="video-container"> 491 <div class="video-caption">LINGO</div> 492 <video preload="auto" controls muted loop playsinline height="100%"> 493 <source src="static/videos/7_store_dynamic_0_lingo.mp4" type="video/mp4"> 494 </video> 495 </div> 496 <div class="video-container"> 497 <div class="video-caption">CHOIS</div> 498 <video preload="auto" controls muted loop playsinline height="100%"> 499 <source src="static/videos/7_store_dynamic_0_chois.mp4" type="video/mp4"> 500 </video> 501 </div> 502 <div class="pair-caption">Text Prompt: "The person is sliding while sitting on the chair."</div> 503 </div> 504 505 <div class="video-triple"> 506 <div class="video-container"> 507 <div class="video-caption">ZeroHSI (Ours)</div> 508 <video preload="auto" controls muted loop playsinline height="100%"> 509 <source src="static/videos/7_store_dynamic_1_ours.mp4" type="video/mp4"> 510 </video> 511 </div> 512 <div class="video-container"> 513 <div class="video-caption">LINGO</div> 514 <video preload="auto" controls muted loop playsinline height="100%"> 515 <source src="static/videos/7_store_dynamic_1_lingo.mp4" type="video/mp4"> 516 </video> 517 </div> 518 <div class="video-container"> 519 <div class="video-caption">CHOIS</div> 520 <video preload="auto" controls muted loop playsinline height="100%"> 521 <source src="static/videos/7_store_dynamic_1_chois.mp4" type="video/mp4"> 522 </video> 523 </div> 524 <div class="pair-caption">Text Prompt: "The person is pushing shopping cart."</div> 525 </div> 526 </div> 527 </div> 528 </div> 529</section> 530<!-- End comparison of dynamic scenario --> 531 532
533<!-- Comparison of static scenario --> 534<section class="hero is-small"> 535 <div class="hero-body"> 536 <div class="container"> 537 <h2 class="title is-2" style="margin-bottom: 0; margin-top: 3rem;">Generated Interactions on Synthetic Scenes</h2> 538 <div class="video-caption">Input Synthetic 3D Scene</div> 539 540 <div class="video-gallery-buttons" id="gallery-buttons-static"> 541 <button class="button is-primary" data-video-index="0"> 542 <img src="static/images/0_bedroom_static.png" alt="Bedroom"> 543 </button> 544 <button class="button" data-video-index="1"> 545 <img src="static/images/1_living_room_static.png" alt="Living Room"> 546 </button> 547 <button class="button" data-video-index="2"> 548 <img src="static/images/4_playground_static.png" alt="Playground"> 549 </button> 550 <button class="button" data-video-index="3"> 551 <img src="static/images/7_store_static.png" alt="Store"> 552 </button> 553 </div> 554 555 <div class="video-gallery-carousel" id="gallery-carousel-static"> 556 <div class="video-triple active"> 557 <div class="video-container"> 558 <div class="video-caption">ZeroHSI (Ours)</div> 559 <video preload="auto" controls muted loop playsinline height="100%"> 560 <source src="static/videos/0_bedroom_static_ours.mp4" type="video/mp4"> 561 </video> 562 </div> 563 <div class="video-container"> 564 <div class="video-caption">TRUMANS</div> 565 <video preload="auto" controls muted loop playsinline height="100%"> 566 <source src="static/videos/0_bedroom_static_trumans.mp4" type="video/mp4"> 567 </video> 568 </div> 569 <div class="video-container"> 570 <div class="video-caption">LINGO</div> 571 <video preload="auto" controls muted loop playsinline height="100%"> 572 <source src="static/videos/0_bedroom_static_lingo.mp4" type="video/mp4"> 573 </video> 574 </div> 575 <div class="pair-caption">Text Prompt: "The person is leaning on the ladder."</div> 576 </div> 577 578 <div class="video-triple"> 579 <div class="video-container"> 580 <div class="video-caption">ZeroHSI (Ours)</div> 581 <video preload="auto" controls muted loop playsinline height="100%"> 582 <source src="static/videos/1_living_room_static_ours.mp4" type="video/mp4"> 583 </video> 584 </div> 585 <div class="video-container"> 586 <div class="video-caption">TRUMANS</div> 587 <video preload="auto" controls muted loop playsinline height="100%"> 588 <source src="static/videos/1_living_room_static_trumans.mp4" type="video/mp4"> 589 </video> 590 </div> 591 <div class="video-container"> 592 <div class="video-caption">LINGO</div> 593 <video preload="auto" controls muted loop playsinline height="100%"> 594 <source src="static/videos/1_living_room_static_lingo.mp4" type="video/mp4"> 595 </video> 596 </div> 597 <div class="pair-caption">Text Prompt: "The person is sitting on the sofa back."</div> 598 </div> 599 600 <div class="video-triple"> 601 <div class="video-container"> 602 <div class="video-caption">ZeroHSI (Ours)</div> 603 <video preload="auto" controls muted loop playsinline height="100%"> 604 <source src="static/videos/4_playground_static_ours.mp4" type="video/mp4"> 605 </video> 606 </div> 607 <div class="video-container"> 608 <div class="video-caption">TRUMANS</div> 609 <video preload="auto" controls muted loop playsinline height="100%"> 610 <source src="static/videos/4_playground_static_trumans.mp4" type="video/mp4"> 611 </video> 612 </div> 613 <div class="video-container"> 614 <div class="video-caption">LINGO</div> 615 <video preload="auto" controls muted loop playsinline height="100%"> 616 <source src="static/videos/4_playground_static_lingo.mp4" type="video/mp4"> 617 </video> 618 </div> 619 <div class="pair-caption">Text Prompt: "The person is sliding down the slide."</div> 620 </div> 621 622 <div class="video-triple"> 623 <div class="video-container"> 624 <div class="video-caption">ZeroHSI (Ours)</div> 625 <video preload="auto" controls muted loop playsinline height="100%"> 626 <source src="static/videos/7_store_static_ours.mp4" type="video/mp4"> 627 </video> 628 </div> 629 <div class="video-container"> 630 <div class="video-caption">TRUMANS</div> 631 <video preload="auto" controls muted loop playsinline height="100%"> 632 <source src="static/videos/7_store_static_trumans.mp4" type="video/mp4"> 633 </video> 634 </div> 635 <div class="video-container"> 636 <div class="video-caption">LINGO</div> 637 <video preload="auto" controls muted loop playsinline height="100%"> 638 <source src="static/videos/7_store_static_lingo.mp4" type="video/mp4"> 639 </video> 640 </div> 641 <div class="pair-caption">Text Prompt: "The person is picking out snacks on the shelf."</div> 642 </div> 643 </div> 644 </div> 645 </div> 646</section>
647<!-- End comparison of static scenario --> 648 649 650<!-- Paper abstract --> 651<section class="hero is-small"> 652 <div class="hero-body"> 653 <div class="container"> 654 <h2 class="title is-2" style="margin-bottom: 0.5rem; margin-top: 3rem;">Abstract</h2> 655 <figure class="image" style="margin-bottom: 0rem;"> 656 <figcaption style="margin-top: 0rem;"> 657 Human-scene interaction (HSI) generation is crucial for applications in embodied AI, virtual reality, and robotics. While existing methods can synthesize realistic human motions in 3D scenes and generate plausible human-object interactions, they heavily rely on datasets containing paired 3D scene and motion capture data, which are expensive and time-consuming to collect across diverse environments and interactions. We present ZeroHSI, a novel approach that enables zero-shot 4D human-scene interaction synthesis by integrating video generation and neural human rendering. Our key insight is to leverage the rich motion priors learned by state-of-the-art video generation models, which have been trained on vast amounts of natural human movements and interactions, and use differentiable rendering to reconstruct human-scene interactions. ZeroHSI can synthesize realistic human motions in both static scenes and environments with dynamic objects, without requiring any ground-truth motion data. We evaluate ZeroHSI on a curated dataset of different types of various indoor and outdoor scenes with different interaction prompts, demonstrating its ability to generate diverse and contextually appropriate human-scene interactions. 658 </figcaption> 659 </figure> 660 </div> 661 </div> 662</section> 663<!-- End paper abstract --> 664 665 666<!-- Method --> 667<section class="hero is-small"> 668 <div class="hero-body"> 669 <div class="container"> 670 <h2 class="title is-2" style="margin-bottom: 1rem; margin-top: 3rem;">Method Overview</h2> 671 <figure class="image" style="margin-bottom: 0;"> 672 <img src="static/images/method.png" alt="Research Poster"> 673 <figcaption class="mt-3"> 674 ZeroHSI takes a 3D scene, an interactable object, a language description, and initial states as input to synthesize both human and dynamic object motion sequences. Our approach begins with HSI video generation conditioned on the rendered initial state and text prompt. Through differentiable neural rendering, we optimize per-frame camera pose, human pose parameters, and object 6D pose by minimizing the discrepancy between the rendered and generated reference videos. 675 </figcaption> 676 </figure> 677 </div> 678 </div> 679</section> 680<!--End method --> 681 682 683<!-- Dataset --> 684<section class="hero is-small"> 685 <div class="hero-body"> 686 <div class="container"> 687 <h2 class="title is-2" style="margin-bottom: 1rem; margin-top: 3rem;">The AnyInteraction Dataset</h2> 688 <figure class="grid-container" style="margin-bottom: 0;"> 689 <!-- Small figures --> 690 <div class="subfigure"> 691 <img src="static/images/0_bedroom.png" alt="Figure 00"> 692 <div class="subfigure-caption">Bedroom</div> 693 </div> 694 <div class="subfigure"> 695 <img src="static/images/1_living_room.png" alt="Figure 01"> 696 <div class="subfigure-caption">Living Room</div> 697 </div> 698 <div class="subfigure"> 699 <img src="static/images/2_gym.png" alt="Figure 02"> 700 <div class="subfigure-caption">Gym</div> 701 </div> 702 <div class="subfigure"> 703 <img src="static/images/3_bar.png" alt="Figure 03"> 704 <div class="subfigure-caption">Bar</div> 705 </div> 706 <div class="subfigure"> 707 <img src="static/images/4_playground.png" alt="Figure 04"> 708 <div class="subfigure-caption">Playground</div> 709 </div> 710 <div class="subfigure"> 711 <img src="static/images/5_greenhouse.png" alt="Figure 05"> 712 <div class="subfigure-caption">Greenhouse</div> 713 </div> 714 <div class="subfigure"> 715 <img src="static/images/6_cafe.png" alt="Figure 06"> 716 <div class="subfigure-caption">Cafe</div> 717 </div> 718 <div class="subfigure"> 719 <img src="static/images/7_store.png" alt="Figure 07"> 720 <div class="subfigure-caption">Store</div> 721 </div> 722 <div class="subfigure"> 723 <img src="static/images/8_garden.png" alt="Figure 08"> 724 <div class="subfigure-caption">Garden</div> 725 </div> 726 <div class="subfigure"> 727 <img src="static/images/9_bicycle.png" alt="Figure 09"> 728 <div class="subfigure-caption">Bicycle</div> 729 </div> 730 <div class="subfigure"> 731 <img src="static/images/10_room.png" alt="Figure 10"> 732 <div class="subfigure-caption">Room</div> 733 </div> 734 <div class="subfigure"> 735 <img src="static/images/11_truck.png" alt="Figure 11"> 736 <div class="subfigure-caption">Truck</div> 737 </div> 738 739 <!-- Enlarged figures (hidden by default) --> 740 <div id="enlarged-figure-0" class="enlarged-figure"> 741 <img src="static/images/0_bedroom.png" alt="Big Figure 00"> 742 </div> 743 <div id="enlarged-figure-1" class="enlarged-figure"> 744 <img src="static/images/1_living_room.png" alt="Big Figure 01"> 745 </div> 746 <div id="enlarged-figure-2" class="enlarged-figure"> 747 <img src="static/images/2_gym.png" alt="Big Figure 02"> 748 </div> 749 <div id="enlarged-figure-3" class="enlarged-figure"> 750 <img src="static/images/3_bar.png" alt="Big Figure 03"> 751 </div> 752 <div id="enlarged-figure-4" class="enlarged-figure"> 753 <img src="static/images/4_playground.png" alt="Big Figure 04"> 754 </div> 755 <div id="enlarged-figure-5" class="enlarged-figure"> 756 <img src="static/images/5_greenhouse.png" alt="Big Figure 05"> 757 </div> 758 <div id="enlarged-figure-6" class="enlarged-figure"> 759 <img src="static/images/6_cafe.png" alt="Big Figure 06"> 760 </div> 761 <div id="enlarged-figure-7" class="enlarged-figure"> 762 <img src="static/images/7_store.png" alt="Big Figure 07"> 763 </div> 764 <div id="enlarged-figure-8" class="enlarged-figure"> 765 <img src="static/images/8_garden.png" alt="Big Figure 08"> 766 </div> 767 <div id="enlarged-figure-9" class="enlarged-figure"> 768 <img src="static/images/9_bicycle.png" alt="Big Figure 09"> 769 </div> 770 <div id="enlarged-figure-10" class="enlarged-figure"> 771 <img src="static/images/10_room.png" alt="Big Figure 10"> 772 </div> 773 <div id="enlarged-figure-11" class="enlarged-figure"> 774 <img src="static/images/11_truck.png" alt="Big Figure 11"> 775 </div> 776 777 <!-- Main caption -->
778 <figcaption class="main-caption mt-3"> 779 We curate the AnyInteraction dataset to evaluate zero-shot HSI synthesis. The dataset encompasses 12 diverse scenes (7 indoor and 5 outdoor) from various sources including TRUMANS, public 3D assets, and reconstructed real scenes from Mip-NeRF 360 and Tanks and Temples datasets. It features 7 types of rigid dynamic objects (Guitar, Barbell, Watering Can, Office Chair, Shopping Cart, Vase, and Mower), supporting 22 evaluation instances that include 13 static and 9 dynamic object interaction. 780 </figcaption> 781 782 </figure> 783 </div> 784 </div> 785</section> 786<!--End Dataset --> 787 788 789<!--BibTex citation --> 790 <section class="hero is-small"> 791 <div class="hero-body"> 792 <div class="container"> 793 <h2 class="title is-2" style="margin-bottom: 1rem; margin-top: 3rem;">BibTex</h2> 794 <figure class="image" style="margin-bottom: 0;"> 795 <pre><code>@inproceedings{li2026zerohsi, 796 title={Zerohsi: Zero-shot 4d human-scene interaction by video generation}, 797 author={Li, Hongjie and Yu, Hong-Xing and Li, Jiaman and Wu, Jiajun}, 798 booktitle={International Conference on 3D Vision (3DV)}, 799 year={2026} 800}</code></pre> 801 </figure> 802 </div> 803 </div> 804</section> 805<!--End BibTex citation --> 806 807 808<footer class="footer"> 809 <div class="container"> 810 <div class="columns is-centered"> 811 <div class="column is-8"> 812 <div class="content has-text-centered"> 813 <p> 814 This page was built using the <a href="https://github.com/eliahuhorwitz/Academic-project-page-template" target="_blank">Academic Project Page Template</a> which was adopted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a> project page. 815 <br> 816 This website is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative Commons Attribution-ShareAlike 4.0 International License</a>. 817 </p> 818 </div> 819 </div> 820 </div> 821 </div> 822</footer> 823 824 825</body> 826</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.