PageSourceSearch

https://awfuact.github.io/zerohsi/

html awfuact.github.io collected 2026-10-03 08:40:44 UTC 47,871 bytes, 826 lines download raw bytes

1<!DOCTYPE html>
2<html>
3<head>
4    <meta charset="utf-8">
5    <!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" -->
6    <!-- Replace the content tag with appropriate information -->
7    <meta name="description" content="Project page for 'ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation'.">
8    <meta property="og:title" content="ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation"/>
9    <meta property="og:description" content="Human-scene interaction (HSI) generation is crucial for applications in embodied AI, virtual reality, and robotics. While existing methods can synthesize realistic human motions in 3D scenes and generate plausible human-object interactions, they heavily rely on datasets containing paired 3D scene and motion capture data, which are expensive and time-consuming to collect across diverse environments and interactions. We present ZeroHSI, a novel approach that enables zero-shot 4D human-scene interaction synthesis by integrating video generation and neural human rendering. Our key insight is to leverage the rich motion priors learned by state-of-the-art video generation models, which have been trained on vast amounts of natural human movements and interactions, and use differentiable rendering to reconstruct human-scene interactions. ZeroHSI can synthesize realistic human motions in both static scenes and environments with dynamic objects, without requiring any ground-truth motion data. We evaluate ZeroHSI on a curated dataset of different types of various indoor and outdoor scenes with different interaction prompts, demonstrating its ability to generate diverse and contextually appropriate human-scene interactions."/>
10    <meta property="og:url" content="https://awfuact.github.io/zerohsi/"/>
11    <!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630-->
12    <meta property="og:image" content="static/images/banner_image.jpg" />
13    <meta property="og:image:width" content="1200"/>
14    <meta property="og:image:height" content="630"/>
15
16
17    <meta name="twitter:title" content="ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation">
18    <meta name="twitter:description" content="Human-scene interaction (HSI) generation is crucial for applications in embodied AI, virtual reality, and robotics. While existing methods can synthesize realistic human motions in 3D scenes and generate plausible human-object interactions, they heavily rely on datasets containing paired 3D scene and motion capture data, which are expensive and time-consuming to collect across diverse environments and interactions. We present ZeroHSI, a novel approach that enables zero-shot 4D human-scene interaction synthesis by integrating video generation and neural human rendering. Our key insight is to leverage the rich motion priors learned by state-of-the-art video generation models, which have been trained on vast amounts of natural human movements and interactions, and use differentiable rendering to reconstruct human-scene interactions. ZeroHSI can synthesize realistic human motions in both static scenes and environments with dynamic objects, without requiring any ground-truth motion data. We evaluate ZeroHSI on a curated dataset of different types of various indoor and outdoor scenes with different interaction prompts, demonstrating its ability to generate diverse and contextually appropriate human-scene interactions.">
19    <!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600-->
20    <meta name="twitter:image" content="static/images/twitter_banner_image.jpg">
21    <meta name="twitter:card" content="summary_large_image">
22    <!-- Keywords for your paper to be indexed by-->
23    <meta name="keywords" content="zero-shot, human-scene interaction, video generation, 4D">
24    <meta name="viewport" content="width=device-width, initial-scale=1">
25
26
27    <title>ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generation</title>
28    <!-- <link rel="icon" type="image/x-icon" href="static/images/favicon.ico"> -->
29
30    <link rel="stylesheet" href="static/css/bulma.min.css">
31    <link rel="stylesheet" href="static/css/bulma-carousel.min.css">
32    <link rel="stylesheet" href="static/css/bulma-slider.min.css">
33    <link rel="stylesheet" href="static/css/fontawesome.all.min.css">
34    <link rel="stylesheet" href="static/css/index.css">
35    <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
36    <link rel="stylesheet" href="static/css/video-gallery.css">
37    <link rel="stylesheet" href="static/css/grid.css">
38
39    
39<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
39
40    
40<script src="https://documentcloud.adobe.com/view-sdk/main.js"></script>
40
41    
41<script defer src="static/js/fontawesome.all.min.js"></script>
41
42    
42<script src="https://kit.fontawesome.com/d383128d10.js" crossorigin="anonymous"></script>
42
43    
43<script src="static/js/bulma-carousel.min.js"></script>
43
44    
44<script src="static/js/bulma-slider.min.js"></script>
44
45    
45<script src="static/js/index.js"></script>
45
46    
46<script src="static/js/video-gallery.js"></script>
46
47    
47<script src="static/js/grid.js"></script>
47
48    
48<script type="text/javascript" src="https://www.statcounter.com/counter/counter.js" async></script>
48
49</head>
50<body>
51
52
53<section class="hero">
54    <div class="hero-body">
55        <div class="container is-max-desktop">
56            <div class="columns is-centered">
57                <div class="column has-text-centered">
58                    <h1 class="title is-1 publication-title">ZeroHSI: Zero-Shot 4D Human-Scene Interaction<br> by Video Generation</h1>
59                    <div class="is-size-5 publication-authors">
60                        <!-- Paper authors -->
61                        <span class="author-block">
62                            <a href="https://awfuact.github.io/" target="_blank">Hongjie Li</a><sup>*</sup>&nbsp&nbsp&nbsp&nbsp
63                        </span>
64                        <span class="author-block">
65                            <a href="https://kovenyu.com/" target="_blank">Hong-Xing (Koven) Yu</a><sup>*</sup>&nbsp&nbsp&nbsp&nbsp
66                        </span>
67                        <span class="author-block">
68                            <a href="https://lijiaman.github.io/" target="_blank">Jiaman Li</a>&nbsp&nbsp&nbsp&nbsp
69                        </span>
70                        <span class="author-block">
71                            <a href="https://jiajunwu.com/" target="_blank">Jiajun Wu</a>&nbsp&nbsp&nbsp&nbsp
72                        </span>
73                    </div>
74                    <div class="is-size-5 publication-authors">
75                        <!-- Affiliations -->
76                        <span class="author-block">
77                            Stanford University &nbsp&nbsp <sup>*</sup>Contributed Equally
78                        </span>
79                    </div>
80                    <div class="is-size-4 publication-authors" style="padding:1.5%;">
81                        <!-- Conference -->
82                        <span class="author-block">
83                            <font color="#008FFF">3DV 2026</font>
84                        </span>
85                    </div>
86                    <div class="column has-text-centered" style="padding-top:0;">
87                        <div class="publication-links">
88                            <!-- Arxiv PDF link -->
89                            <span class="link-block">
90                                <a href="https://arxiv.org/pdf/2412.18600" class="external-link button is-normal is-rounded is-dark">
91                                    <span class="icon"><i class="fas fa-file-pdf"></i></span>
92                                    <span>Paper</span>
93                                </a>
94                            </span>
95            
96                            <!-- ArXiv Abstract Link -->
97                            <span class="link-block">
98                                <a href="https://arxiv.org/abs/2412.18600" class="external-link button is-normal is-rounded is-dark">
99                                    <span class="icon"><i class="ai ai-arxiv"></i></span>
100                                    <span>arXiv</span>
101                                </a>
102                            </span>
103            
104                            <!-- Code link -->
105                            <!-- <span class="link-block">
106                                <a href="" class="external-link button is-normal is-rounded is-dark">
107                                    <span class="icon"><i class="fab fa-github"></i></span>
108                                    <span>Code (coming soon)</span>
109                                </a>
110                            </span> -->
111            
112                            <!-- Data link -->
113                            <span class="link-block">
114                                <a href="https://drive.google.com/file/d/1BN2etWLa9WQPf9lnNrh6HpeLKnKVcmdk/view?usp=sharing" class="external-link button is-normal is-rounded is-dark">
115                                    <span class="icon"><i class="fa-solid fa-download"></i></span>
116                                    <span>Data</span>
117                                </a>
118                            </span>
119
120                            <!-- Twitter link -->
121                            <span class="link-block">
122                                <a href="https://x.com/Koven_Yu/status/1872927109724946920" class="external-link button is-normal is-rounded is-dark">
123                                    <span class="icon"><i class="fa-brands fa-x-twitter"></i></span>
124                                    <span>TL;DR</span>
125                                </a>
126                            </span>
127                        </div>
128                    </div>
129                </div>
130            </div>
131        </div>
132    </div>
133</section>
134
135
136<!-- teaser -->
137<section class="hero teaser">
138    <div class="hero-body">
139        <div class="container">
140            <figure class="image" style="margin-bottom: 0;">
141                <img src="static/images/teaser.jpg" alt="Research Poster">
142                <figcaption class="mt-3">
143                    ZeroHSI generates natural 3D human motion to interact with various environments (indoor/outdoor, real reconstructed/synthesized) and dynamic objects by leveraging video generation models and neural rendering, eliminating the need for paired motion-scene training data.
144                </figcaption>
145            </figure>
146        </div>
147    </div>
148</section>
149<!--End teaser -->
150
151
152<!-- Real scene demos -->
153<section class="hero is-small">
154    <div class="hero-body">
155        <div class="container">
156            <h2 class="title is-2" style="margin-bottom: 0; margin-top: 3rem;">Generated Interactions on Real Reconstructed Scenes</h2>
157            <div class="video-caption">Input Reconstructed 3D Scene</div>
158
159            <div class="video-gallery-buttons" id="gallery-buttons-real-scene">
160                <button class="button is-primary" data-video-index="0">
161                    <img src="static/images/garden_1.png" alt="Garden 1">
162                </button>
163                <button class="button" data-video-index="1">
164                    <img src="static/images/bicycle.png" alt="Bicycle">
165                </button>
166                <button class="button" data-video-index="2">
167                    <img src="static/images/garden_0.png" alt="Garden 0">
168                </button>
169                <button class="button" data-video-index="3">
170                    <img src="static/images/room.png" alt="Room">
171                </button>
172                <button class="button" data-video-index="4">
173                    <img src="static/images/garden_2.png" alt="Garden 2">
174                </button>
175                <button class="button" data-video-index="5">
176                    <img src="static/images/truck.png" alt="Truck">
177                </button>
178            </div>
179
180            <div class="video-gallery-carousel" id="gallery-carousel-real-scene">
181                <div class="video-pair active">
182                    <div class="video-container">
183                        <div class="video-caption">Generated HSI Video</div>
184                        <video preload="auto" controls muted loop playsinline height="100%">
185                            <source src="static/videos/garden_1_kling.mp4" type="video/mp4">
186                        </video>
187                    </div>
188                    <div class="video-container">
189                        <div class="video-caption">4D HSI Generated by ZeroHSI</div>
190                        <video preload="auto" controls muted loop playsinline height="100%">
191                            <source src="static/videos/garden_1_ours.mp4" type="video/mp4">
192                        </video>
193                    </div>
194                    <div class="pair-caption">Text Prompt: "The person is lifting a vase."</div>
195                </div>
196
197                <div class="video-pair">
198                    <div class="video-container">
199                        <div class="video-caption">Generated HSI Video</div>
200                        <video preload="auto" controls muted loop playsinline height="100%">
201                            <source src="static/videos/bicycle_kling.mp4" type="video/mp4">
202                        </video>
203                    </div>
204                    <div class="video-container">
205                        <div class="video-caption">4D HSI Generated by ZeroHSI</div>
206                        <video preload="auto" controls muted loop playsinline height="100%">
207                            <source src="static/videos/bicycle_ours.mp4" type="video/mp4">
208                        </video>
209                    </div>
210                    <div class="pair-caption">Text Prompt: "The person is sitting on the bench."</div>
211                </div>
212
213                <div class="video-pair">
214                    <div class="video-container">
215                        <div class="video-caption">Generated HSI Video</div>
216                        <video preload="auto" controls muted loop playsinline height="100%">
217                            <source src="static/videos/garden_0_kling.mp4" type="video/mp4">
218                        </video>
219                    </div>
220                    <div class="video-container">
221                        <div class="video-caption">4D HSI Generated by ZeroHSI</div>
222                        <video preload="auto" controls muted loop playsinline height="100%">
223                            <source src="static/videos/garden_0_ours.mp4" type="video/mp4">
224                        </video>
225                    </div>
226                    <div class="pair-caption">Text Prompt: "The person is watering flowers with a watering can."</div>
227                </div>
228
229                <div class="video-pair">
230                    <div class="video-container">
231                        <div class="video-caption">Generated HSI Video</div>
232                        <video preload="auto" controls muted loop playsinline height="100%">
233                            <source src="static/videos/room_kling.mp4" type="video/mp4">
234                        </video>
235                    </div>
236                    <div class="video-container">
237                        <div class="video-caption">4D HSI Generated by ZeroHSI</div>
238                        <video preload="auto" controls muted loop playsinline height="100%">
239                            <source src="static/videos/room_ours.mp4" type="video/mp4">
240                        </video>
241                    </div>
242                    <div class="pair-caption">Text Prompt: "The person is playing guitar while sitting on the sofa."</div>
243                </div>
244
245                <div class="video-pair">
246                    <div class="video-container">
247                        <div class="video-caption">Generated HSI Video</div>
248                        <video preload="auto" controls muted loop playsinline height="100%">
249                            <source src="static/videos/garden_2_kling.mp4" type="video/mp4">
250                        </video>
251                    </div>
252                    <div class="video-container">
253                        <div class="video-caption">4D HSI Generated by ZeroHSI</div>
254                        <video preload="auto" controls muted loop playsinline height="100%">
255                            <source src="static/videos/garden_2_ours.mp4" type="video/mp4">
256                        </video>
257                    </div>
258                    <div class="pair-caption">Text Prompt: "The person is operating a lawn mower."</div>
259                </div>
260
261                <div class="video-pair">
262                    <div class="video-container">
263                        <div class="video-caption">Generated HSI Video</div>
264                        <video preload="auto" controls muted loop playsinline height="100%">
265                            <source src="static/videos/truck_kling.mp4" type="video/mp4">
266                        </video>
267                    </div>
268                    <div class="video-container">
269                        <div class="video-caption">4D HSI Generated by ZeroHSI</div>
270                        <video preload="auto" controls muted loop playsinline height="100%">
271                            <source src="static/videos/truck_ours.mp4" type="video/mp4">
272                        </video>
273                    </div>
274                    <div class="pair-caption">Text Prompt: "The person is cleaning the car."</div>
275                </div>
276            </div>
277        </div>
278    </div>
279</section>
280<!--End real scene demos-->
281
282
283<!-- Long-term demos -->
284<section class="hero is-small">
285    <div class="hero-body">
286        <div class="container">
287            <h2 class="title is-2" style="margin-bottom: 0; margin-top: 3rem;">Generated Long-term Interactions</h2>
288            <div class="video-caption">Input Reconstructed 3D Scene</div>
289
290            <div class="video-gallery-buttons" id="gallery-buttons-long-term">
291                <button class="button is-primary" data-video-index="0">
292                    <img src="static/images/garden_0.png" alt="Garden Long 1">
293                </button>
294                <button class="button" data-video-index="1">
295                    <img src="static/images/bicycle_1.png" alt="Bicycle Long 1">
296                </button>
297                <button class="button" data-video-index="2">
298                    <img src="static/images/bicycle_2.png" alt="Bicycle Long 2">
299                </button>
300                <button class="button" data-video-index="3">
301                    <img src="static/images/bicycle_3.png" alt="Bicycle Long 3">
302                </button>
303            </div>
304
305            <div class="video-gallery-carousel" id="gallery-carousel-long-term">
306                <div class="video-single active">
307                    <div class="video-container">
308                        <div class="video-caption">Long-term HSI Generated by ZeroHSI</div>
309                        <video preload="auto" controls muted loop playsinline height="100%">
310                            <source src="static/videos/garden_long_1.mp4" type="video/mp4">
311                        </video>
312                    </div>
313                    <div class="pair-caption">
314                        <div class="prompt-container">
315                            <div class="prompt-line">
316                                <span class="prompt-label">Text Prompt 1:</span>
317                                <span class="prompt-text">"The person is walking forward to the table."</span>
318                            </div>
319                            <div class="prompt-line">
320                                <span class="prompt-label">Text Prompt 2:</span>
321                                <span class="prompt-text">"The person is watering flowers with a watering can."</span>
322                            </div>
323                            <div class="prompt-line">
324                                <span class="prompt-label">Text Prompt 3:</span>
325                                <span class="prompt-text">"The person is putting down the watering can on the table."</span>
326                            </div>
327                            <div class="prompt-line">
328                                <span class="prompt-label">Text Prompt 4:</span>
329                                <span class="prompt-text">"The person is leaning on the table."</span>
330                            </div>
331                        </div>
332                    </div>
333                </div>
334
335                <div class="video-single">
336                    <div class="video-container">
337                        <div class="video-caption">Long-term HSI Generated by ZeroHSI</div>
338                        <video preload="auto" controls muted loop playsinline height="100%">
339                            <source src="static/videos/bicycle_long_1.mp4" type="video/mp4">
340                        </video>
341                    </div>
342                    <div class="pair-caption">
343                        <div class="prompt-container">
344                            <div class="prompt-line">
345                                <span class="prompt-label">Text Prompt 1:</span>
346                                <span class="prompt-text">"The person is sitting on the bench."</span>
347                            </div>
348                            <div class="prompt-line">
349                                <span class="prompt-label">Text Prompt 2:</span>
350                                <span class="prompt-text">"The person is standing up."</span>
351                            </div>
352                            <div class="prompt-line">
353                                <span class="prompt-label">Text Prompt 3:</span>
354                                <span class="prompt-text">"The person is practice Tai Chi."</span>
355                            </div>
356                        </div>
357                    </div>
358                </div>
359
360                <div class="video-single">
361                    <div class="video-container">
362                        <div class="video-caption">Long-term HSI Generated by ZeroHSI</div>
363                        <video preload="auto" controls muted loop playsinline height="100%">
364                            <source src="static/videos/bicycle_long_2.mp4" type="video/mp4">
365                        </video>
366                    </div>
367                    <div class="pair-caption">
368                        <div class="prompt-container">
369                            <div class="prompt-line">
370                                <span class="prompt-label">Text Prompt 1:</span>
371                                <span class="prompt-text">"The person is sitting on the bench."</span>
372                            </div>
373                            <div class="prompt-line">
374                                <span class="prompt-label">Text Prompt 2:</span>
375                                <span class="prompt-text">"The person is standing up."</span>
376                            </div>
377                            <div class="prompt-line">
378                                <span class="prompt-label">Text Prompt 3:</span>
379                                <span class="prompt-text">"The person is practice Tai Chi."</span>
380                            </div>
381                        </div>
382                    </div>
383                </div>
384
385                <div class="video-single">
386                    <div class="video-container">
387                        <div class="video-caption">Long-term HSI Generated by ZeroHSI</div>
388                        <video preload="auto" controls muted loop playsinline height="100%">
389                            <source src="static/videos/bicycle_long_3.mp4" type="video/mp4">
390                        </video>
391                    </div>
392                    <div class="pair-caption">
393                        <div class="prompt-container">
394                            <div class="prompt-line">
395                                <span class="prompt-label">Text Prompt 1:</span>
396                                <span class="prompt-text">"The person is sitting on the bench."</span>
397                            </div>
398                            <div class="prompt-line">
399                                <span class="prompt-label">Text Prompt 2:</span>
400                                <span class="prompt-text">"The person is standing up."</span>
401                            </div>
402                            <div class="prompt-line">
403                                <span class="prompt-label">Text Prompt 3:</span>
404                                <span class="prompt-text">"The person is practice Tai Chi."</span>
405                            </div>
406                        </div>
407                    </div>
408                </div>
409            </div>
410        </div>
411    </div>
412</section>
413<!-- End long-term demos -->
414
415
416<!-- Comparison of dynamic scenario -->
417<section class="hero is-small">
418    <div class="hero-body">
419        <div class="container">
420            <h2 class="title is-2" style="margin-bottom: 0; margin-top: 3rem;">Generated Interactions on Synthetic Scenes with Objects</h2>
421            <div class="video-caption">Input Synthetic 3D Scene</div>
422
423            <div class="video-gallery-buttons" id="gallery-buttons-dynamic">
424                <button class="button is-primary" data-video-index="0">
425                    <img src="static/images/1_living_room_dynamic.png" alt="Living Room Dynamic">
426                </button>
427                <button class="button" data-video-index="1">
428                    <img src="static/images/2_gym_dynamic.png" alt="Gym Dynamic">
429                </button>
430                <button class="button" data-video-index="2">
431                    <img src="static/images/7_store_dynamic_0.png" alt="Store Dynamic 0">
432                </button>
433                <button class="button" data-video-index="3">
434                    <img src="static/images/7_store_dynamic_1.png" alt="Store Dynamic 1">
435                </button>
436            </div>
437  
438            <div class="video-gallery-carousel" id="gallery-carousel-dynamic">
439                <div class="video-triple active">
440                    <div class="video-container">
441                        <div class="video-caption">ZeroHSI (Ours)</div>
442                        <video preload="auto" controls muted loop playsinline height="100%">
443                            <source src="static/videos/1_living_room_dynamic_ours.mp4" type="video/mp4">
444                        </video>
445                    </div>
446                    <div class="video-container">
447                        <div class="video-caption">LINGO</div>
448                        <video preload="auto" controls muted loop playsinline height="100%">
449                            <source src="static/videos/1_living_room_dynamic_lingo.mp4" type="video/mp4">
450                        </video>
451                    </div>
452                    <div class="video-container">
453                        <div class="video-caption">CHOIS</div>
454                        <video preload="auto" controls muted loop playsinline height="100%">
455                            <source src="static/videos/1_living_room_dynamic_chois.mp4" type="video/mp4">
456                        </video>
457                    </div>
458                    <div class="pair-caption">Text Prompt: "The person is playing guitar while sitting on the sofa."</div>
459                </div>
460
461                <div class="video-triple">
462                    <div class="video-container">
463                        <div class="video-caption">ZeroHSI (Ours)</div>
464                        <video preload="auto" controls muted loop playsinline height="100%">
465                            <source src="static/videos/2_gym_dynamic_ours.mp4" type="video/mp4">
466                        </video>
467                    </div>
468                    <div class="video-container">
469                        <div class="video-caption">LINGO</div>
470                        <video preload="auto" controls muted loop playsinline height="100%">
471                            <source src="static/videos/2_gym_dynamic_lingo.mp4" type="video/mp4">
472                        </video>
473                    </div>
474                    <div class="video-container">
475                        <div class="video-caption">CHOIS</div>
476                        <video preload="auto" controls muted loop playsinline height="100%">
477                            <source src="static/videos/2_gym_dynamic_chois.mp4" type="video/mp4">
478                        </video>
479                    </div>
480                    <div class="pair-caption">Text Prompt: "The person is lifting weights."</div>
481                </div>
482
483                <div class="video-triple">
484                    <div class="video-container">
485                        <div class="video-caption">ZeroHSI (Ours)</div>
486                        <video preload="auto" controls muted loop playsinline height="100%">
487                            <source src="static/videos/7_store_dynamic_0_ours.mp4" type="video/mp4">
488                        </video>
489                    </div>
490                    <div class="video-container">
491                        <div class="video-caption">LINGO</div>
492                        <video preload="auto" controls muted loop playsinline height="100%">
493                            <source src="static/videos/7_store_dynamic_0_lingo.mp4" type="video/mp4">
494                        </video>
495                    </div>
496                    <div class="video-container">
497                        <div class="video-caption">CHOIS</div>
498                        <video preload="auto" controls muted loop playsinline height="100%">
499                            <source src="static/videos/7_store_dynamic_0_chois.mp4" type="video/mp4">
500                        </video>
501                    </div>
502                    <div class="pair-caption">Text Prompt: "The person is sliding while sitting on the chair."</div>
503                </div>
504
505                <div class="video-triple">
506                    <div class="video-container">
507                        <div class="video-caption">ZeroHSI (Ours)</div>
508                        <video preload="auto" controls muted loop playsinline height="100%">
509                            <source src="static/videos/7_store_dynamic_1_ours.mp4" type="video/mp4">
510                        </video>
511                    </div>
512                    <div class="video-container">
513                        <div class="video-caption">LINGO</div>
514                        <video preload="auto" controls muted loop playsinline height="100%">
515                            <source src="static/videos/7_store_dynamic_1_lingo.mp4" type="video/mp4">
516                        </video>
517                    </div>
518                    <div class="video-container">
519                        <div class="video-caption">CHOIS</div>
520                        <video preload="auto" controls muted loop playsinline height="100%">
521                            <source src="static/videos/7_store_dynamic_1_chois.mp4" type="video/mp4">
522                        </video>
523                    </div>
524                    <div class="pair-caption">Text Prompt: "The person is pushing shopping cart."</div>
525                </div>
526            </div>
527        </div>
528    </div>
529</section>
530<!-- End comparison of dynamic scenario -->
531
532
533<!-- Comparison of static scenario -->
534<section class="hero is-small">
535    <div class="hero-body">
536        <div class="container">
537            <h2 class="title is-2" style="margin-bottom: 0; margin-top: 3rem;">Generated Interactions on Synthetic Scenes</h2>
538            <div class="video-caption">Input Synthetic 3D Scene</div>
539            
540            <div class="video-gallery-buttons" id="gallery-buttons-static">
541                <button class="button is-primary" data-video-index="0">
542                    <img src="static/images/0_bedroom_static.png" alt="Bedroom">
543                </button>
544                <button class="button" data-video-index="1">
545                    <img src="static/images/1_living_room_static.png" alt="Living Room">
546                </button>
547                <button class="button" data-video-index="2">
548                    <img src="static/images/4_playground_static.png" alt="Playground">
549                </button>
550                <button class="button" data-video-index="3">
551                    <img src="static/images/7_store_static.png" alt="Store">
552                </button>
553            </div>
554
555            <div class="video-gallery-carousel" id="gallery-carousel-static">
556                <div class="video-triple active">
557                    <div class="video-container">
558                        <div class="video-caption">ZeroHSI (Ours)</div>
559                        <video preload="auto" controls muted loop playsinline height="100%">
560                            <source src="static/videos/0_bedroom_static_ours.mp4" type="video/mp4">
561                        </video>
562                    </div>
563                    <div class="video-container">
564                        <div class="video-caption">TRUMANS</div>
565                        <video preload="auto" controls muted loop playsinline height="100%">
566                            <source src="static/videos/0_bedroom_static_trumans.mp4" type="video/mp4">
567                        </video>
568                    </div>
569                    <div class="video-container">
570                        <div class="video-caption">LINGO</div>
571                        <video preload="auto" controls muted loop playsinline height="100%">
572                            <source src="static/videos/0_bedroom_static_lingo.mp4" type="video/mp4">
573                        </video>
574                    </div>
575                    <div class="pair-caption">Text Prompt: "The person is leaning on the ladder."</div>
576                </div>
577
578                <div class="video-triple">
579                    <div class="video-container">
580                        <div class="video-caption">ZeroHSI (Ours)</div>
581                        <video preload="auto" controls muted loop playsinline height="100%">
582                            <source src="static/videos/1_living_room_static_ours.mp4" type="video/mp4">
583                        </video>
584                    </div>
585                    <div class="video-container">
586                        <div class="video-caption">TRUMANS</div>
587                        <video preload="auto" controls muted loop playsinline height="100%">
588                            <source src="static/videos/1_living_room_static_trumans.mp4" type="video/mp4">
589                        </video>
590                    </div>
591                    <div class="video-container">
592                        <div class="video-caption">LINGO</div>
593                        <video preload="auto" controls muted loop playsinline height="100%">
594                            <source src="static/videos/1_living_room_static_lingo.mp4" type="video/mp4">
595                        </video>
596                    </div>
597                    <div class="pair-caption">Text Prompt: "The person is sitting on the sofa back."</div>
598                </div>
599
600                <div class="video-triple">
601                    <div class="video-container">
602                        <div class="video-caption">ZeroHSI (Ours)</div>
603                        <video preload="auto" controls muted loop playsinline height="100%">
604                            <source src="static/videos/4_playground_static_ours.mp4" type="video/mp4">
605                        </video>
606                    </div>
607                    <div class="video-container">
608                        <div class="video-caption">TRUMANS</div>
609                        <video preload="auto" controls muted loop playsinline height="100%">
610                            <source src="static/videos/4_playground_static_trumans.mp4" type="video/mp4">
611                        </video>
612                    </div>
613                    <div class="video-container">
614                        <div class="video-caption">LINGO</div>
615                        <video preload="auto" controls muted loop playsinline height="100%">
616                            <source src="static/videos/4_playground_static_lingo.mp4" type="video/mp4">
617                        </video>
618                    </div>
619                    <div class="pair-caption">Text Prompt: "The person is sliding down the slide."</div>
620                </div>
621
622                <div class="video-triple">
623                    <div class="video-container">
624                        <div class="video-caption">ZeroHSI (Ours)</div>
625                        <video preload="auto" controls muted loop playsinline height="100%">
626                            <source src="static/videos/7_store_static_ours.mp4" type="video/mp4">
627                        </video>
628                    </div>
629                    <div class="video-container">
630                        <div class="video-caption">TRUMANS</div>
631                        <video preload="auto" controls muted loop playsinline height="100%">
632                            <source src="static/videos/7_store_static_trumans.mp4" type="video/mp4">
633                        </video>
634                    </div>
635                    <div class="video-container">
636                        <div class="video-caption">LINGO</div>
637                        <video preload="auto" controls muted loop playsinline height="100%">
638                            <source src="static/videos/7_store_static_lingo.mp4" type="video/mp4">
639                        </video>
640                    </div>
641                    <div class="pair-caption">Text Prompt: "The person is picking out snacks on the shelf."</div>
642                </div>
643            </div>
644        </div>
645    </div>
646</section>
647<!-- End comparison of static scenario -->
648
649
650<!-- Paper abstract -->
651<section class="hero is-small">
652    <div class="hero-body">
653        <div class="container">
654            <h2 class="title is-2" style="margin-bottom: 0.5rem; margin-top: 3rem;">Abstract</h2>
655            <figure class="image" style="margin-bottom: 0rem;">
656                <figcaption style="margin-top: 0rem;">
657                    Human-scene interaction (HSI) generation is crucial for applications in embodied AI, virtual reality, and robotics. While existing methods can synthesize realistic human motions in 3D scenes and generate plausible human-object interactions, they heavily rely on datasets containing paired 3D scene and motion capture data, which are expensive and time-consuming to collect across diverse environments and interactions. We present ZeroHSI, a novel approach that enables zero-shot 4D human-scene interaction synthesis by integrating video generation and neural human rendering. Our key insight is to leverage the rich motion priors learned by state-of-the-art video generation models, which have been trained on vast amounts of natural human movements and interactions, and use differentiable rendering to reconstruct human-scene interactions. ZeroHSI can synthesize realistic human motions in both static scenes and environments with dynamic objects, without requiring any ground-truth motion data. We evaluate ZeroHSI on a curated dataset of different types of various indoor and outdoor scenes with different interaction prompts, demonstrating its ability to generate diverse and contextually appropriate human-scene interactions.
658                </figcaption>
659            </figure>
660        </div>
661    </div>
662</section>
663<!-- End paper abstract -->
664
665
666<!-- Method -->
667<section class="hero is-small">
668  <div class="hero-body">
669        <div class="container">
670            <h2 class="title is-2" style="margin-bottom: 1rem; margin-top: 3rem;">Method Overview</h2>
671            <figure class="image" style="margin-bottom: 0;">
672                <img src="static/images/method.png" alt="Research Poster">
673                <figcaption class="mt-3">
674                    ZeroHSI takes a 3D scene, an interactable object, a language description, and initial states as input to synthesize both human and dynamic object motion sequences. Our approach begins with HSI video generation conditioned on the rendered initial state and text prompt. Through differentiable neural rendering, we optimize per-frame camera pose, human pose parameters, and object 6D pose by minimizing the discrepancy between the rendered and generated reference videos.
675                </figcaption>
676            </figure>
677        </div>
678  </div>
679</section>
680<!--End method -->
681
682
683<!-- Dataset -->
684<section class="hero is-small">
685    <div class="hero-body">
686        <div class="container">
687            <h2 class="title is-2" style="margin-bottom: 1rem; margin-top: 3rem;">The AnyInteraction Dataset</h2>
688            <figure class="grid-container" style="margin-bottom: 0;">
689                <!-- Small figures -->
690                <div class="subfigure">
691                    <img src="static/images/0_bedroom.png" alt="Figure 00">
692                    <div class="subfigure-caption">Bedroom</div>
693                </div>
694                <div class="subfigure">
695                    <img src="static/images/1_living_room.png" alt="Figure 01">
696                    <div class="subfigure-caption">Living Room</div>
697                </div>
698                <div class="subfigure">
699                    <img src="static/images/2_gym.png" alt="Figure 02">
700                    <div class="subfigure-caption">Gym</div>
701                </div>
702                <div class="subfigure">
703                    <img src="static/images/3_bar.png" alt="Figure 03">
704                    <div class="subfigure-caption">Bar</div>
705                </div>
706                <div class="subfigure">
707                    <img src="static/images/4_playground.png" alt="Figure 04">
708                    <div class="subfigure-caption">Playground</div>
709                </div>
710                <div class="subfigure">
711                    <img src="static/images/5_greenhouse.png" alt="Figure 05">
712                    <div class="subfigure-caption">Greenhouse</div>
713                </div>
714                <div class="subfigure">
715                    <img src="static/images/6_cafe.png" alt="Figure 06">
716                    <div class="subfigure-caption">Cafe</div>
717                </div>
718                <div class="subfigure">
719                    <img src="static/images/7_store.png" alt="Figure 07">
720                    <div class="subfigure-caption">Store</div>
721                </div>
722                <div class="subfigure">
723                    <img src="static/images/8_garden.png" alt="Figure 08">
724                    <div class="subfigure-caption">Garden</div>
725                </div>
726                <div class="subfigure">
727                    <img src="static/images/9_bicycle.png" alt="Figure 09">
728                    <div class="subfigure-caption">Bicycle</div>
729                </div>
730                <div class="subfigure">
731                    <img src="static/images/10_room.png" alt="Figure 10">
732                    <div class="subfigure-caption">Room</div>
733                </div>
734                <div class="subfigure">
735                    <img src="static/images/11_truck.png" alt="Figure 11">
736                    <div class="subfigure-caption">Truck</div>
737                </div>
738
739                <!-- Enlarged figures (hidden by default) -->
740                <div id="enlarged-figure-0" class="enlarged-figure">
741                    <img src="static/images/0_bedroom.png" alt="Big Figure 00">
742                </div>
743                <div id="enlarged-figure-1" class="enlarged-figure">
744                    <img src="static/images/1_living_room.png" alt="Big Figure 01">
745                </div>
746                <div id="enlarged-figure-2" class="enlarged-figure">
747                    <img src="static/images/2_gym.png" alt="Big Figure 02">
748                </div>
749                <div id="enlarged-figure-3" class="enlarged-figure">
750                    <img src="static/images/3_bar.png" alt="Big Figure 03">
751                </div>
752                <div id="enlarged-figure-4" class="enlarged-figure">
753                    <img src="static/images/4_playground.png" alt="Big Figure 04">
754                </div>
755                <div id="enlarged-figure-5" class="enlarged-figure">
756                    <img src="static/images/5_greenhouse.png" alt="Big Figure 05">
757                </div>
758                <div id="enlarged-figure-6" class="enlarged-figure">
759                    <img src="static/images/6_cafe.png" alt="Big Figure 06">
760                </div>
761                <div id="enlarged-figure-7" class="enlarged-figure">
762                    <img src="static/images/7_store.png" alt="Big Figure 07">
763                </div>
764                <div id="enlarged-figure-8" class="enlarged-figure">
765                    <img src="static/images/8_garden.png" alt="Big Figure 08">
766                </div>
767                <div id="enlarged-figure-9" class="enlarged-figure">
768                    <img src="static/images/9_bicycle.png" alt="Big Figure 09">
769                </div>
770                <div id="enlarged-figure-10" class="enlarged-figure">
771                    <img src="static/images/10_room.png" alt="Big Figure 10">
772                </div>
773                <div id="enlarged-figure-11" class="enlarged-figure">
774                    <img src="static/images/11_truck.png" alt="Big Figure 11">
775                </div>
776
777                <!-- Main caption -->
778                <figcaption class="main-caption mt-3">
779                    We curate the AnyInteraction dataset to evaluate zero-shot HSI synthesis. The dataset encompasses 12 diverse scenes (7 indoor and 5 outdoor) from various sources including TRUMANS, public 3D assets, and reconstructed real scenes from Mip-NeRF 360 and Tanks and Temples datasets. It features 7 types of rigid dynamic objects (Guitar, Barbell, Watering Can, Office Chair, Shopping Cart, Vase, and Mower), supporting 22 evaluation instances that include 13 static and 9 dynamic object interaction.
780                </figcaption>
781
782            </figure>
783        </div>
784    </div>
785</section>
786<!--End Dataset -->
787
788
789<!--BibTex citation -->
790 <section class="hero is-small">
791    <div class="hero-body">
792        <div class="container">
793            <h2 class="title is-2" style="margin-bottom: 1rem; margin-top: 3rem;">BibTex</h2>
794            <figure class="image" style="margin-bottom: 0;">
795                <pre><code>@inproceedings{li2026zerohsi,
796  title={Zerohsi: Zero-shot 4d human-scene interaction by video generation},
797  author={Li, Hongjie and Yu, Hong-Xing and Li, Jiaman and Wu, Jiajun},
798  booktitle={International Conference on 3D Vision (3DV)},
799  year={2026}
800}</code></pre>
801            </figure>
802        </div>
803    </div>
804</section>
805<!--End BibTex citation -->
806
807
808<footer class="footer">
809    <div class="container">
810        <div class="columns is-centered">
811            <div class="column is-8">
812                <div class="content has-text-centered">
813                    <p>
814                        This page was built using the <a href="https://github.com/eliahuhorwitz/Academic-project-page-template" target="_blank">Academic Project Page Template</a> which was adopted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a> project page.
815                        <br>
816                        This website is licensed under a <a rel="license"  href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative Commons Attribution-ShareAlike 4.0 International License</a>.
817                    </p>
818                </div>
819            </div>
820        </div>
821    </div>
822</footer>
823
824
825</body>
826</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.