1<!DOCTYPE html> 2<html> 3<head> 4 <meta charset="utf-8"> 5 <meta name="description" 6 content="GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration"> 7 <meta name="keywords" content="GUI-Bee, Q-ICRL, GUI screen action grounding"> 8 <meta name="viewport" content="width=device-width, initial-scale=1"> 9 <title>GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration</title> 10 11 <!-- Global site tag (gtag.js) - Google Analytics -->
vendor: 66 bytes, lines 11-12
11 12 <script async src="https://www.googletagmanager.com/gtag/js?id=
12G-PYVRSFMDRL
vendor: 14 bytes, lines 12-13
12"></script> 13
13<script> 14
vendor: 155 bytes, lines 14-22
14window.dataLayer = window.dataLayer || []; 15 16 function gtag() { 17 dataLayer.push(arguments); 18 } 19 20 gtag('js', new Date()); 21 22 gtag('config', '
22G-PYVRSFMDRL
vendor: 6 bytes, lines 22-23
22'); 23
23</script>
23 24 25 <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" 26 rel="stylesheet"> 27 28 <link rel="stylesheet" href="./static/css/bulma.min.css"> 29 <link rel="stylesheet" href="./static/css/bulma-carousel.min.css"> 30 <link rel="stylesheet" href="./static/css/bulma-slider.min.css"> 31 <link rel="stylesheet" href="./static/css/fontawesome.all.min.css"> 32 <link rel="stylesheet" 33 href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> 34 <link rel="icon" href="./static/images/icon.png"> 35 <link rel="stylesheet" href="./static/css/index.css"> 36 <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0-beta3/css/all.min.css"> 37 38 39
39<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
39 40
40<script defer src="./static/js/fontawesome.all.min.js"></script>
40 41
41<script src="./static/js/bulma-carousel.min.js"></script>
41 42
42<script src="./static/js/bulma-slider.min.js"></script>
42 43
43<script src="./static/js/index.js"></script>
43 44 <!--
44<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
44 --> 45<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0-beta3/css/all.min.css"> 46 47 48 49 50 51 52 <style> 53 54 .image-row { 55 display: flex; 56 justify-content: center; 57 align-items: center; 58 gap: 20px; /* Adjust the gap between images as needed */ 59 } 60 .image-container { 61 position: relative; 62 display: inline-block; 63 text-align: center; 64 width: 45%; /* Adjust the width of each container as needed */ 65 66 } 67 .image-container img { 68 width: 100%; /* Ensures the image fills the container width */ 69 } 70 .image-container p { 71 font-family: "Times New Roman", Times, serif; 72 font-weight: bold; 73 } 74 75 .image-container p { 76 font-family: "Times New Roman", Times, serif; 77 font-weight: bold; 78 } 79 .has-text-centered { 80 text-align: center; 81 font-size: 16px; 82 } 83 .has-text-left { 84 font-size: 16px; 85 } 86 87 #image-container { 88 position: relative; 89 display: inline-block; 90 } 91 #startButton { 92 display: block; 93 margin: 20px auto; 94 padding: 10px 20px; 95 font-size: 16px; 96 cursor: pointer; 97 } 98 #imageModal { 99 display: none; /* Hidden by default */ 100 position: fixed; 101 top: 0; 102 left: 0; 103 width: 100%; 104 height: 100%; 105 overflow: auto; 106 background-color: rgb(0,0,0); 107 background-color: rgba(0,0,0,0.4); 108 z-index: 1; 109 } 110 #imageModalContent { 111 position: relative; 112 margin: auto; 113 padding: 0px; 114 width: 360px; 115 max-width: 360px; 116 background-color: #fff; 117 border-radius: 10px; 118 } 119 #imageContainer { 120 position: relative; 121 display: inline-block; 122 width: 100%; 123 height: auto; 124 } 125 .dot { 126 position: absolute; 127 width: 10px; 128 height: 10px; 129 background-color: rgba(255, 0, 0, 0.5); /* Red with 50% opacity */ 130 border-radius: 50%; 131 } 132 .bbox { 133 position: absolute; 134 border: 4px solid red; /* Red border for bbox */ 135 } 136 .bbox-large { 137 position: absolute; 138 border: 8px solid blue; /* Green border for bbox_large */ 139 } 140 crowd-alert { 141 display: none !important; 142 } 143 </style> 144 145</head> 146<body> 147 148<section class="hero"> 149 <div class="hero-body"> 150 <div class="container is-max-desktop"> 151 <div class="columns is-centered"> 152 <div class="column has-text-centered"> 153 <div style="display: flex; align-items: center; justify-content: center;"> 154 <!-- <img src="./static/image/worldevallogo.png" style="width:6em;vertical-align: middle" alt="Logo"/> --> 155 156 <h1 class="title is-1 publication-title" style="display: inline-block;">GUI-Bee<img style="width:2em" src="./static/image/bee3_AdobeStock_422301907.jpg" style="width:6em;vertical-align: middle" alt="Logo"/>: 157 <br> Align GUI Action Grounding to Novel Environments via Autonomous Exploration</h1> 158 </div> 159 <div class="is-size-5 publication-authors"> 160 <span class="author-block"> 161 <a href="http://www.yfan.site/">Yue Fan</a><sup style="color: #FFB6C1;">1</sup>,</span> 162 <span class="author-block"> 163 <a href="https://research.adobe.com/person/handong-zhao/">Handong Zhao</a><sup style="color: #90EE90;">2</sup>,</span> 164 <span class="author-block"> 165 <a href="https://research.adobe.com/person/ruiyi-zhang/">Ruiyi Zhang</a><sup style="color: #90EE90;">2</sup>,</span> 166 <span class="author-block"> 167 <a href="https://research.adobe.com/person/yu-shen/">Yu Shen</a><sup style="color: #90EE90;">2</sup>,</span> 168
169 <span class="author-block"> 170 <a href="https://eric-xw.github.io/">Xin Eric Wang</a><sup style="color: #FFB6C1;">*1</sup>,</span> 171 <span class="author-block"> 172 <a href="https://research.adobe.com/person/gangwu/">Gang Wu</a><sup style="color: #90EE90;">*2</sup>,</span> 173 </div> 174 <div class="is-size-5 publication-authors"> 175 <span class="author-block"><sup style="color: #FFB6C1;">1</sup>UC Santa Cruz,</span> 176 <span class="author-block"><sup style="color: #90EE90;">2</sup>Adobe Research</span> 177 </div> 178 179 <br> 180 <div class="is-size-5 publication-authors"> 181 <!--<span class="author-block">â Corresponding to:</span> 182 <span class="author-block"><a href="mailto:[email protected]">[email protected]</a>,</span> 183 <span class="author-block"><a href="mailto:[email protected]">[email protected]</a></span>--> 184 <sup >*</sup> Co-advising. This work was partly performed when the first author interned at Adobe Research. 185 </div> 186 187 <div class="column has-text-centered"> 188 <div class="publication-links"> 189 <!-- PDF Link. --> 190 191 <span class="link-block"> 192 <a href="https://arxiv.org/pdf/2501.13896" 193 class="external-link button is-normal is-rounded is-dark"> 194 <span class="icon"> 195 <i class="fas fa-file-pdf"></i> 196 </span> 197 <span>Paper</span> 198 </a> 199 </span> 200 <!-- <span class="link-block"> 201 <a href="https://github.com/eric-ai-lab/Screen-Point-and-Read" 202 class="external-link button is-normal is-rounded is-dark"> 203 <span class="icon"> 204 <i class="fab fa-github"></i> 205 </span> 206 <span>Code</span> 207 </a> 208 </span> 209 <span class="link-block"> 210 <a href="https://huggingface.co/datasets/yfan1997/ScreenPR" 211 class="external-link button is-normal is-rounded is-dark"> 212 <span class="icon"> 213 <i class="fas fa-database"></i> 214 </span> 215 <span>Benchmark with dataviewer</span> 216 </a> 217 </span> --> 218 <!-- Code Link. --> 219 220 221 222 </div> 223 224 </div> 225 </div> 226 </div> 227 </div> 228 </div> 229</section> 230 231 232 233<section class="section"> 234 <div class="container is-max-desktop"> 235 <!-- Abstract. --> 236 <div class="columns is-centered has-text-centered"> 237 <div class="column is-four-fifths"> 238 <h2 class="title is-3">Abstract</h2> 239 <div class="content has-text-justified"> 240 <p> 241 Graphical User Interface (GUI) action grounding is a critical step in GUI automation that maps language instructions to actionable elements on GUI screens. Most recent works of GUI action grounding leverage large GUI datasets to fine-tune MLLMs. However, the fine-tuning data always covers limited GUI environments, and we find the performance of the resulting model deteriorates in novel environments. We argue that the GUI grounding models should be further aligned to the novel environments to reveal their full potential, when the inference is known to involve novel environments, i.e., environments not used during the previous fine-tuning. To realize this, we first propose GUI-Bee, an MLLM-based autonomous agent, to collect high-quality, environment-specific data through exploration and then continuously fine-tune GUI grounding models with the collected data. Our agent leverages a novel Q-value-Incentive In-Context Reinforcement Learning (Q-ICRL) method to optimize exploration efficiency and data quality. Additionally, we introduce NovelScreenSpot, a benchmark for testing how well the data can help align GUI action grounding models to novel environments and demonstrate the effectiveness of data collected by GUI-Bee in the experiments. Furthermore, we conduct an ablation study to validate the Q-ICRL method in enhancing the efficiency of GUI-Bee. </p> 242 </div> 243 </div> 244 </div> 245 </div> 246</section> 247 248 249<section class="hero teaser"> 250 <div class="container is-max-desktop"> 251 <div class="hero-body"> 252 <h2 class="subtitle has-text-centered"> 253 <div class="image-row"> 254 255 <div class="image-container" style="width: 100%;"> 256 <img src="./static/image/web-f1.png" alt="figure1" > 257 </div> 258 259 </div> 260 <p style="font-family:Times New Roman; text-align: left;"> 261 <!-- TL;DR: <br> 262 Prior works convert pre-trained MLLMs to GUI grounding models by fine-tuning with large amount of data. <br> 263 Our work align those GUI grounding models to novel GUI environments, boosting their performance for deployment scenarios.<br><br> 264 265 Long version: <br> --> 266 App owners may want to deploy GUI action models in their own specific GUI environments, potentially novel environments on which the models have not been trained previously. 267 We propose to build <a style="color: #ffc800;">environment-aligned GUI action grounding model</a>, based on <a style="color: #b5b5b5;">GUI grounding models from prior works</a>. The aligned models are specifically fine-tuned to the novel environments to achieve better performance. 268 Our proposed alignment process includes first exploring the specific novel environment with the GUI-Bee agent to generate the exploration graph and then fine-tuning the model with the data from the exploration graph. 269 In the <a style="color: #d4a3cc;">inference example at the bottom</a>, the models encounter a query requiring knowledge of an environment-specific action outcome, which highlights the importance of the proposed alignment process. 270 271 272 273 274 </p> 275 276 </h2> 277 278 </div> 279 </div> 280</section> 281 282 283 284<section class="section"> 285 <div class="container is-max-desktop"> 286 <div class="columns is-centered has-text-centered"> 287 <div class="column is-five-fifths"> 288 289 <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>Overview</h2> 290 </div> 291 </div> 292 293 <!-- First Row --> 294 <div class="columns is-fullwidth"> 295 296 297 <div class="image-container" style="text-align: center; margin-left: 25%;"> 298 <img src="./static/image/web-f2.png" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 299 </div> 300 </div> 301 <br> 302 <h3 style="text-align: left; margin-left: 20%;"> 303 We propose aligning GUI grounding models to novel environments, which includes<br> âµ exploring the specific GUI environment <br>â¶ generating high-quality data from the exploration and<br> â· continously fine-tuning the model with the collected data. 304 </h3> 305 <br> 306 </div> 307</section> 308 309<section class="section"> 310 <div class="container is-max-desktop"> 311 <div class="columns is-centered has-text-centered"> 312 <div class="column is-five-fifths"> 313 314 <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>Autonomous Exploration with GUI-Bee and Q-ICRL </h2> 315 </div> 316 </div> 317 <!-- First Row --> 318 <div class="columns is-fullwidth"> 319 <div class="column is-3"> 320 <h3 style="text-align: left; margin-left: 40%;"><strong><u>·Goal:</u></strong></h3> 321 </div> 322 <div class="column is-6"> 323 <div class="image-container" style="text-align: center; margin-left: 20%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 324 </div> 325 </div> 326 </div> 327 328 <h3 style="text-align: left; margin-left: 15%; margin-right: 15%;"> 329 Autonomously predict and execute minimal numbers of actions to reach a sequence of GUI screens that are as diverse as possible. 330 </h3> 331 <br><br> 332 333 <!-- First Row --> 334 <div class="columns is-fullwidth"> 335 <div class="column is-3"> 336 <h3 style="text-align: left; margin-left: 40%;"><strong><u>·Challenges:</u></strong></h3> 337 </div> 338 <div class="column is-6"> 339 <div class="image-container" style="text-align: center; margin-left: 20%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 340 </div> 341 </div> 342 </div> 343 344 <h3 style="text-align: left; margin-left: 15%; margin-right: 15%;"> 345 1. Noisy Action Space and Action Validity: The GUI environment usually provides a set of candidate actions, but many are invalid, i.e. non-executable elements. 346 <br> 347 2. Uncertain Screen Transitions: The outcomes of GUI actions are unknown, and the screen transitions caused by actions are irreversible 348 </h3> 349 <br><br> 350 351 352 <!-- Second Row --> 353 <div class="columns is-fullwidth"> 354 <div class="column is-3"> 355 <h3 style="text-align: left; margin-left: 40%;"><strong><u>·Our solution <br>Q-ICRL:</u></strong> <br>(Refer to our paper for psudo code and more)</h3> 356 </div> 357 <div class="column is-6"> 358 <div class="image-container" style="text-align: center; width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5); display: block; margin: 0 auto;"> 359 <video controls autoplay loop> 360 <source src="./static/image/paper_1.mp4" type="video/mp4"> 361 </video> 362 </div> 363 </div> 364 </div> 365 <h3 style="text-align: left; margin-left: 15%; margin-right: 15%;"> 366 We equip the GUI-Bee with a novel Q-value-Incentive In-Context Reinforcement Learning (Q-ICRL) algorithm. It is a training-free method designed to maximize the exploration efficie
366ncy by treating the process as a Markov Decision Problem. It uses a memory-based Q-value function to quantify the outcome of actions and employs in-context learning with a Multimodal Large Language Model (MLLM) to select the most promising action. This approach balances exploration and exploitation. </h3> 367 <br><br> 368 369 370 <!-- third Row --> 371 <div class="columns is-fullwidth"> 372 <div class="column is-3"> 373 <h3 style="text-align: left; margin-left: 40%;"><strong><u>·Exploration <br>Result:</u></strong></h3> 374 </div> 375 376 <div class="column is-5"> 377 <img src="./static/image/web-f4.png" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 378 <br><br> 379 The exploration process of GUI-Bee builds exploration graph of visited screens and actions. 380 Left: an example of the exploration graph showing screens connected by actions. Right: A zoomed-in view of the graph with examples of two GUI screens and some explored and unexplored actions (ever/never selected during the exploration). 381 </div> 382 383 <div class="column is-2"> 384 <img src="./static/image/paper_2.gif" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 385 <br><br> 386 Illustration of the generation process of an exploration graph. 387 </div> 388 </div> 389 390 391 </div> 392</section> 393 394 395<section class="section"> 396 <div class="container is-max-desktop"> 397 <div class="columns is-centered has-text-centered"> 398 <div class="column is-five-fifths"> 399 400 <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>Autonomous Data Annotation with GUI-Bee</h2> 401 </div> 402 </div> 403 404 <!-- <div class="columns is-centered has-text-centered"> 405 <div class="image-container " style="width: 70%;box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 406 <img src="./static/image/web-f6.png" alt="figure2"> 407 </div> 408 409 </div> 410 <h3 style="text-align: left;"> 411 Once a new edge is added to the exploration graph during the exploration, we send this information to the MLLM (GPT-4o) to generate a list of action grounding queries (u^t) for the target element (e^t). The generation process uses a carefully crafted prompt, designed to ensure the queries cover both System 1 (focused on current screen content) and System 2 (anticipating interaction outcomes) grounding challenges. 412 </h3> 413 414 <br> 415 <br> --> 416 417 <div class="columns is-centered has-text-centered"> 418 <div class="image-container " style="width: 70%;box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 419 <img src="./static/image/web-f5.png" alt="figure2"> 420 </div> 421 422 </div> 423 <h3 style="text-align: left;"> 424 Once a new edge is added to the exploration graph during the exploration, we send the pair of screens connected by the edge to the MLLM (GPT-4o) to generate a list of action grounding queries (u^t) for the target element (e^t). The generation process uses a carefully crafted prompt, designed to ensure the queries cover both System 1 (focused on current screen content) and System 2 (anticipating interaction outcomes) grounding challenges. 425<!-- The full version of the text prompt for the MLLM is provided in Figure \ref{prompt1}. We also show an example of the input images in Figure \ref{prompt1_output} along with the GUI action grounding queries $u^t$ in the corresponding output. --> 426 </h3> 427 428 </div> 429</section> 430 431 432 433<section class="section"> 434 <div class="container is-max-desktop"> 435 <div class="columns is-centered has-text-centered"> 436 <div class="column is-five-fifths"> 437 438 <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>NovelSreenSpot Benchmark</h2> 439 </div> 440 </div> 441 442 <div class="columns is-centered has-text-centered"> 443 <div class="image-container " style="width: 70%;box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 444 <img src="./static/image/web-f7.png" alt="figure2"> 445 </div> 446 447 </div> 448 <h3 style="text-align: left; "> 449 NovelScreenSpot is a new human-annotated benchmark for evaluating GUI action grounding models in five diverse web GUI environments and model performance improvements after they are continuously fine-tuned with
449new data. Unlike existing benchmarks that emphasize diversity across many environments, NovelScreenSpot provides a greater data variation within each environment and includes a large number of queries about action outcomes, requiring environment-specific knowledge. 450 <!-- The full version of the text prompt for the MLLM is provided in Figure \ref{prompt1}. We also show an example of the input images in Figure \ref{prompt1_output} along with the GUI action grounding queries $u^t$ in the corresponding output. --> 451 </h3> 452 <br> 453 <div class="columns is-centered has-text-centered"> 454 <div class="column is-six-fifths"> 455 456 457 <div class="gallery"> 458 <img src="./static/image/e1.png" alt="Image 11" > 459 <img src="./static/image/e2.png" alt="Image 12" > 460 <img src="./static/image/e3.png 461 " alt="Image 13" > 462 <img src="./static/image/e4.png 463 " alt="Image 14" > 464 <img src="./static/image/e5.png 465 " alt="Image 33" > 466 467 468 469 </div> 470 471 472 <h3 class="subtitle has-text-centered"> 473 <h3 style="text-align: left;"> Examples of the NovelScreenSpot benchmark. Each figure illustrates the GUI screen, the A11y string, the corresponding query, and the ground truth target element. We show one example for each environment in the benchmark. </p> 474 </h3> 475 </div> 476 </div> 477</section> 478 479 480 481<section class="section"> 482 <div class="container is-max-desktop"> 483 <div class="columns is-centered has-text-centered"> 484 <div class="column is-five-fifths"> 485 486 <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>Experiments </h2> 487 </div> 488 </div> 489 490 491 <div class="columns is-centered has-text-centered"> 492 <div class="image-container " style="width: 70%;box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 493 <img src="./static/image/web-f6.png" alt="figure2"> 494 </div> 495 496 </div> 497 <h3 style="text-align: left; "> 498 We employ the GUI-Bee agent to explore the five environments in NovelScreenSpot, conducting up to 400 exploration steps per environment. To ensure diverse screen data, the exploration is repeated three times per environment at varying screen resolutions. The resulting exploration statistics are summarized in Table. We leverage GPT-4o as the MLLM for GUI-Bee and the cost for the exploration is under $50 per environment. 499 </h3> 500 501 <br> 502 503 504 <!-- third Row --> 505 <div class="columns is-fullwidth"> 506 <div class="column is-8"> 507 <img src="./static/image/web-f8.png" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 508 <br><br> 509 Using the data generated from GUI-Bee exploration, we continuously fine-tune three GUI grounding models for each environments explored. Table here shows the benchmarking GUI grounding models on the NovelScreenSpot and Eventbrite environment of Multimodal-Mind2Web benchmark. This table shows the model accuracy and, in parentheses, the absolute improvement over the vanilla models after the models are continuously fine-tuned. The results demonstrate that our GUI-Bee model significantly improves the performance of GUI action grounding models in novel environments. 510 </div> 511 512 <div class="column is-4"> 513 <img src="./static/image/web-f9.png" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);"> 514 <br><br> 515 We propose Depth-fixed DOM Diversity Counts (D3C) to evaluate the coverage of the exploration process. This result shows the mean and standard deviation of D3C at various exploration steps across three runs in three environments. GUI-Bee agent demonstrates a wider exploration coverage. 516 </div> 517 </div> 518 </div> 519</section> 520 521<!-- 522<section class="section"> 523 <div class="container is-max-desktop"> 524 <div class="columns is-centered has-text-centered"> 525 <div class="column is-five-fifths"> 526 <h2 class="title is-3">Screen Point-and-Read (ScreenPR) Benchmark <a href="https://huggingface.co/datasets/yfan1997/ScreenPR" 527 class="external-link button is-normal is-rounded is-dark">
528 <span class="icon"> 529 <i class="fas fa-database"></i> 530 </span> 531 </a></h2> 532 </div> 533 </div> 534 <div class="image-row"> 535 <div class="gallery"> 536 <img src="./static/image/web_2b491da3-f798-4866-b002-a25ab7d15263.png" alt="Image 11" style="height: 100px"> 537 <img src="./static/image/web_2f3df3a1-d287-4457-91b3-a672d0f72f84.png" alt="Image 12" style="height: 100px"> 538 <img src="./static/image/web_352ae757-fdf0-4d02-9167-c24a42629a60.png 539" alt="Image 13" style="height: 100px"> 540 <img src="./static/image/web_89d60b8b-3f6d-4160-93d4-1791d4899455.png 541" alt="Image 14" style="height: 100px"> 542 <img src="./static/image/frame_73.jpg 543" alt="Image 21" style="height: 100px"> 544 <img src="./static/image/frame_81.jpg 545" alt="Image 22" style="height: 100px"> 546 <img src="./static/image/frame_162.jpg 547" alt="Image 3" style="height: 100px"> 548 <img src="./static/image/frame_246.jpg" alt="Image 24" style="height: 100px"> 549 <img src="./static/image/mobile_d8a4069b-ec35-4a62-a3cd-5b5e7d95b50c.png" alt="Image 31" style="height: 100px"> 550 <img src="./static/image/mobile_4e17db8b-85df-4278-ba0a-9f03e5ca85da.png 551" alt="Image 32" style="height: 100px"> 552 <img src="./static/image/mobile_5c9fb5f1-7d84-4283-8ba4-7821a9545c95.png 553" alt="Image 33" style="height: 100px"> 554 <img src="./static/image/mobile_6cc4e2c1-b5bd-4347-bee4-63a1217d052c.png 555" alt="Image 34" style="height: 100px"> 556 <img src="./static/image/and_more.png 557" alt="Image 35" style="height: 100px"> 558 559 560 561 </div> 562 <div class="image-container" style="width: 150%;"> 563 <img src="./static/image/benchmark_stats.png" alt="stats"> 564 565 </div> 566 567 </div> 568 <h3 class="subtitle has-text-centered"> 569 <p style="font-family:Times New Roman">Screen Point-and-Read (ScreenPR) benchmark is introduced to rigorously evaluate our ToL agent on the ScreenPR task, where descriptions are required based on user-indicated points on screenshots. It covers a diverse domain of GUIs. Each screenshot is annotated with about 2 target points and their corresponding local regions on the screenshot.</p> 570 </h3> 571 572</section> --> 573 574 575 576 577 578 579 580 581<section class="section" id="BibTeX"> 582 <div class="container is-max-desktop content"> 583 <h2 class="title">BibTeX</h2> 584 <pre><code> 585 @misc{fan2025guibeealignguiaction, 586 title={GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration}, 587 author={Yue Fan and Handong Zhao and Ruiyi Zhang and Yu Shen and Xin Eric Wang and Gang Wu}, 588 year={2025}, 589 eprint={2501.13896}, 590 archivePrefix={arXiv}, 591 primaryClass={cs.CL}, 592 url={https://arxiv.org/abs/2501.13896}, 593 } 594 </code></pre> 595 </div> 596</section> 597 598 599 600 601<footer class="footer"> 602 <div class="container"> 603 <div class="columns is-centered"> 604 <div class="column is-8"> 605 <div class="content"> 606 <p> 607 This website is adapted from <a rel="license" 608 href="https://github.com/nerfies/nerfies.github.io">Nerfies</a> and <a rel="license" 609 href="https://gligen.github.io/">GLIGEN</a>, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. 610 This website is licensed under a <a rel="license" 611 href="http://creativecommons.org/licenses/by-sa/4.0/">Creative 612 Commons Attribution-ShareAlike 4.0 International License</a>. 613 </p> 614 </div> 615 </div> 616 </div> 617 </div> 618 619</footer> 620 621 622<!-- Additional libraries for parsing CSV -->
623<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-csv/1.0.21/jquery.csv.min.js"></script>
623 624 625</body> 626</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.