PageSourceSearch

https://gui-bee.github.io/

html gui-bee.github.io collected 2026-10-03 09:15:41 UTC 28,710 bytes, 626 lines download raw bytes

1<!DOCTYPE html>
2<html>
3<head>
4  <meta charset="utf-8">
5  <meta name="description"
6        content="GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration">
7  <meta name="keywords" content="GUI-Bee, Q-ICRL, GUI screen action grounding">
8  <meta name="viewport" content="width=device-width, initial-scale=1">
9  <title>GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration</title>
10
11  <!-- Global site tag (gtag.js) - Google Analytics -->
vendor: 66 bytes, lines 11-12
11
12  <script async src="https://www.googletagmanager.com/gtag/js?id=
12G-PYVRSFMDRL
vendor: 14 bytes, lines 12-13
12"></script>
13  
13<script>
14    
vendor: 155 bytes, lines 14-22
14window.dataLayer = window.dataLayer || [];
15
16    function gtag() {
17      dataLayer.push(arguments);
18    }
19
20    gtag('js', new Date());
21
22    gtag('config', '
22G-PYVRSFMDRL
vendor: 6 bytes, lines 22-23
22');
23  
23</script>
23
24
25  <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
26  rel="stylesheet">
27
28  <link rel="stylesheet" href="./static/css/bulma.min.css">
29  <link rel="stylesheet" href="./static/css/bulma-carousel.min.css">
30  <link rel="stylesheet" href="./static/css/bulma-slider.min.css">
31  <link rel="stylesheet" href="./static/css/fontawesome.all.min.css">
32  <link rel="stylesheet"
33  href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
34  <link rel="icon" href="./static/images/icon.png">
35  <link rel="stylesheet" href="./static/css/index.css">
36  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0-beta3/css/all.min.css">
37
38  
39  
39<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
39
40  
40<script defer src="./static/js/fontawesome.all.min.js"></script>
40
41  
41<script src="./static/js/bulma-carousel.min.js"></script>
41
42  
42<script src="./static/js/bulma-slider.min.js"></script>
42
43  
43<script src="./static/js/index.js"></script>
43
44  <!-- 
44<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
44 -->
45<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0-beta3/css/all.min.css">
46
47
48
49
50
51
52  <style>
53	    
54		.image-row {
55            display: flex;
56            justify-content: center;
57            align-items: center;
58            gap: 20px; /* Adjust the gap between images as needed */
59        }
60        .image-container {
61	        position: relative;            
62	        display: inline-block;
63            text-align: center;
64            width: 45%; /* Adjust the width of each container as needed */
65            
66        }
67        .image-container img {
68            width: 100%; /* Ensures the image fills the container width */
69        }
70        .image-container p {
71            font-family: "Times New Roman", Times, serif;
72            font-weight: bold;
73        }
74       
75        .image-container p {
76            font-family: "Times New Roman", Times, serif;
77            font-weight: bold;
78        }
79        .has-text-centered {
80            text-align: center;
81            font-size: 16px;
82        }
83        .has-text-left {
84            font-size: 16px;
85        }
86        
87        #image-container {
88            position: relative;
89            display: inline-block;
90        }
91        #startButton {
92            display: block;
93            margin: 20px auto;
94            padding: 10px 20px;
95            font-size: 16px;
96            cursor: pointer;
97        }
98        #imageModal {
99            display: none; /* Hidden by default */
100            position: fixed;
101            top: 0;
102            left: 0;
103            width: 100%;
104            height: 100%;
105            overflow: auto;
106            background-color: rgb(0,0,0);
107            background-color: rgba(0,0,0,0.4);
108            z-index: 1;
109        }
110        #imageModalContent {
111            position: relative;
112            margin: auto;
113            padding: 0px;
114            width: 360px;
115            max-width: 360px;
116            background-color: #fff;
117            border-radius: 10px;
118        }
119        #imageContainer {
120            position: relative;
121            display: inline-block;
122            width: 100%;
123            height: auto;
124        }
125        .dot {
126            position: absolute;
127            width: 10px;
128            height: 10px;
129            background-color: rgba(255, 0, 0, 0.5); /* Red with 50% opacity */
130            border-radius: 50%;
131        }
132        .bbox {
133            position: absolute;
134            border: 4px solid red; /* Red border for bbox */
135        }
136        .bbox-large {
137            position: absolute;
138            border: 8px solid blue; /* Green border for bbox_large */
139        }
140        crowd-alert {
141      display: none !important;
142    }
143  </style>
144  
145</head>
146<body>
147
148<section class="hero">
149  <div class="hero-body">
150    <div class="container is-max-desktop">
151      <div class="columns is-centered">
152        <div class="column has-text-centered">
153          <div style="display: flex; align-items: center; justify-content: center;">
154            <!-- <img src="./static/image/worldevallogo.png" style="width:6em;vertical-align: middle" alt="Logo"/>  -->
155            
156            <h1 class="title is-1 publication-title" style="display: inline-block;">GUI-Bee<img style="width:2em" src="./static/image/bee3_AdobeStock_422301907.jpg" style="width:6em;vertical-align: middle" alt="Logo"/>:
157              <br> Align GUI Action Grounding to Novel Environments via Autonomous Exploration</h1>
158          </div>
159          <div class="is-size-5 publication-authors">
160            <span class="author-block">
161              <a href="http://www.yfan.site/">Yue Fan</a><sup style="color: #FFB6C1;">1</sup>,</span>
162            <span class="author-block">
163              <a href="https://research.adobe.com/person/handong-zhao/">Handong Zhao</a><sup style="color: #90EE90;">2</sup>,</span>
164            <span class="author-block">
165              <a href="https://research.adobe.com/person/ruiyi-zhang/">Ruiyi Zhang</a><sup style="color: #90EE90;">2</sup>,</span>
166            <span class="author-block">
167              <a href="https://research.adobe.com/person/yu-shen/">Yu Shen</a><sup style="color: #90EE90;">2</sup>,</span>
168
169            <span class="author-block">
170              <a href="https://eric-xw.github.io/">Xin Eric Wang</a><sup style="color: #FFB6C1;">*1</sup>,</span>
171            <span class="author-block">
172              <a href="https://research.adobe.com/person/gangwu/">Gang Wu</a><sup style="color: #90EE90;">*2</sup>,</span>
173          </div>
174          <div class="is-size-5 publication-authors">
175            <span class="author-block"><sup style="color: #FFB6C1;">1</sup>UC Santa Cruz,</span>
176            <span class="author-block"><sup style="color: #90EE90;">2</sup>Adobe Research</span>
177          </div>
178
179      	 <br>
180          <div class="is-size-5 publication-authors">
181            <!--<span class="author-block">†Corresponding to:</span>
182            <span class="author-block"><a href="mailto:[email protected]">[email protected]</a>,</span>
183            <span class="author-block"><a href="mailto:[email protected]">[email protected]</a></span>-->
184            <sup >*</sup> Co-advising. This work was partly performed when the first author interned at Adobe Research.
185          </div> 
186
187          <div class="column has-text-centered">
188            <div class="publication-links">
189              <!-- PDF Link. -->
190
191              <span class="link-block">
192                <a href="https://arxiv.org/pdf/2501.13896"
193                   class="external-link button is-normal is-rounded is-dark">
194                  <span class="icon">
195                      <i class="fas fa-file-pdf"></i>
196                  </span>
197                  <span>Paper</span>
198                </a>
199              </span>
200              <!-- <span class="link-block">
201                <a href="https://github.com/eric-ai-lab/Screen-Point-and-Read"
202                   class="external-link button is-normal is-rounded is-dark">
203                  <span class="icon">
204                      <i class="fab fa-github"></i>
205                  </span>
206                  <span>Code</span>
207                  </a>
208              </span>
209              <span class="link-block">
210                <a href="https://huggingface.co/datasets/yfan1997/ScreenPR"
211                   class="external-link button is-normal is-rounded is-dark">
212                  <span class="icon">
213                      <i class="fas fa-database"></i>
214                  </span>
215                  <span>Benchmark with dataviewer</span>
216                </a>
217              </span> -->
218              <!-- Code Link. -->
219              
220
221
222            </div>
223
224          </div>
225        </div>
226      </div>
227    </div>
228  </div>
229</section>
230
231
232
233<section class="section">
234  <div class="container is-max-desktop">
235    <!-- Abstract. -->
236    <div class="columns is-centered has-text-centered">
237      <div class="column is-four-fifths">
238        <h2 class="title is-3">Abstract</h2>
239        <div class="content has-text-justified">
240          <p>
241            Graphical User Interface (GUI) action grounding is a critical step in GUI automation that maps language instructions to actionable elements on GUI screens. Most recent works of GUI action grounding leverage large GUI datasets to fine-tune MLLMs. However, the fine-tuning data always covers limited GUI environments, and we find the performance of the resulting model deteriorates in novel environments. We argue that the GUI grounding models should be further aligned to the novel environments to reveal their full potential, when the inference is known to involve novel environments, i.e., environments not used during the previous fine-tuning. To realize this, we first propose GUI-Bee, an MLLM-based autonomous agent, to collect high-quality, environment-specific data through exploration and then continuously fine-tune GUI grounding models with the collected data. Our agent leverages a novel Q-value-Incentive In-Context Reinforcement Learning (Q-ICRL) method to optimize exploration efficiency and data quality. Additionally, we introduce NovelScreenSpot, a benchmark for testing how well the data can help align GUI action grounding models to novel environments and demonstrate the effectiveness of data collected by GUI-Bee in the experiments. Furthermore, we conduct an ablation study to validate the Q-ICRL method in enhancing the efficiency of GUI-Bee.          </p>
242        </div>
243      </div>
244    </div>
245  </div>
246</section>
247
248
249<section class="hero teaser">
250  <div class="container is-max-desktop">
251    <div class="hero-body">
252    	<h2 class="subtitle has-text-centered">
253	        <div class="image-row">
254	            
255	            <div class="image-container" style="width: 100%;">
256	                <img  src="./static/image/web-f1.png" alt="figure1" >
257              </div>
258            
259          </div>
260                  <p style="font-family:Times New Roman; text-align: left;">
261                  <!-- TL;DR: <br>
262                  Prior works convert pre-trained MLLMs to GUI grounding models by fine-tuning with large amount of data. <br>
263                  Our work align those GUI grounding models to novel GUI environments, boosting their performance for deployment scenarios.<br><br>
264
265                  Long version: <br> -->
266                  App owners may want to deploy GUI action models in their own specific GUI environments, potentially novel environments on which the models have not been trained previously. 
267                  We propose to build <a style="color: #ffc800;">environment-aligned GUI action grounding model</a>, based on <a style="color: #b5b5b5;">GUI grounding models from prior works</a>. The aligned models are specifically fine-tuned to the novel environments to achieve better performance.
268                  Our proposed alignment process includes first exploring the specific novel environment with the GUI-Bee agent to generate the exploration graph and then fine-tuning the model with the data from the exploration graph.
269                  In the <a style="color: #d4a3cc;">inference example at the bottom</a>, the models encounter a query requiring knowledge of an environment-specific action outcome, which highlights the importance of the proposed alignment process. 
270
271
272
273
274	          </p>
275           
276	    </h2>
277        
278    </div>
279  </div>
280</section>
281
282
283
284<section class="section">
285  <div class="container is-max-desktop">
286    <div class="columns is-centered has-text-centered">
287      <div class="column is-five-fifths">
288      
289        <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>Overview</h2> 
290      </div>
291    </div>
292
293      <!-- First Row -->
294      <div class="columns is-fullwidth">
295
296
297          <div class="image-container" style="text-align: center; margin-left: 25%;">
298            <img src="./static/image/web-f2.png" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
299          </div>
300      </div>
301      <br>
302      <h3 style="text-align: left; margin-left: 20%;">
303        We propose aligning GUI grounding models to novel environments, which includes<br> ⓵ exploring the specific GUI environment <br>⓶ generating high-quality data from the exploration and<br>  ⓷ continously fine-tuning the model with the collected data.
304      </h3>
305      <br>
306  </div>
307</section>
308
309<section class="section">
310  <div class="container is-max-desktop">
311    <div class="columns is-centered has-text-centered">
312      <div class="column is-five-fifths">
313      
314        <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>Autonomous Exploration with GUI-Bee and Q-ICRL </h2> 
315      </div>
316    </div>
317      <!-- First Row -->
318      <div class="columns is-fullwidth">
319        <div class="column is-3">
320          <h3  style="text-align: left; margin-left: 40%;"><strong><u>·Goal:</u></strong></h3>
321        </div>
322        <div class="column is-6">
323          <div class="image-container" style="text-align: center; margin-left: 20%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
324          </div>
325        </div>
326      </div>
327    
328      <h3  style="text-align: left; margin-left: 15%; margin-right: 15%;">
329        Autonomously predict and execute minimal numbers of actions to reach a sequence of GUI screens that are as diverse as possible.
330      </h3>
331      <br><br>
332
333      <!-- First Row -->
334      <div class="columns is-fullwidth">
335        <div class="column is-3">
336          <h3  style="text-align: left; margin-left: 40%;"><strong><u>·Challenges:</u></strong></h3>
337        </div>
338        <div class="column is-6">
339          <div class="image-container" style="text-align: center; margin-left: 20%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
340          </div>
341        </div>
342      </div>
343    
344      <h3  style="text-align: left; margin-left: 15%; margin-right: 15%;">
345        1. Noisy Action Space and Action Validity:  The GUI environment usually provides a set of candidate actions, but many are invalid, i.e. non-executable elements.
346        <br>
347        2. Uncertain Screen Transitions: The outcomes of GUI actions are unknown, and the screen transitions caused by actions are irreversible
348      </h3>
349      <br><br>
350
351    
352      <!-- Second Row -->
353      <div class="columns is-fullwidth">
354        <div class="column is-3">
355          <h3  style="text-align: left; margin-left: 40%;"><strong><u>·Our solution <br>Q-ICRL:</u></strong> <br>(Refer to our paper for psudo code and more)</h3>
356        </div>
357        <div class="column is-6">
358          <div class="image-container" style="text-align: center; width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5); display: block; margin: 0 auto;">
359            <video controls autoplay loop>
360              <source src="./static/image/paper_1.mp4" type="video/mp4">
361            </video>
362          </div>
363        </div>
364      </div>
365      <h3  style="text-align: left; margin-left: 15%; margin-right: 15%;">
366        We equip the GUI-Bee with a novel Q-value-Incentive In-Context Reinforcement Learning (Q-ICRL) algorithm. It is a training-free method designed to maximize the exploration efficie
366ncy by treating the process as a Markov Decision Problem.  It uses a memory-based Q-value function to quantify the outcome of actions and employs in-context learning with a Multimodal Large Language Model (MLLM) to select the most promising action.  This approach balances exploration and exploitation.     </h3>
367      <br><br>
368
369      
370      <!-- third Row -->
371      <div class="columns is-fullwidth">
372        <div class="column is-3">
373          <h3  style="text-align: left; margin-left: 40%;"><strong><u>·Exploration <br>Result:</u></strong></h3>
374        </div>
375
376        <div class="column is-5">
377            <img src="./static/image/web-f4.png" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
378          <br><br>
379          The exploration process of GUI-Bee builds exploration graph of visited screens and actions.
380          Left: an example of the exploration graph showing screens connected by actions. Right: A zoomed-in view of the graph with examples of two GUI screens and some explored and unexplored actions (ever/never selected during the exploration).      
381        </div>
382      
383        <div class="column is-2">
384            <img src="./static/image/paper_2.gif" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
385          <br><br>
386          Illustration of the generation process of an exploration graph. 
387        </div>
388      </div>
389
390
391  </div>
392</section>
393
394
395<section class="section">
396  <div class="container is-max-desktop">
397    <div class="columns is-centered has-text-centered">
398      <div class="column is-five-fifths">
399      
400        <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>Autonomous Data Annotation with GUI-Bee</h2> 
401      </div>
402    </div>
403
404        <!-- <div class="columns is-centered has-text-centered">
405          <div class="image-container " style="width: 70%;box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
406            <img src="./static/image/web-f6.png" alt="figure2">
407          </div>
408            
409        </div>
410      <h3  style="text-align: left;">
411        Once a new edge is added to the exploration graph during the exploration, we send this information to the MLLM (GPT-4o) to generate a list of action grounding queries (u^t) for the target element (e^t). The generation process uses a carefully crafted prompt, designed to ensure the queries cover both System 1 (focused on current screen content) and System 2 (anticipating interaction outcomes) grounding challenges.  
412      </h3>
413
414      <br>
415      <br> -->
416
417      <div class="columns is-centered has-text-centered">
418        <div class="image-container " style="width: 70%;box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
419          <img src="./static/image/web-f5.png" alt="figure2">
420        </div>
421          
422      </div>
423    <h3  style="text-align: left;">
424      Once a new edge is added to the exploration graph during the exploration, we send the pair of screens connected by the edge to the MLLM (GPT-4o) to generate a list of action grounding queries (u^t) for the target element (e^t). The generation process uses a carefully crafted prompt, designed to ensure the queries cover both System 1 (focused on current screen content) and System 2 (anticipating interaction outcomes) grounding challenges.  
425<!-- The full version of the text prompt for the MLLM is provided in Figure \ref{prompt1}. We also show an example of the input images in Figure \ref{prompt1_output} along with the GUI action grounding queries $u^t$ in the corresponding output. -->
426    </h3>
427    
428  </div>
429</section>
430
431
432
433<section class="section">
434  <div class="container is-max-desktop">
435    <div class="columns is-centered has-text-centered">
436      <div class="column is-five-fifths">
437      
438        <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>NovelSreenSpot Benchmark</h2> 
439      </div>
440    </div>
441
442    <div class="columns is-centered has-text-centered">
443      <div class="image-container " style="width: 70%;box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
444        <img src="./static/image/web-f7.png" alt="figure2">
445      </div>
446        
447    </div>
448    <h3  style="text-align: left; ">
449      NovelScreenSpot is a new human-annotated benchmark for evaluating GUI action grounding models in five diverse web GUI environments and model performance improvements after they are continuously fine-tuned with 
449new data. Unlike existing benchmarks that emphasize diversity across many environments, NovelScreenSpot provides a greater data variation within each environment and includes a large number of queries about action outcomes, requiring environment-specific knowledge.
450  <!-- The full version of the text prompt for the MLLM is provided in Figure \ref{prompt1}. We also show an example of the input images in Figure \ref{prompt1_output} along with the GUI action grounding queries $u^t$ in the corresponding output. -->
451    </h3>
452    <br>
453    <div class="columns is-centered has-text-centered">
454      <div class="column is-six-fifths">  
455
456        
457        <div class="gallery">
458          <img src="./static/image/e1.png" alt="Image 11"  >
459          <img src="./static/image/e2.png" alt="Image 12"  >
460          <img src="./static/image/e3.png
461  " alt="Image 13"  >
462          <img src="./static/image/e4.png
463  " alt="Image 14"  >
464          <img src="./static/image/e5.png
465  " alt="Image 33"  >
466          
467          
468  
469        </div>
470
471
472        <h3 class="subtitle has-text-centered">
473          <h3  style="text-align: left;"> Examples of the NovelScreenSpot benchmark. Each figure illustrates the GUI screen, the A11y string, the corresponding query, and the ground truth target element. We show one example for each environment in the benchmark.  </p>
474        </h3>   
475    </div>
476  </div>
477</section>
478
479
480
481<section class="section">
482  <div class="container is-max-desktop">
483    <div class="columns is-centered has-text-centered">
484      <div class="column is-five-fifths">
485      
486        <h2 class="title is-3"><img style="width:1em" src="./static/image/web-icon.png" style="width:2em;vertical-align: bottom" alt="Logo"/>Experiments </h2> 
487      </div>
488    </div>
489
490
491    <div class="columns is-centered has-text-centered">
492      <div class="image-container " style="width: 70%;box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
493        <img src="./static/image/web-f6.png" alt="figure2">
494      </div>
495        
496    </div>
497  <h3  style="text-align: left; ">
498    We employ the GUI-Bee agent to explore the five environments in NovelScreenSpot, conducting up to 400 exploration steps per environment. To ensure diverse screen data, the exploration is repeated three times per environment at varying screen resolutions. The resulting exploration statistics are summarized in Table. We leverage GPT-4o as the MLLM for GUI-Bee and the cost for the exploration is under $50 per environment.
499  </h3>
500
501  <br>
502
503
504      <!-- third Row -->
505      <div class="columns is-fullwidth">
506        <div class="column is-8">
507            <img src="./static/image/web-f8.png" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
508          <br><br>
509          Using the data generated from GUI-Bee exploration, we continuously fine-tune three GUI grounding models for each environments explored. Table here shows  the benchmarking GUI grounding models on the NovelScreenSpot and Eventbrite environment of Multimodal-Mind2Web benchmark. This table shows the model accuracy and, in parentheses, the absolute improvement over the vanilla models after the models are continuously fine-tuned. The results demonstrate that our GUI-Bee model significantly improves the performance of GUI action grounding models in novel environments. 
510        </div>
511
512        <div class="column is-4">
513            <img src="./static/image/web-f9.png" alt="figure2" style="max-width: 100%; box-shadow: 5px 5px 10px rgba(0, 0, 0, 0.5);">
514          <br><br>
515          We propose Depth-fixed DOM Diversity Counts (D3C) to evaluate the coverage of the exploration process. This result shows the mean and standard deviation of D3C at various exploration steps across three runs in three environments. GUI-Bee agent demonstrates a wider exploration coverage.      
516        </div>
517      </div>
518  </div>
519</section>
520
521<!-- 
522<section class="section">
523  <div class="container is-max-desktop">
524    <div class="columns is-centered has-text-centered">
525      <div class="column is-five-fifths">
526        <h2 class="title is-3">Screen Point-and-Read (ScreenPR) Benchmark <a href="https://huggingface.co/datasets/yfan1997/ScreenPR"
527                   class="external-link button is-normal is-rounded is-dark">
528                  <span class="icon">
529                      <i class="fas fa-database"></i>
530                  </span>
531                </a></h2> 
532      </div>
533    </div>
534    <div class="image-row">
535        <div class="gallery">
536        <img src="./static/image/web_2b491da3-f798-4866-b002-a25ab7d15263.png" alt="Image 11" style="height: 100px">
537        <img src="./static/image/web_2f3df3a1-d287-4457-91b3-a672d0f72f84.png" alt="Image 12" style="height: 100px">
538        <img src="./static/image/web_352ae757-fdf0-4d02-9167-c24a42629a60.png
539" alt="Image 13" style="height: 100px">
540        <img src="./static/image/web_89d60b8b-3f6d-4160-93d4-1791d4899455.png
541" alt="Image 14" style="height: 100px">
542        <img src="./static/image/frame_73.jpg
543" alt="Image 21" style="height: 100px">
544        <img src="./static/image/frame_81.jpg
545" alt="Image 22" style="height: 100px">
546        <img src="./static/image/frame_162.jpg
547" alt="Image 3" style="height: 100px">
548        <img src="./static/image/frame_246.jpg" alt="Image 24" style="height: 100px">
549        <img src="./static/image/mobile_d8a4069b-ec35-4a62-a3cd-5b5e7d95b50c.png" alt="Image 31" style="height: 100px">
550        <img src="./static/image/mobile_4e17db8b-85df-4278-ba0a-9f03e5ca85da.png
551" alt="Image 32" style="height: 100px">
552        <img src="./static/image/mobile_5c9fb5f1-7d84-4283-8ba4-7821a9545c95.png
553" alt="Image 33" style="height: 100px">
554        <img src="./static/image/mobile_6cc4e2c1-b5bd-4347-bee4-63a1217d052c.png
555" alt="Image 34" style="height: 100px">
556		<img src="./static/image/and_more.png
557" alt="Image 35" style="height: 100px">
558        
559        
560
561   		</div>
562        <div class="image-container" style="width: 150%;">
563            <img  src="./static/image/benchmark_stats.png" alt="stats">
564            
565        </div>
566        
567    </div>
568	<h3 class="subtitle has-text-centered">
569              <p style="font-family:Times New Roman">Screen Point-and-Read (ScreenPR) benchmark is introduced to rigorously evaluate our ToL agent on the ScreenPR task, where descriptions are required based on user-indicated points on screenshots. It covers a diverse domain of GUIs. Each screenshot is annotated with about 2 target points and their corresponding local regions on the screenshot.</p>
570            </h3>   
571	  
572</section> -->
573
574
575
576
577
578    
579    
580    
581<section class="section" id="BibTeX">
582  <div class="container is-max-desktop content">
583    <h2 class="title">BibTeX</h2>
584    <pre><code>
585      @misc{fan2025guibeealignguiaction,
586        title={GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration}, 
587        author={Yue Fan and Handong Zhao and Ruiyi Zhang and Yu Shen and Xin Eric Wang and Gang Wu},
588        year={2025},
589        eprint={2501.13896},
590        archivePrefix={arXiv},
591        primaryClass={cs.CL},
592        url={https://arxiv.org/abs/2501.13896}, 
593  }
594    </code></pre>
595  </div>
596</section>
597
598
599
600
601<footer class="footer">
602  <div class="container">
603    <div class="columns is-centered">
604      <div class="column is-8">
605        <div class="content">
606          <p>
607            This website is adapted from <a rel="license"
608            href="https://github.com/nerfies/nerfies.github.io">Nerfies</a> and <a rel="license"
609            href="https://gligen.github.io/">GLIGEN</a>, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
610            This website is licensed under a <a rel="license"
611                                                href="http://creativecommons.org/licenses/by-sa/4.0/">Creative
612            Commons Attribution-ShareAlike 4.0 International License</a>.
613          </p>
614        </div>
615      </div>
616    </div>
617  </div>
618
619</footer>
620
621
622<!-- Additional libraries for parsing CSV -->
623<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-csv/1.0.21/jquery.csv.min.js"></script>
623
624
625</body>
626</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.