Announced 10 Oct 2026 · Sources checked
What is GUI-Decisions?
GUI-Decisions is a pair of open models and a serving library for the grounding step of computer-use agents: given a screenshot, an instruction and earlier actions, decide what to do next and where. Umapathi announced them in “Stop Decoding Coordinates for 3x Faster Computer-Use Grounding”, a Hugging Face community blog post dated 10 October 2026. The 31B model, the 2B model and a demo Space went up the same day.
The idea comes from the typed decision models that have filled AI news this month. TypeSafe’s Jev and alternatives such as Microsoft’s Decision-1 and OpenAI’s Decisions API answer closed questions by returning a probability over fixed labels instead of writing text. Umapathi’s question was whether the same trick could work for a click, which needs two continuous numbers rather than a category.
His starting point is that ordinary LLMs already work as decision models zero-shot. On a 2,000-decision typed benchmark from his SystemOne library, Gemma 4 31B read this way scored 0.709 accuracy at 81 ms per five-question case, against 0.727 at 710 ms for the published TypeSafe Jev 1.13 result. A small LoRA trained with plain cross-entropy lifted Gemma to 0.791.
How does slot reading replace decoding?
The assistant’s reply is a fixed template of slots: action, x, y, a second point for swipes, a key, and scroll direction and amount. Each slot ends in a placeholder, and the answer for a slot is the model’s next-token distribution just before it, restricted to that slot’s allowed labels. Because no slot waits for another to be written, every slot can be read from the same prefill. In practice the client sends one request per slot in parallel and lets vLLM’s prefix cache share the screenshot.
Coordinates become classification by splitting each axis into 256 bins, each with its own single-token label. Training uses soft targets, a small Gaussian over neighbouring bins, so being one bin off costs less than being half a screen off. At read time the client takes the peak bin and refines it with its neighbours.
Label choice turned out to matter. Using ordinary words such as apple and river as bins failed almost completely, at 0.6% accuracy on ScreenSpot, because the model has to unlearn their meanings and has no sense that bin 117 sits next to bin 118. Byte-like tokens with low similarity to each other (mean cosine 0.14) worked. A coarse-then-fine scheme borrowed from robotics rescued the bad word labels, but with byte labels a single 256-bin slot matched or beat it and was 15–20 ms faster. 256 bins also beat the 101-bin grid of the earlier GPC-1 approach by 9.5 points on agent steps.
Does the trick hold on a 2B model?
To check that the recipe is not a Gemma quirk, Umapathi trained the same slot heads on mPLUG’s GUI-Owl-1.5-2B using computer-use data only. With slots it took 93 ms per step and scored 0.626 on computer-use steps and 0.770 on ScreenSpot clicks. Vanilla GUI-Owl at native resolution, using its own tool-call format, matched the 0.626 step score but took 323 ms and scored 0.645 on clicks. With the same capped images it took 194 ms and scored 0.610 and 0.626.
The post notes that vanilla GUI-Owl’s click numbers used its agent prompt rather than its dedicated grounding prompt, which may understate its best click accuracy. Even so, the pattern is the same as at 31B: similar step accuracy at roughly 30% of the latency.
The 31B model also reads bounding boxes (centre plus log-scaled size) and coarse 24-ray masks. After adding 5,000 RefCOCO photo boxes to training, box IoU on RefCOCO validation photos reached 0.770 without hurting computer use. Handing the predicted box and centre to SAM 2 as a prompt produced 0.792 mask mIoU on RefCOCO, for about 20 ms more per image.
How do you run it, and under which licences?
GUI-Decisions-31B is Google’s Gemma 4 31B instruction model with the slot-reading LoRA merged in bf16, licensed Apache-2.0 like its base. GUI-Decisions-2B is MIT, matching GUI-Owl-1.5-2B. A separate LoRA-only repository lets you keep the base model available for text, and the LoRA on an NVFP4 base was faster than the merged checkpoint in the author’s tests, 152 ms against 190 ms per step in bf16.
Serving the merged 31B takes 62 GB of weights, so the card suggests a single 80–96 GB GPU, vLLM 0.30, a patch that raises the label-id cap, prefix caching and eight API-server processes. A Python client returns a step such as a click at x 0.928, y 0.247 as fractions of the screenshot, and an in-process transformers path skips the server. Training took about four hours on one RTX PRO 6000 with a rank-16 LoRA on AGUVIS, Wave-UI and RefCOCO data, whose own terms the card tells users to check.
The serving library lives in a GitHub repository created on 10 October; when we checked on 12 October it showed no licence file, so reuse of that code is legally unclear even though the weights are permissive. For a broader primer on how these agents work, see our explainer on AI that clicks your screen.
What it does not solve
Umapathi is direct about scope. Slot reading speeds up grounding, not whole agents. Typing and final answers still wait for the base model to write text, multi-step tasks still need planning and often reasoning, and latency still grows with each turn. Points can miss small targets, and boxes on UI elements are weak (0.467 mean IoU on ScreenSpot). Heavier agent stacks such as mncai’s Hunmin-397B computer-use model target the planning side this work leaves open.
All figures are the author’s, measured on his hardware and datasets, and none has been independently reproduced. We read the blog post, both model cards, the Space metadata and the code repository listing on 12 October 2026, and did not run the models.
The broader point is worth watching. If agents can skip reasoning on steps that do not need it and read their clicks instead of writing them, the slowest part of today’s computer-use loops could shrink a lot. Umapathi’s next steps are overlapping text generation with slot reads and testing the approach on full multi-step tasks.
Common questions
Does GUI-Decisions replace a full computer-use agent?
No. It decides one grounding step at a time from a screenshot, the instruction and earlier actions. It has no memory and does not act; typing and answers still come from a base model that writes text.
What hardware does the 31B model need?
The merged bf16 checkpoint is about 62 GB, so the model card suggests one 80–96 GB GPU. The 2B model is far lighter, and the author’s published timings use an RTX PRO 6000.
Can I use it commercially?
The 31B weights are Apache-2.0 and the 2B weights are MIT. The training datasets carry their own terms, and the serving-code repository showed no licence file when we checked on 12 October 2026.
What to remember
GUI-Decisions shows that a computer-use model can pick its action and click point from token probabilities instead of writing JSON, about 3.5× faster on the author’s tests, with open weights and clear limits.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards
