Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps

Full transcript

Imagine doing reinforcement learning for a phone agent only on iPhone apps, then handing it an Android phone it has never trained on. The authors report that it succeeds nearly seven times as often as the same model without that reinforcement learning stage. How does a small model get that good? The paper is called Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps, and it comes from a team at Apple. It matters because the models that already do this well are too expensive to use. Frontier vision-language models can read a screen, tap, swipe, and type their way through a task, but the authors note this often costs several dollars per task. That is fine for a demo and hopeless for something you would ask your phone to do every day. So why not just train something smaller? The catch is how small models are usually taught. The common recipe is supervised fine-tuning: you show the model recorded demonstrations and have it imitate the next action. Those demonstrations are mostly successful runs, so the model rarely sees what a mistake looks like or what to do after one. The authors say it ends up memorizing paths. Land it on a screen it did not expect, and it has nothing to fall back on. The result is brittle execution and poor error recovery. Reinforcement learning, where a model learns from trial and reward instead of imitation, ought to fix this. The authors argue that the existing versions fall short in two ways. Some are single-step, rewarding the model for matching a reference action, which is imitation again. Others do learn from live interaction, but only hand out a reward at the very end, success or failure. That is a sparse signal. An agent that did most of a task right and slipped on the final screen gets no credit for any of it. Here is the authors' answer in one breath. They split the agent into a planner that decides what to do on the current screen and a grounding model that turns that decision into an exact action and screen position, then train the planner with online reinforcement learning on live iOS devices running real apps. On UINavBench, a benchmark of real iOS apps, the eight-billion-parameter agents tie Gemini-3.1-Pro. A few ideas need to be on the table first. UI navigation is a loop. You give the agent a goal in plain language, say, search for a pair of shoes and add it to the cart. At each step it sees the current screenshot plus a running summary of what it has done, and picks one action from a short menu: tap, swipe, type text, go home, long press, ask the user, or declare the task complete. The ask-user action lets the agent stop and ask which kind of shoes you want, rather than guessing. How do you teach a small model this? The obvious route is supervised fine-tuning: show it recorded runs and have it copy the next action. The authors point out that those demonstrations are mostly successful ones, so the model rarely sees what a mistake looks like. It memorizes the path, and an unfamiliar screen leaves it lost. Reinforcement learning instead lets the model try things, get a score, and shift toward what scored well. Offline versions learn from a fixed pile of collected runs. Online means the model acts in the real environment while it trains, so it lives with the consequences of its own mistakes. The shape of the score matters. A sparse terminal reward says nothing until the end, then pass or fail. An agent that does nine steps right and fumbles the tenth gets the same nothing as one that wandered off at the start. A dense reward scores every step, so partial progress counts. And who does the scoring? A second language model acting as a judge, which looks at screenshots and decides whether a step helped. It needs no recorded golden path, so it works on any app and task.

Then there's the optimizer. GRPO, which many earlier GUI agents use, samples several attempts at one task and pushes up the ones that beat the group's average. Every attempt collapses into one number, so you can't tell which step deserved the credit. PPO trains a second network, the critic, that guesses how much reward is still to come from any situation. An action's advantage is how much better things went than the critic expected. The authors note that no earlier GUI-agent system learned such a critic. GiGPO gets step-level credit without a critic, by finding identical screens that recur across attempts and averaging the rewards there. ClawGUI-RL, a concurrent paper, uses it to train one model for both planning and grounding on Android emulators. The authors' objection is that exact repeats almost never happen in their setting. The discount factor sets how much a later reward counts compared with one right now. At one, the future counts fully. At one half, each step further out counts half as much, so the agent mostly cares about the near term. Two contrasts and the benchmarks. UI-TARS fine-tunes on human-relabeled reflection pairs, so its error recovery comes from people, not live interaction. Qwen-UI-Agent runs across about ten thousand environments, but its online stage uses only terminal rewards. Early benchmarks scored static screenshot-and-action pairs, which can't test recovery. The field has since moved to live environments, and live iOS evaluation arrived only recently, with UINavBench and iOSWorld. So what does the agent they train actually look like? It's two models, not one, and that split is the first design decision worth understanding. The planner sees the task, the current screenshot, its earlier step plans with their reasoning, and the remaining plan it left last time. It reasons out loud, then writes a step plan, the one immediate thing to do on this screen. It also writes a remaining plan, the high-level goals still needed to finish the task. The grounder takes that step plan and turns it into an action, such as a tap, plus exact x and y coordinates on the screen. Why split them? Grounding is precise spatial matching, like going from "tap the search field" to a spot on the screen, and supervised fine-tuning handles it well. Planning is open-ended: tracking progress, adapting when the app surprises you, recovering from errors. That's where reinforcement learning helps, and since RL is the expensive part, the authors spend it only on the planner. RL also needs somewhere to practice. The agent runs on cloud-hosted iOS virtual machines, driven through typed commands like tap, swipe and type. Each is set up once, with apps installed and an Apple ID signed in, and that state is saved as a snapshot. When an episode ends, the device rolls back to the snapshot instead of being rebuilt, which cuts the restart to about 10 seconds. That lets the system run 512 trajectories at once, each on its own device. Before RL, the planner gets a warm start. They fine-tune Qwen3-VL-8B-Instruct, a vision-language model, on human-annotated trajectories from three datasets. Two are in-house, covering first-party and third-party iOS apps. The third is a public Android set called GUIOdyssey, included because it has tasks that hop between several apps, which the iOS data lacks.

None of these record reasoning, only the goal, the screens and the plan at each step. So Gemini wrote the chain-of-thought for each step plan, plus the remaining plan and its reasoning, as training targets. The RL stage draws tasks from those same iOS datasets, with one exception. Android multi-app tasks can't run on iOS, so the authors generated new ones. Gemini first proposed sensible app pairs, like Weather and Calendar, which supports finding a clear day to schedule an outdoor event. Then it wrote tasks for each pair, and one author checked every one by hand. When the agent asks the user a question, Gemini-3.5-Flash plays the user, in both training and evaluation. Last, the grounder. It's an internal 3B model trained on human-annotated iOS screens, with over 90 percent accuracy on held-out data. It only understands directional swipes, like swipe up, so every swipe goes to Gemini-3.5-Flash, which can swipe from a specific spot. RL training ran with exactly this setup, with the grounder frozen while the planner learned. The planner is the only part learning, so what tells it whether it did well? There's no answer key to compare against. The authors use language models as judges instead. No human-recorded sequence of taps is needed, so the same setup works on the synthetic multi-app tasks they generated. The reward comes in two channels. The first is dense. After each action, a judge sees the screen before and after, the plan, the reasoning, and the history so far, and returns success, fail, or overcomplete. Success needs visible progress, and the agent's reasoning has to match what's really on screen. If it stumbles into progress while its stated reasoning is hallucinated, that counts as a fail. Repeating a failed action also gets penalized. Overcomplete is what the agent gets for carrying on after the task is already done, and it teaches the agent to recognize when to stop. This judge runs at every step, so the authors use Gemini-3.5-Flash. A separate judge handles the ask-user action, looking only at the current screenshot to check that the information really is missing. The second channel is terminal, paid once at the end. For this the authors use Gemini-3.1-Pro, which sees the last three screenshots, the full action history, and the earlier step verdicts. Its verdicts are success, fail, or blocked complete. Blocked complete covers an agent that did everything right until it hit something unavoidable, like a paywall or a CAPTCHA, so it isn't punished for a state nobody could act on. How do those verdicts become numbers? Per step, a success earns plus one, an ordinary fail earns zero, and overcomplete costs minus two. At the end, finishing the task and declaring it earns plus two, blocked complete earns half a point, running out of steps costs three quarters of a point, and claiming completion falsely costs one. An ordinary failure costs nothing, which keeps exploration cheap. Empty or unparseable output, though, is charged at the terminal scale, because a garbage action scored at zero would look no worse than a real attempt that failed. Why those sizes? Step rewards fade by half with each step, so even a run that succeeds at every step collects at most two in total. That is worth the same as one terminal success, so neither channel drowns out the other. Sometimes a run fails for reasons that have nothing to do with the agent: an app that never loads, a failed install, a dropped device connection. Those trajectories are masked out rather than penalized. They don't touch the policy update or the critic, and they're left out of the statistics used to normalize advantages. That matters because their zero rewards would shrink the spread and inflate every genuine advantage.

So the agent now gets two streams of reward. How do you turn them into a learning signal? The common choice for UI agents is GRPO, which samples several attempts at the same task and scores each one against the group average. The authors argue it's a poor fit here. To get a single group-relative number, the step rewards have to be folded into the terminal reward, and the information about when progress happened is lost. They use PPO instead, which keeps a critic, a second model that estimates how much reward to expect from the current screen onward. That lets the step-level feedback be used directly. They also have a cautionary tale from their own GRPO run. In it, every reward was zero or positive: one point per good step, ten for a successful finish, and nothing negative. A bad step, a hallucinated one, or a run that used up the whole budget all scored exactly zero. With a cap of 30 steps, a trajectory that wandered and collected partial credit could earn up to 30 from step rewards, while actually finishing paid only 10. The authors say the agent learned to prolong trajectories instead of completing the task and stopping. And since a harmful action looked identical to doing nothing, nothing told the model that wrong moves were bad. The fix is a discount factor, a number that shrinks the value of rewards the further away they are. Without one, a reward fifty steps out counts as much as one right now, so the best strategy is simply to keep taking steps that look locally fine. The authors discount the step rewards by one half. Each state's value then depends mostly on what happens nearby, which is also much easier for the critic to predict. The terminal reward is deliberately not discounted. If it were, its effect on early actions would fade to almost nothing on long trajectories. Leaving it alone means a policy that takes a longer, unexpected route to success still gets full credit for it. For the advantage, meaning how much better an action did than the critic expected, they set the smoothing parameter to one. That reduces it to a plain comparison: the return the trajectory actually earned, minus the critic's estimate. It is unbiased with respect to the critic, at the price of a noisier sampled return. The critic reads its value from the last token before the model starts writing its next action, so the estimate can't peek at a choice that hasn't been made. Now the second idea. GiGPO, a group-based method, improves on GRPO by collecting identical screens that recur across attempts and averaging their rewards. Here that almost never works. With only four or eight attempts per task, an exact repeat is rare, so the groups are nearly all single states. Loosening the match to "similar screens" would average over genuinely different situations and pass that bias into the value estimates. GiGPPO routes the same intuition through the critic instead. Each state is paired with its closest state among the sibling attempts, judged by a 64-bit perceptual hash, a compact fingerprint that barely changes when two images look alike. Closeness is the Hamming distance, the count of bits that differ. A small neural network then predicts the value from both states' embeddings, the neighbor's return, and that distance. The distance tells the critic how far to trust the neighbor. At zero, it's the same screen and the neighbor's return is nearly a direct observation. Further out, it counts for less. The state's own reward is never an input, so the target can't leak.

Training the critic and policy together from the start was highly unstable, the authors report. Early value guesses are uninformative, so the advantages are mostly value error, and the policy takes large, damaging steps. So for the first 15 steps the policy stays frozen while the critic learns from collected returns. The critic's learning rate is also 100 times the policy's. That's intentional: the critic chases a moving target and gets thrown away at the end, while the policy is the thing being kept and has to change slowly to avoid wiping out its supervised starting point. For scale, each training step runs 128 attempts in total, eight on each of sixteen tasks, and no episode is allowed to run past 30 steps. So does any of this actually work? The authors test on three benchmarks, and the third one matters most. Two are iOS benchmarks. UINavBench has 116 tasks spread across 36 real iOS apps, staged with user data. iOSWorld has 133 tasks across 26 synthetic apps, all filled with data for one user persona. Both mix single-app and multi-app tasks, and a vision-language model checks whether each task's requirements were met. The third is AndroidWorld, with 116 tasks across 20 real apps. The agent never saw Android during reinforcement learning, so this is the out-of-distribution test. Start from the supervised baseline. It scores about 17 percent on UINavBench, 4.5 percent on iOSWorld, and 4.7 percent on AndroidWorld. After two epochs of training, which is 336 updates, PPO reaches 32.4 percent on UINavBench. That ties Gemini-3.1-Pro exactly. Gemini-3.5-Flash gets 28.4 percent. Qwen3.8-27B gets 19.6, and it is over three times larger than the 8B agent. On Android the jump is bigger. PPO reaches 32.8 percent against the baseline's 4.7, nearly seven times higher, and every reinforcement learning checkpoint beats the baseline there. The authors take this as a sign that the agent learned general navigation strategies instead of memorizing iOS paths. Do the dense rewards earn their cost? The authors compare standard PPO with a version trained on the terminal reward alone. At equal steps, the dense version generally wins. PPO after one epoch already matches or beats terminal-only after two. On UINavBench, for example, one epoch of dense PPO gets 25.5 percent, while two epochs of terminal-only gets 21.6. So the step-level signal makes learning faster. GiGPPO shows a similar pattern. On UINavBench, it reaches about the same scores as standard PPO in roughly half as many updates. The authors say this likely comes from the critic borrowing signal from visually similar screens, since UINavBench is the benchmark closest to the training environment. One more detail affects how to read these numbers. Using the supervised planner, the authors tried three grounding setups: the 3B grounder alone, Gemini-3.5-Flash alone, and the 3B grounder with swipes sent to Gemini. On UINavBench the hybrid won, about 17 percent against 13 and 15 for the others. That fits, because the 3B model was trained on real iOS screenshots, and sending swipes to Gemini covers its direction-only limit. On iOSWorld, with synthetic apps, Gemini alone was best. AndroidWorld looks even less like the 3B model's training data, so the authors concluded Gemini is the best choice there too. Each benchmark in the results uses its best setup.

Numbers only say so much, so the authors also show traces. Take a UINavBench task: find a route from Apple Park to the Santa Clara Caltrain Station. Apple Maps wants the destination before the starting point, but the task mentions the start first, and that trips up both agents. The GiGPPO agent first sets Apple Park as the destination. Then it backs out of the directions search and restarts with the Caltrain station as the destination. The supervised agent opens the wrong search page and keeps trying to set a custom starting point until it hits the step limit. A harder one comes from iOSWorld, spanning three apps. The agent must find a recent Chipotle order in QuickBite, find the matching receipt in Mail, check the charge in a bank app, and compare the three amounts. The supervised agent opens all three apps but never searches them for what it needs. The GiGPPO agent works through each app and finds every detail. The last trace is from AndroidWorld: delete three named recipes from the Broccoli app. The PPO agent trained with dense rewards searches for each recipe and deletes it. The terminal-only agent scrolls through the list by hand, wastes many steps, and runs out of its step budget before finishing. So how far does all this get you? Not as far as the headline suggests, and the paper is fairly open about it, even though it has no dedicated limitations section. The clearest weak spot is iOSWorld, the benchmark built on synthetic apps. There the gains are modest and uneven. The best PPO checkpoint falls to 2.3 percent, below the SFT baseline. GiGPPO's last checkpoint reaches 9.8 percent, more than double SFT, yet Gemini-3.1-Pro scores 39.5. The authors give two reasons. The synthetic apps' layouts differ from the real apps used in training, and the tasks are harder, since most involve personalization or span more than two apps, which was out of distribution for the agent. They also saw the agent search for a synthetic app by name, collide with a real app of the same name, and try to install it from the App Store. Android tells a similar story. RL lifted the agent nearly sevenfold, yet Gemini-3.5-Flash scores about 58 percent and Gemini-3.1-Pro about 61, against the agent's best of roughly 33. The authors suspect the frontier models saw large public Android datasets in pretraining, while public iOS data is scarce. Absolute rates are low everywhere: even the best RL agent solves only about a third of the real iOS tasks. Is GiGPPO then simply the better algorithm? Not consistently. It reaches comparable UINavBench scores earlier, but its final checkpoint there sits below PPO's. And the authors only say the speedup can likely be attributed to the critic using similar screens, so that's an inference, not a tested cause. There's also what the system leans on. Gemini models serve as the step judge, the terminal judge, the simulated user, and the swipe grounder, and the main grounder is an internal 3B model. Reinforcement learning touches only the planner. So what do you take away? A small 8B agent, trained online on live iOS devices with a judge scoring both steps and outcomes, beats supervised fine-tuning by a wide margin and matches Gemini-3.1-Pro on real iOS apps. The authors see it as a route to UI agents cheap enough for everyday use.