The Road Not Taken
In our previous post, we introduced our research on unprompted browser agents where given the current custom DOM representation and recent action history, the model predicts the next action.
The hardest part was inferring intent. A page can contain several plausible elements, and choosing among them depends on what the user is trying to accomplish. Once that intent is known, finding the corresponding element is considerably easier.
Several experiments led us to this conclusion. We tried predicting future page changes from ordered browser history, generating actions as continuous vectors with diffusion, and learning from browser dynamics. This post covers what we tried, what happened, and what each result taught us.
Predicting the Next Page Change
We began with an oracle experiment to measure how informative the next page state could be.
For each recorded action, we supplied the current and resulting custom DOM state and asked the model to identify which element caused the transition between page states. We found that element-ID prediction accuracy reached 93.14% showing the value in having the next page state. However, when predicting live, the agent doesn’t have access to the next page state before acting. Here we explore a set of JEPA-inspired experiments to predict that next state.
JEPA-Inspired Next-State Prediction
A Joint-Embedding Predictive Architecture (JEPA) predicts a target in embedding space rather than generating it as a raw observation. The context and the target are encoded separately, and a predictor learns to map the context toward the target. The target supervises training and is not available at inference.
In our case, the context was the sequence of custom DOM states up to and including the current page. The target was the change caused by the next action, written as a text diff. The predictor never saw the actions, so the experiment tested whether the predictor can use page history alone to anticipate what happens next.
States and changes were embedded with the same frozen intfloat/e5-small-v2 encoder. Each document was split on line boundaries into at most four chunks of about 1,800 characters per chunk. Each chunk was mean-pooled over its tokens and normalized, and the chunk vectors were averaged and normalized again into one 384-dimensional embedding. Using a small frozen encoder gave us a stable target space. A strong signal would have justified a larger encoder.
The predictor was trained contrastively. For each history, the recorded next change was the single positive example. The next changes associated with the other histories in the batch served as negative examples. We applied cross-entropy to their cosine similarities at a temperature of 0.07, training the predictor to place its output closer to the recorded future than to the alternatives.
Four-state Transformer. Our first predictor was a two-layer Transformer with about 3.85 million parameters, reading the four most recent state embeddings. The control was an identical model trained with length-matched future changes shuffled across trajectories. Shuffling kept the target distribution but broke the link between a history and what followed it. The ordered model reached 1.39% top-1 retrieval, about four times the control’s 0.35%. As expected for a sequential task, page order carries strong signals about what happens next.
128-state GRU. To test whether a longer history recovers more signal, we trained a two-layer GRU on either the last four states or up to 128. Most next changes are unique, so exact retrieval is a very strict measure. After training, we grouped the training targets into 64 clusters with k-means and checked whether each prediction landed in the same cluster as the true change. The clusters come purely from E5-small’s embedding space; the GRU predicted a continuous embedding, not a cluster ID. Four states reached 7.53% and 128 states reached 7.88%.
A much longer history barely outperforms the short one, showing most of the signal sits in the most recent pages. Here we learned that order matters, but length of history hits diminishing returns.
Diffusion-Based Action Generation
Our baseline generates an element ID as text. The valid IDs and the candidate set change on every page, so the model must emit the exact identifier before the system can ground the action.
We tested a decoder built around the page’s candidate elements instead. We ran our fine-tuned Qwen once over each prompt and kept it frozen. Its final hidden state at the last token (2,048 dimensions) became the conditioning vector. Each candidate element was represented by the mean of its tokens’ hidden states, compressed to 256 dimensions by a fixed random projection. An action was that 256-dimensional element vector plus two dimensions for click or fill, 258 dimensions in total. A 10.8M-parameter denoiser learned to generate this vector from noise given the conditioning vector. The generated vector was then scored against every valid element-and-method pair on the page by dot product.
This design guaranteed that every prediction mapped to a valid page element. It also let us test whether continuous action vectors generalize across layouts better than page-specific ID strings.
On the matched validation rows, the diffusion model reached 40.71% element-ID accuracy, compared to 52.26% for the baseline.
The main limitation was information lost before the decoder made its decision. The baseline picks an element while attending to every token of the prompt, so it can weigh each candidate against the full page and history. The diffusion head saw the whole prompt as one vector and each candidate as one pooled, randomly projected vector. We tried richer representations: multiple Qwen layers, separate element-ID vectors, and cross-attention over several context vectors. Each added some signal, but none beat the original design.
The experiment changed how the model pointed at the page, not its decision about which element mattered, so arbitrary ID generation was not our main bottleneck. It was, however, significantly faster. The language model reads the prompt once instead of generating the action token by token, and the decision comes from a small 10.8M-parameter head that needs only a few denoising steps. If the information loss can be fixed, for example with decoders that read candidate tokens directly instead of a frozen, pooled cache, diffusion could be a practical way to cut inference latency. It may also already suit tasks with fewer, more distinct candidates. However, for our case we decided against this because the accuracy drop from the lost prompt detail outweighed the latency gains.
Browser Dynamics as Supervision
We then asked whether a model trained to understand how pages change could make better next-action predictions.
Forward World Model Initialization
AgentWorld is trained on browser dynamics: given an interaction, it learns how the page changes afterward. We hypothesized that this knowledge would transfer, helping the model tell candidate elements apart and learn next-action prediction faster than one starting from a general model. We initialized from an AgentWorld checkpoint and fine-tuned it for 400 steps on our proprietary next-action dataset.
It reached 38.39% element-ID accuracy, compared with 48.18% for our existing Qwen baseline.
The model predicted an action on 99.10% of rows. That was not a strength. It rarely abstained, correctly identifying only 9 of 275 no-op rows. Its action accuracy also fell below the baseline, so poor abstention calibration explained only part of the regression.
To separate the prior from the amount of training, we also fine-tuned our Qwen model for the same 400 steps on the same data. It reached 38.31%, essentially the same as AgentWorld. The browser-dynamics prior gave no head start: it did not reduce the amount of direct next-action learning our task required. Modeling the effects of an action does not determine which of several available actions matches the user’s goal.
Inverse Dynamics
At equal training, the forward prior gave no advantage. We suspected AgentWorld’s knowledge was useful but pointed the wrong way: forward modeling predicts an action’s effect, while our task requires choosing the action. We hypothesized that training AgentWorld directly on recovering actions, at a much larger scale, would turn its browser-dynamics knowledge into a better policy. We trained it on a 121,513-example curriculum over 3,000 steps. Seventy-five percent of examples asked the model to infer the observed action from a causal trajectory and the current page. The remaining 25% came from the proprietary next-action dataset our baseline was trained on.
This model reached 42.20% element-ID accuracy. It improved over the forward-dynamics initialization but remained below the 48.18% baseline.
It predicted an action on every validation row and never emitted a no-op. The training mixture contained far fewer no-op examples than the validation set (2.18% versus 9.19%), which explains much of that behavior. Its accuracy on rows that required an action was also only 46.47%, so abstention was only part of the problem.
The takeaway was clearer than the source of the gain: inverse-dynamics examples transfer more naturally to next-action prediction than forward web modeling. They still did not resolve the central ambiguity in our data. Several actions can be plausible on the same page, and choosing among them requires identifying the user’s current objective.
Future-Contrastive Training
Neither dynamics experiment helped the model decide which plausible action the user wanted. One experiment showed that the future resolves this ambiguity, and the JEPA probes showed the future is hard to predict from page history. Building on both, we hypothesized that the future could still help if used only as a training signal. If the model’s internal representation were pushed toward what actually happened next, it might learn to encode the features that anticipate the user’s next step, without ever having to predict the future explicitly at inference.
During training, we encoded the events following the current action into a target vector. Alongside the ordinary next-action loss, we added a contrastive term with weight 0.05. It projected the model’s final hidden state for the causal prompt (everything available before acting) to 256 dimensions and trained it to match its own future vector over those of other examples. At inference, the future encoder and future events were removed, and the deployed model received the same causal prompt as the baseline and produced one action. We trained for 600 steps on a 22,005-example mixture: 75% from our next-action dataset and 25% from the external trajectory corpus.
This reached 48.81% element-ID accuracy, compared with 48.18% for the baseline. Training on what happened next did teach the model something useful, but only a little. The gains didn’t land where we needed them: the model fixed about as many history-dependent cases as it broke. Instead of getting better at working out what the user wanted, it mostly reshuffled which rows it got right and wrong. The net gain was too small to ship.
What We Learned
Browser history carries real signals about what happens next. Ordered page histories predicted the next page change about four times better than shuffled ones, and most of that signal came from the last few pages.
But predicting what happens next is not the same as choosing what to do. Each experiment improved one part of the problem without solving the decision itself. Diffusion guaranteed that every prediction pointed at a real element and ran faster, but it chose the right element less often. A model pretrained on how pages change gave no head start over our own model. Training it directly on recovering actions helped, but it still fell short of the baseline. Using the future as a training signal produced a small net gain that did not land on the cases where history mattered most.
Every experiment ran into the same wall: a page offers several plausible actions, and picking the right one requires knowing what the user is trying to do. The next-action problem comes down to inferring the user’s current goal from an incomplete trajectory, then finding the element on the page that serves it.