SentinelBench: A Benchmark for Long-Running Monitoring Agents
Microsoft Research AI Frontiers (arXiv preprint), 2026 · Microsoft Research
Matheus Kunzler Maldaner
Adam Fourney
Amanda Swearngin
Hussein Mozannar
Gagan Bansal
Maya Murad
Rafah Hosn
Saleema Amershi
What is SentinelBench?
SentinelBench is an open-source benchmark of 100 time-evolving monitoring tasks across 10 synthetic web environments, built at Microsoft Research to evaluate long-running AI agents. Each task asks an agent to watch a live web interface, notice when a scripted event makes progress possible, and respond promptly without wasting resources while waiting.
I spent my internship with the AI Frontiers group at Microsoft Research watching web agents burn tokens while they waited for something to happen. Ask an agent to tell you when a repository hits a star milestone, and it will refresh pages, issue tool calls, and hunt for alternatives on a task where the only correct move is to keep watching. That gap is what SentinelBench measures.
The benchmark packages 100 monitoring tasks into 10 self-contained web applications that replay scripted events on a live interface. The agent has to navigate pages whose state shifts underfoot, recognize when the target condition becomes true, and act on it. I led this work as first author alongside Adam Fourney, Amanda Swearngin, Hussein Mozannar, Gagan Bansal, Maya Murad, Rafah Hosn, and Saleema Amershi. All of it is public at github.com/microsoft/sentinel_environments.
Why long-running agents need a different benchmark
Existing web agent benchmarks such as WebArena test single-session navigation over state that never changes on its own. Long-running monitoring tasks are different: the target condition does not exist yet when the task begins, the correct action is often to wait, and cost accumulates the entire time the agent runs.
Most agent benchmarks hand the agent a goal whose answer already exists somewhere in the environment. The agent explores, finds it, and finishes. Monitoring tasks invert that shape. When a SentinelBench task begins, the condition has not happened yet. A benchmark for this work has to script the future, replay it on a schedule, and check whether the agent noticed.
The paper maps the landscape. WebArena offers 812 single-session navigation tasks across 4 sites with no temporal evolution. GAIA2 introduces timed events, though agents receive them through clean APIs and notification queues. SentinelBench requires discovery through the page itself, with no notification queue and no API shortcut. That design pushes the tradeoff we care about, responsiveness against cost, into plain view.
How SentinelBench works
Every SentinelBench task specifies an initial database state, a timed sequence of scripted events, and a SQL query that validates success. The 100 tasks split into 38 passive watch tasks, 42 active tasks requiring the agent to take actions, and 20 no-operation tasks where the condition never occurs and the correct behavior is silence.
Tasks vary along two axes. The first is what the agent must do: passive tasks succeed when the agent reports back through a contact form after the target event, active tasks succeed when the agent drives the environment database into the correct state, and no-operation tasks succeed when the agent stays quiet until the simulation ends. An agent that cries wolf is useless as a monitor.
The second axis is how the condition is judged. Absolute tasks (41 of them) can be verified from a single page snapshot, for example "watch the star count on this repo and let me know when it hits at least 2,000." Relative tasks (39) require comparing state across time, so the agent has to remember what the page looked like earlier. Everything is graded automatically.
Interactive demo: two agents watch the same repo
The task, straight from the benchmark: "Watch the star count on this repo and let me know when it hits at least 2,000." The sleep agent wakes on a fixed timer and takes a fresh screenshot every time, whether or not anything changed. The wait_for agent diffs the page quietly about once a second and only wakes the model when something actually moved. Press start and watch the meters.
Costs here are calibrated to the paper's per-task medians for GPT-5.4: $1.17 with sleep against $0.23 with wait_for, a 5.1 times gap. Reaction time is subtler. Over full runs, sleep actually reacted a little faster on median (42.3s against 51.7s); it just paid dearly for the privilege, and the cost gap grows to 9.7 times on forty-minute tasks.
The 10 simulated web environments
SentinelBench ships 10 web application replicas: MicroMail (email), MicroChat (team messaging), MicroDin (professional networking), MicroFy (music streaming), MicroGram (photo sharing), MicroHood (stock trading), MicroHub (code hosting), MicroLendar (calendar), MicroScholar (academic search), and MicroTube (video streaming). Each pairs a React frontend with a FastAPI backend over SQLite.
We built replicas instead of pointing agents at live websites for a simple reason: determinism. Events must fire at the same moment on every run, and databases must reset cleanly between tasks. Each environment is fully interactive, so MicroHub has repositories, issues, pull requests, and workflows, while MicroHood has a portfolio, market orders, a watchlist, and news.
The environments are populated from synthetic catalogs and 100 synthetic personas with per-environment subprofiles, so the same character shows up coherently in their email, calendar, and social feed. A server life cycle handles reset, ready, running, and completed states, so tasks run back to back without contamination. The whole thing runs on a laptop.
Interactive demo: explore the task taxonomy
The 100 tasks vary along two independent axes. Pick a value on each and see what that kind of task looks like, and what it takes to pass it.
What the agent must do
How the condition is judged
The four example prompts for tasks whose event does occur are quoted from the paper. The no-op entries are descriptions rather than quotes: the paper defines them by what never happens, so their prompts read exactly like the others.
What we measured: completion, reaction time, and cost
SentinelBench reports three metrics: task success, reaction time (the delay between an event occurring and the agent responding to it), and resource use, tracked as input and output tokens and converted to dollar cost from API pricing. Together they show whether an agent is correct, responsive, and affordable at the same time.
Success alone tells you very little about a monitoring agent. An agent that refreshes a page every two seconds catches most events and costs a fortune over long horizons. So we measure the tradeoff directly. Completion time counts the seconds the simulation runs before the agent responds or times out. Reaction time subtracts the moment the condition became true, counted only on successful tasks.
Cost is the metric most people underestimate. Agents self-report token usage through a sidecar file, converted into dollars using each model's API pricing. On a two-minute task, a wasteful strategy is a rounding error. On a task that runs all afternoon, every unnecessary refresh compounds. The benchmark makes that compounding visible.
Interactive demo: you are the monitoring agent
A live MicroHub activity feed will scroll below. Click the report button the moment your condition becomes true, and not a moment before. One run in five, mirroring the 20 no-op tasks out of 100 in the benchmark, the event never happens, and the only correct move is to never click at all.
For scale: on real SentinelBench tasks, GPT-5.4 posted median reaction times of 42.3 seconds with the sleep tool and 51.7 seconds with wait_for. That is not a fair comparison with your score. You watched one compressed feed for a few seconds with a single button to press; the agents watched a full web application over a window of up to ten minutes, paid tokens for every look, and had to navigate to a contact form to file their report.
Results across three models and two waiting tools
GPT-5.4 with a purpose-built wait_for tool solved 75 percent of SentinelBench tasks at a median cost of $0.23 per task. The same model with a plain sleep tool solved 68 percent at $1.17, about 5.1 times the cost. GPT-4o and Qwen 3.5:9B both landed near 48 percent, showing clear headroom.
We evaluated three models (GPT-5.4 on its low reasoning setting, GPT-4o, and Qwen 3.5:9B) crossed with two waiting strategies. The first gives the agent a sleep(time) tool that blocks unconditionally. The second gives it wait_for(condition, timeout), a tool we built that watches the page and wakes the model only when something changes. That tool choice mattered about as much as the model choice. For GPT-4o, sleep cost $0.29 per task against $0.13 with wait_for; for Qwen, $0.02 against $0.01.
The gap widens with time. At roughly forty minutes per task, GPT-5.4 with wait_for held a 69 percent success rate at a median cost of $0.48, while sleep fell to 56 percent at $4.65, a 9.7 times difference. Reaction time cuts the other way in places: sleep reacted faster for GPT-5.4 (42.3 seconds versus 51.7), while wait_for was far faster for Qwen (60.1 versus 123.8). Active tasks proved consistently harder than passive ones, and the weaker models struggled most on relative criteria. That is exactly what a benchmark should expose.
How to run SentinelBench
SentinelBench is MIT licensed at github.com/microsoft/sentinel_environments. A single ./start.sh script installs dependencies, builds the SQLite databases, and launches the API server, frontend, and harness shell in a tmux session. It needs Python 3.11+, Node 18+, and runs on Linux, macOS, or WSL2.
We wanted the setup cost to be one command, because a benchmark nobody runs is a paper appendix. After start.sh brings the stack up, the eval harness drives everything: you describe your agent in an eval_config.yaml (server URL, frontend URL, speed factor, and your agent's launch command), then run scenarios with the harness's run command and score them with grade. The harness itself needs no API keys; your agent brings its own.
The repository includes the environments, scenario files, synthetic catalogs, and the data-generation pipeline, so new tasks and environments are easy to add. If you are building a browser agent, the paper's numbers give you a baseline to beat.
What comes next
SentinelBench currently uses artificial event timing, conditions that persist once true, and a single-client design, and every task was authored and validated by hand. The next steps are realistic event distributions, fleeting conditions an agent can permanently miss, and scaled task generation so monitoring becomes a training signal as well as an evaluation.
We tried to be direct about limitations. Event timing is sampled artificially rather than drawn from real system traces, which a future agent could exploit. Conditions persist once true, which is forgiving; the real world is full of fleeting states that a distracted monitor misses forever. The environments are single-client and built for evaluation. And because tasks were written and validated manually, the dataset grows at human speed.
Each of those is a research direction. As agents take on work that spans hours instead of minutes, waiting attentively and cheaply becomes a core competency. Agents that wait well will be trusted with longer work. SentinelBench is how we check.
Abstract
AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: issuing tool calls, refreshing pages, searching for alternatives, or otherwise trying to force progress. This is the wrong approach for many long-running tasks, which are better served by a strategy of sustained attention. Instead, agents should monitor an environment, notice when an external event makes progress possible, then respond promptly without wasting resources while waiting. To measure progress on this class of tasks, we introduce SentinelBench, an open-source benchmark for time-evolving monitoring tasks. SentinelBench contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment. Each environment exposes a live web interface and replays a scripted sequence of events, requiring agents to navigate and reason about web pages whose state shifts underfoot. SentinelBench measures task completion, reaction time, and resource use, exposing the tradeoff between responsiveness and cost. We report results across three models and two browser-agent harnesses, establishing performance baselines for future comparison and demonstrating how agent design choices can dramatically impact key metrics. Together, these results show that SentinelBench distinguishes meaningful differences in agent behavior.
Frequently Asked Questions
How do you evaluate long-running AI agents?
SentinelBench runs each agent inside a time-scaled simulation. The harness first sends the agent through a redirect endpoint, decoupling browser startup from task timing. Events then replay on schedule, a fixed 30-second grace period absorbs jitter, and success is graded automatically by a SQL query against the environment database.
What is the difference between SentinelBench and WebArena?
WebArena evaluates single-session navigation where the environment holds still while the agent works. SentinelBench environments change during the task on an external schedule. The paper also contrasts GAIA2, ARE, AppWorld, and TheAgentCompany; the closest peers deliver events through APIs or notification queues, while SentinelBench makes the agent discover changes by watching ordinary web pages.
Is SentinelBench open source?
Yes. The code, all 10 environments, the scenario files, the synthetic catalogs, and the data-generation pipeline are released under the MIT license at github.com/microsoft/sentinel_environments. The evaluation harness itself requires no API keys, so you can bring any agent, commercial or local, and score it under identical conditions.
What is the wait_for tool in SentinelBench?
wait_for(condition, timeout) is a monitoring primitive we built for the baselines. It captures a textual snapshot of the page, loops about once per second computing diffs against that baseline, and passes only the changed blocks to the model to judge the condition. Periodic page reloads and rate limiting keep it honest and cheap.
How do monitoring tasks differ from regular agent tasks?
Regular tasks reward acting; monitoring tasks also reward restraint. A fifth of SentinelBench tasks are no-operation scenarios where the condition never occurs and any report is a failure. We also observed a failure mode unique to waiting agents: with sleep, models sometimes concluded the task before the target event had happened, effectively a negative reaction time.
How long do SentinelBench tasks run?
By default a task runs inside a ten-minute window, with the target event scheduled uniformly between 10 and 600 seconds in. A speed_factor parameter rescales event timing, so a 0.25 setting stretches the same scenarios to roughly forty minutes, which is how we studied waiting strategies as horizons grow.
What agent harness was used in the SentinelBench paper?
The baseline is a web browsing agent adapted from Magentic-UI, the open-source agentic system from Microsoft Research. It runs a screenshot-based tool-calling loop, receiving an image of the browser at each step. Tool choice reshaped its behavior: with wait_for, GPT-5.4 needed a median of 6 tool calls per task, against 19.5 with sleep.
BibTeX
@misc{maldaner2026sentinelbench,
title={SentinelBench: A Benchmark for Long-Running Monitoring Agents},
author={Matheus Kunzler Maldaner and Adam Fourney and Amanda Swearngin and Hussein Mozannar and Gagan Bansal and Maya Murad and Rafah Hosn and Saleema Amershi},
year={2026},
eprint={2606.05342},
archivePrefix={arXiv},
primaryClass={cs.AI}
}