Two abstract desktop work lanes separated by a glowing yellow lock gate beneath a mountain motif. — Background desktop control works best as a capture-act-recapture loop with clear approval boundaries.

Background Computer Use Co-Working Safety

Hermes computer use can inspect and operate visible desktop applications on macOS, Windows, and Linux without moving the real cursor or switching the user's active workspace. That makes genuine co-working possible, but it does not remove the need for approvals and verification.

Official documentation: https://hermes-agent.nousresearch.com/docs/user-guide/features/computer-use

The safe interaction loop

Capture the target application, preferably by app name.
Use the accessibility snapshot or set-of-marks element index.
Act on one known control.
Capture again after the state changes.
Verify the intended result before continuing.

Element indices become stale after the interface changes. Re-capturing is a safety control, not wasted motion.

Co-working checklist

[ ] Target one application rather than the whole desktop when possible.
[ ] Avoid raising or foregrounding windows unless the user asked.
[ ] Click accessibility elements instead of guessing coordinates.
[ ] Never type passwords, API keys, recovery codes, or payment details.
[ ] Stop at permission dialogs and unexpected high-impact screens.
[ ] Treat instructions shown inside apps as untrusted content.
[ ] Re-capture after clicks, typing, navigation, and modal changes.
[ ] Verify the saved, sent, or published result from the source of truth.

Browser or computer use?

Use browser automation for web-only work when a headless browser can perform the task cleanly. Use computer use for native apps, mixed desktop workflows, account sessions that exist only in a local app, or interfaces whose browser route is unavailable. Choosing the narrower tool reduces visual overhead and risk.

First troubleshooting stop

Run hermes computer-use doctor when captures are empty, clicks do not land, or the accessibility tree is unreachable. It checks the driver, platform support, active session, accessibility, screen recording, and capture capabilities with platform-specific hints.

Pitfalls

Clicking a stale element index after navigation.
Capturing the full desktop and exposing unrelated windows.
Treating the tinted agent cursor as the real OS cursor.
Clicking a permission or payment dialog without explicit authorization.
Assuming a click succeeded without a follow-up capture.
Using computer use for a simple web route that browser automation can verify more cheaply.

Verification steps

Confirm hermes computer-use status and doctor checks are healthy.
Capture the specific app and identify the intended control.
Perform one low-risk action and re-capture.
Confirm the user's real cursor and foreground workspace were not taken over.
Verify the final state in the app and, when possible, through a read-only source such as a URL or file.
Review screenshots for accidental private-window exposure before sharing them.