Computer Use Without Taking Over the Mac
Computer use should feel like a careful coworker, not a hijacker. Background control can inspect and operate apps without moving the user’s real cursor.
When this matters
This page is for operators who want Rocky to produce usable work without turning the system into a mystery box. Use it when you need a practical path, a clear verification step, and a boundary between suggestion and action.
The operating pattern
- Capture first. Take an app-scoped screenshot or accessibility snapshot before acting.
- Click by element. Use stable element indices rather than guessing coordinates.
- Avoid secrets. Never type or inspect passwords, tokens, 2FA codes, or payment data.
- Verify state changes. Capture again or read back status after every important click.
- Prefer APIs when possible. Desktop control is for real GUI-only surfaces, not routine file edits.
Pre-flight checklist
- The target app is clear
- No personal tabs are touched
- Permission/payment dialogs are not clicked
- Secrets stay with the user
- After-action verification is captured
Common failure modes
- Stale UI: Old element numbers can click the wrong thing.
- Secret exposure: Screenshots after secret entry are unsafe.
- Overusing GUI: APIs and files are often more reliable and auditable.
Verification
A page is not done because it was drafted. Verify the source, run the workflow, inspect the output, and record what changed. If a step touches money, customers, accounts, permissions, or private data, keep it behind an explicit human approval gate.