Control

What This Skill Does

The Control skill is the flagship automation engine for OpenClaw, enabling the agent to interact with a computer system exactly like a human user. By bridging the gap between digital intent and desktop actions, it allows OpenClaw to manipulate the mouse, keyboard, and window state. Whether you need to navigate legacy software, automate repetitive GUI-based tasks, or perform complex multi-step workflows across different applications, the Control skill provides the necessary precision to execute those operations reliably. It features advanced screen reading capabilities, allowing the agent to "see" the desktop using image recognition to identify buttons, menus, and input fields dynamically.

Installation

To integrate this skill into your environment, use the OpenClaw management utility via your terminal:

clawhub install openclaw/skills/skills/breckengan/control

Ensure that your system satisfies the prerequisite Python dependencies, including pyautogui, pillow, and opencv-python. Please verify that your operating system permissions allow the OpenClaw process to control accessibility features, as this is required for cross-application interaction.

Use Cases

• Automated Data Entry: Moving data from a web browser into desktop-bound legacy ERP or accounting software that lacks APIs.
• System Workflow Orchestration: Automatically opening, configuring, and arranging multiple windows (e.g., IDEs, terminals, and documentation) into a preferred workspace layout upon startup.
• Quality Assurance Testing: Simulating user interaction sequences to test desktop software responsiveness and UI flow.
• Periodic Maintenance: Scheduling the agent to perform cleanups, disk operations, or report generation through software interfaces that require manual clicks.

Example Prompts

1. "Open the browser, navigate to the dashboard, and click the Export button, then save the file to my Documents folder."
2. "Rearrange all my open windows so the terminal is on the left half and the code editor occupies the right half of the screen."
3. "Monitor the screen for the error popup, and if it appears, click 'Cancel' and log the timestamp in the activity file."

Tips & Limitations

• Failsafe: Always keep your mouse cursor ready to move to the corner of the screen; this triggers the emergency stop if an automation loop goes wrong.
• Precision: Use image recognition (OpenCV) rather than hard-coded coordinates whenever possible, as window resizing can invalidate static coordinate mapping.
• Speed: While the skill supports instant actions, human-like speed settings (duration > 0) are less likely to trigger anti-bot detection in sensitive applications.
• Approval Mode: If you are automating high-risk actions like deleting files, ensure Approval Mode is enabled to prevent accidental data loss.