desktop-mouse

What This Skill Does

The desktop-mouse skill provides a robust interface for local mouse control within the OpenClaw ecosystem. By leveraging a custom ydotool wrapper, this skill allows the OpenClaw agent to interact with the host system's graphical environment directly. It translates natural language intents into specific mouse events, including relative cursor movement, absolute coordinate positioning, and standard button clicks. This is particularly useful for automation tasks where non-API-accessible applications need to be interacted with or where manual UI navigation is required for specific desktop workflows. It acts as an abstraction layer over low-level input protocols, ensuring that your AI agent can 'see' and 'interact' with the screen as a human would.

Installation

To integrate this skill into your OpenClaw environment, execute the following command in your terminal:

clawhub install openclaw/skills/skills/oguzhaslak/molt-mouse

Ensure that your host system has the necessary dependencies for ydotool to function, as the skill relies on the system's ability to inject input events into the Linux input stack. You may need root or specific group permissions depending on your OS configuration.

Use Cases

This skill is ideal for automating repetitive desktop tasks that lack CLI alternatives. Common scenarios include:

• Clicking buttons in legacy applications that do not expose a REST API.
• Automating routine data entry that requires clicking between fields.
• Performing multi-step workflows that involve UI toggles, menus, or context-sensitive pop-ups.
• Assisting with screen-based testing environments where physical interaction simulation is required.

Example Prompts

1. "Move the mouse 50 pixels to the right and click the left button."
2. "Please move the cursor to absolute position 500 by 500 and double click the left mouse button."
3. "Hold the left mouse button down and drag it across the screen to select the text area."

Tips & Limitations

• Precision: When using relative movement commands, ensure the agent understands the current state of the cursor. It is recommended to use absolute positioning (abs) when possible to ensure reliability.
• Permission Errors: If the skill fails, check your permissions; ydotool often requires access to /dev/uinput.
• Ambiguity: If your request lacks coordinates or specific button identifiers, the agent will prompt for clarification. Always specify the direction or coordinate system clearly to avoid unintended actions.