computer-use

What This Skill Does

The computer-use skill provides OpenClaw agents with full desktop GUI control on headless Linux environments. By utilizing a virtual frame buffer (Xvfb) and the XFCE desktop environment, this skill creates a virtual display (DISPLAY=:99) at 1024x768 resolution. This enables your AI agent to interact with desktop applications exactly like a human user. It supports 17 distinct mouse and keyboard actions, including clicking, dragging, scrolling, text entry, and key combinations. It is the bridge between terminal-based automation and the complex visual world of modern desktop applications.

Installation

To integrate this skill, first ensure your system has the required X11 dependencies installed via: sudo apt install -y xvfb xfce4 xfce4-terminal xdotool scrot imagemagick dbus-x11 chromium-browser. Once the environment is ready, install the skill package directly through the OpenClaw hub: clawhub install openclaw/skills/skills/bodii88/computer-use-1-0-1. Ensure your environment variables are configured with export DISPLAY=:99 so the agent can interface with the virtual desktop display correctly.

Use Cases

This skill is perfect for automating tasks that lack accessible APIs. Use it to: 1) Browse websites that require complex interaction, authentication, or JavaScript execution that standard scrapers cannot handle. 2) Manage GUI-only administrative software or legacy desktop applications on remote VPS instances. 3) Perform end-to-end testing of web or desktop software by simulating real user clicks and keystrokes. 4) Data extraction from visual dashboards or non-exportable UI elements.

Example Prompts

1. "Open the Chromium browser, navigate to the dashboard at monitoring.internal, and take a screenshot of the current system health graph."
2. "Locate the login field on the desktop app, type my username and password, then press Enter and wait 5 seconds for the home screen to load."
3. "Scroll through the current PDF document in the viewer, select all text using a mouse drag operation, and copy it to the clipboard."

Tips & Limitations

• The origin point (0,0) is always the top-left corner of the 1024x768 screen.
• Always capture a screenshot before acting to confirm UI state; the agent is blind without visual feedback.
• Use ctrl+End for quick scrolling in long browser pages.
• Remember that complex actions require small delays; the skill handles text chunking, but custom GUI sequences may need a wait.sh call to ensure the UI has finished rendering before your next interaction.