handsfree-windows-control
Guide skill for controlling native Windows apps (UIA) and web browsers (Playwright) via the handsfree-windows CLI. Use when you need to automate or test desktop applications or websites on a Windows machine: launching apps from Start menu, discovering UI controls without guessing, clicking/typing in native apps, opening/snapshotting/clicking in browsers, recording and replaying YAML macros that mix desktop and web steps. REQUIRES handsfree-windows CLI (auto-installed on first use via setup.py).
Why use this skill?
Master Windows and web automation with the Handsfree Windows Control skill. Easily control native apps via UIA and browsers via Playwright.
Install via CLI (Recommended)
clawhub install openclaw/skills/skills/lijinlar/handsfree-windows-controlWhat This Skill Does
The handsfree-windows-control skill acts as an automation bridge for Windows environments. It leverages the handsfree-windows CLI to provide a unified interface for interacting with both native Windows applications (via Microsoft UI Automation) and modern web browsers (powered by Playwright). Unlike simple macro recorders, this skill emphasizes intelligent discovery. It allows agents to inspect visual trees, identify specific UI elements by control type or name, and execute precise actions without relying on brittle coordinate-based clicks. Whether you need to automate legacy desktop software or perform complex browser-based data gathering, this skill provides the underlying primitives to make that happen reliably.
Installation
Installation is handled through a seamless bootstrap process. Upon running the skill, the setup.py script automatically clones the necessary CLI utilities, configures an editable pip installation, and sets up Playwright with the required browser binaries.
- Run
python scripts/setup.pyin your environment to initialize. - Verify the installation by executing
python scripts/check_setup.py.
For advanced users, flags like --no-browser allow for a lighter installation footprint if desktop-only automation is required. Note that this process populates local folders such as ~/.handsfree-windows/ for persistent browser profiles and session states.
Use Cases
This skill is ideal for business process automation, regression testing, and cross-platform workflows. Example use cases include:
- Software Testing: Automatically opening a desktop app, entering data into specific fields, and verifying state transitions.
- Data Aggregation: Navigating to a corporate web portal, logging in, extracting data, and pasting it into a desktop spreadsheet application.
- Workflow Automation: Combining a desktop startup sequence with browser-based research, then capturing snapshots of the results for reporting.
Example Prompts
- "Open Outlook, find the button labeled 'New mail', and click it so I can start drafting."
- "Open Chrome to example.com, inspect the page for an element containing 'Sign in', and print the structure."
- "Create a macro that launches my ERP desktop application, navigates to the dashboard, and then opens our company website in the browser."
Tips & Limitations
- Safety First: Always use
hf treeorhf inspectbefore acting. Never assume a window or button exists in the same location. - Authentication: Do not attempt to input credentials programmatically. Navigate to the login page and pause for the human to handle MFA or secure entry.
- Cleanup: If the skill consumes too much space, remember that
pip uninstalland manual folder deletion can fully purge browser profiles and binaries. - Destructive Actions: Always ask for user confirmation before executing commands that delete, submit, or send data.
Metadata
Not sure this is the right skill?
Describe what you want to build — we'll match you to the best skill from 16,000+ options.
Find the right skillPaste this into your clawhub.json to enable this plugin.
{
"plugins": {
"official-lijinlar-handsfree-windows-control": {
"enabled": true,
"auto_update": true
}
}
}Tags(AI)
Flags: network-access, file-write, file-read, code-execution