file-deduplicator

What This Skill Does

The File-Deduplicator is a robust OpenClaw skill designed to help users identify, manage, and remove redundant files that clog storage systems. By utilizing advanced content-based hashing algorithms (MD5), the skill ensures that only truly identical files are flagged, rather than relying solely on file names which can often be misleading. It offers a comprehensive suite of tools for scanning recursive directories, analyzing space-saving potential, and safely disposing of, archiving, or moving duplicates. Whether you are a digital archivist cleaning up a decade of disorganized backups or a professional developer needing to purge duplicate assets from a project, this skill provides the precision and safety required to maintain a pristine file system.

Installation

To integrate this utility into your OpenClaw environment, use the command-line interface to pull the skill directly from the repository:

clawhub install file-deduplicator

Use Cases

• Storage Optimization: Automatically scan high-capacity drives to reclaim gigabytes of lost storage taken up by redundant media or documentation.
• Digital Decluttering: Identify and organize scattered files across multiple folders, ensuring only one 'master' version remains in your primary directory.
• Development Cleanup: Rapidly detect identical assets or binary files across large code repositories to reduce project weight and build times.
• Archive Management: Use the 'move to archive' feature to consolidate duplicates into a single secure location without immediate deletion, ensuring no data is lost during the cleanup process.

Example Prompts

1. "Find all duplicate files in my Documents folder and show me a report of how much space I could save if I deleted the older versions."
2. "Perform a dry-run scan on my Downloads directory and list the duplicate files but don't delete anything yet."
3. "Scan my project directory and remove all duplicate assets, keeping the most recently modified files and moving the rest to the 'archived-assets' folder."

Tips & Limitations

• Safety First: Always perform a 'dry-run' before executing mass deletions to verify the results.
• Whitelist: Utilize the whitelist feature for critical system folders to prevent the tool from interfering with essential application data.
• Performance: Content hashing is resource-intensive. Running this on extremely large datasets (terabytes) may take significant time and CPU cycles. Consider running it during off-peak hours.
• Permissions: Ensure the OpenClaw agent has read/write permissions for the target directories; otherwise, the deletion or moving functions will fail.