s3-bulk-upload

What This Skill Does

The s3-bulk-upload skill provides an intelligent, automated solution for organizing large collections of files into Amazon S3 buckets. Instead of dumping files into a single flat directory, which can lead to performance degradation and management overhead, this tool categorizes uploads based on the first character of the filename. Files are sorted into specific prefixes: lowercase letters (a-z), numeric segments (0-9), and a catch-all folder for special characters. By utilizing symlink staging or direct command-line uploads, this skill ensures your S3 storage hierarchy is deterministic and clean from the moment of ingestion.

Installation

To integrate this utility into your OpenClaw environment, execute the following command in your terminal:

clawhub install openclaw/skills/skills/6mile-puppet/s3-sort

Once installed, ensure your AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) are correctly configured via aws configure to allow the agent to communicate with your S3 buckets.

Use Cases

• Log Archiving: Organize thousands of log files automatically by date-string or name to optimize S3 query speeds and prefix listing.
• Asset Management: Cleanly structure a bucket filled with user-uploaded files or images to prevent hitting object-per-folder limits.
• Data Migration: Bulk upload local directory contents to S3 while enforcing a structured namespace, making it easier to sync and manage downstream processing tasks.

Example Prompts

1. "OpenClaw, use the s3-bulk-upload skill to move all files from my ./exports directory to the 'my-production-bucket' using the sync mode."
2. "Can you perform a dry run of the s3-bulk-upload on my local './data' folder to see how the organization will look in my 'backup-bucket'?"
3. "Upload all files in the current directory to my-bucket, ensuring that files starting with numbers go into the '0-9' folder and apply the STANDARD_IA storage class."

Tips & Limitations

For massive datasets (thousands of files), always prefer the --sync method. This approach creates a temporary staging directory using symlinks to prevent unnecessary data duplication on your local disk before the sync operation. Be aware that this skill is case-insensitive regarding the first character; 'A' and 'a' will be grouped in the same 'a/' prefix. If your filenames start with non-alphanumeric characters, they will be consistently routed to the '_other/' directory. Ensure your bucket permissions allow 's3:PutObject' and 's3:ListBucket' to avoid errors during execution.