Parquet Converter

What This Skill Does

The Parquet Converter is a specialized OpenClaw agent skill designed to bridge the gap between legacy construction data formats and modern, high-performance data lakehouses. It facilitates the conversion of bulky, slow-to-query formats like CSV or Excel into the Apache Parquet format. By leveraging columnar storage, Parquet provides superior compression and faster retrieval speeds, which are essential when analyzing multi-year construction project data, financial records, or project schedules. The skill handles schema definitions, partition management (e.g., partitioning by project status or type), and compression settings using robust tools like Snappy, Gzip, or Zstd.

Installation

To integrate this skill into your environment, run the following command in your terminal: clawhub install openclaw/skills/skills/datadrivenconstruction/parquet-converter

Use Cases

• Project Analytics: Converting massive project cost trackers to Parquet to run sub-second queries on budget vs. actual cost across thousands of projects.
• Data Lakehouse Integration: Transforming site sensor data or daily logs into partitioned Parquet files for seamless ingestion into platforms like Snowflake, Databricks, or Amazon Athena.
• Storage Optimization: Drastically reducing the physical storage footprint of archival construction data without sacrificing data integrity or type definitions.

Example Prompts

1. "Convert all CSV files in the /project_data/costs directory to Parquet format using Snappy compression to optimize our current storage."
2. "I need to update the project data schema. Can you re-process the latest construction reports using the standard 'projects' schema and partition them by status?"
3. "Summarize the conversion results for last month’s schedule data. How much space did we save by switching from CSV to Parquet?"

Tips & Limitations

• Pre-defined Schemas: Always prefer using the pre-defined schemas in the ParquetConverter class for consistency across your organization's datasets.
• Partitioning Strategy: Be mindful of your partition columns. Choosing high-cardinality columns (like unique transaction IDs) for partitions can lead to an excessive number of small files, which negates the performance benefits of Parquet. Stick to low-cardinality metadata like 'project_status' or 'project_type'.
• Memory Usage: While Parquet is memory efficient, performing very large conversions may require sufficient RAM to handle the buffer operations before writing to disk.