experiment-tracker

What This Skill Does

The experiment-tracker skill is a robust infrastructure component designed to bring order to the often chaotic workflow of machine learning research. By acting as an intelligent wrapper around standard logging tools like MLflow, Weights & Biases (W&B), or SpecWeave's own native tracking engine, it ensures that every experiment—from hyperparameters to resulting artifacts—is logged, versioned, and tied directly to a SpecWeave increment. It eliminates 'knowledge drift' by documenting the context behind model iterations within living, persistent documentation, ensuring that team members can trace back exactly why a model performed the way it did months after the initial training.

Installation

To integrate this skill into your environment, use the OpenClaw CLI tool. Ensure you have proper permissions to the project directory before running the following command:

clawhub install openclaw/skills/skills/anton-abyzov/sw-experiment-tracker

Once installed, the agent will automatically detect the presence of MLflow or W&B in your dependency tree and provide a unified interface to control them, or fall back to native logging if no external tools are found.

Use Cases

• Model Reproducibility: Ensure that every model checkpoint can be recreated by mapping code commits to specific hyperparameter configurations.
• Collaborative Research: Maintain a central repository of "decision logs" that explain why specific algorithms or features were selected, preventing redundant experimentation.
• Hyperparameter Tuning: Automatically track parameter sweeps and compare metrics like accuracy, precision, and F1-score across different iterations using the built-in comparison engine.
• Knowledge Transfer: When a team member leaves a project, the living documentation associated with the increment folder provides a complete historical narrative of the research journey.

Example Prompts

1. "Track the current experiment with MLflow and log the accuracy metric after the final epoch."
2. "Compare the experiments in this increment and generate a summary report of the best-performing model."
3. "Configure Weights & Biases for my latest training run and save the metadata to the current SpecWeave increment."

Tips & Limitations

• Directory Hygiene: Always ensure your increment folders are properly initialized before running experiments; the tool relies on the SpecWeave directory structure to maintain traceability.
• External APIs: If using W&B or MLflow as remote backends, ensure your environment variables (like WANDB_API_KEY) are securely set, as this skill facilitates the bridge between local code and cloud-based tracking servers.
• Limitations: The skill is optimized for structured ML workflows; it may require custom configurations if you are using specialized non-standard deep learning frameworks that do not integrate cleanly with typical logger callbacks.