memory-bench-pioneer

What This Skill Does

The memory-bench-pioneer skill provides a peer-review-grade evaluation suite for agent memory systems, specifically designed to function alongside the agent-memory-ultimate architecture. It serves as a rigorous testing framework to measure, analyze, and report on the efficacy of AI memory. By running standardized benchmarks, including LLM-as-a-judge assessments, nDCG, MAP, and MRR metrics with 95% confidence intervals, developers can gain deep insights into their agent's retrieval performance. The skill also performs ablation studies to isolate the impact of mechanisms like spreading activation, ensuring that memory improvements are data-driven rather than anecdotal. All results are anonymized and can be submitted to the ENGRAM and CORTEX research initiatives, fostering a collaborative, evidence-based approach to AI memory design.

Installation

To add this capability to your agent, execute the following command in your terminal:

clawhub install openclaw/skills/skills/globalcaos/memory-bench-pioneer

Ensure you have the gh CLI installed if you intend to use the automated PR submission workflow, and verify that your openai API key is configured if you prefer the higher-accuracy GPT-4o-mini judge method.

Use Cases

• Research Benchmarking: Contribute to the ENGRAM and CORTEX papers by providing anonymized, statistically significant data about memory retrieval performance.
• System Optimization: Identify bottlenecks in your agent's retrieval logic by using ablation studies to see how specific features affect metrics like Hit Rate and Precision.
• Cross-Site Comparability: Use the standardized test set to ensure your agent's memory performance is measured using the same criteria as other OpenClaw users, allowing for objective comparison.
• Longitudinal Tracking: Run benchmarks over time to ensure that memory system updates, consolidation, and pruning lead to actual performance gains rather than degradation.

Example Prompts

1. "Run a full memory performance assessment using the GPT-4o-mini judge and include an ablation analysis."
2. "Collect my current memory statistics, review the summary, and generate a pull request for the ENGRAM research repository."
3. "Evaluate my agent's retrieval quality against the standard test set and output the results for my internal review."

Tips & Limitations

For the most accurate results, always aim for at least 30 queries across the full spectrum of difficulty and type (semantic, episodic, procedural, strategic). Note that the openai judge incurs a small cost (approx. $0.01 per run), while the local judge option is free but less precise. Your privacy is paramount; the collect.py script ensures that your actual memory content, hostnames, and user queries are never included in the exported report. Only aggregate statistics, histograms, and anonymized performance metrics are shared in the PR submission. Always verify the agent-memory-ultimate configuration before beginning a benchmark run.