expertpack-eval

What This Skill Does

The expertpack-eval skill serves as the analytical backbone for the ExpertPack ecosystem. It allows developers and AI engineers to quantitatively measure the unique value proposition of their custom AI knowledge packs by calculating an Esoteric Knowledge (EK) ratio. This ratio indicates how much of your knowledge pack consists of information that frontier LLMs (such as GPT-4, Claude 3.5, or Gemini) do not possess natively in their training weights. By leveraging LLM-as-a-judge patterns, the skill performs automated 'blind probes'—testing a model's ability to answer questions without your pack, then comparing those answers to ground truth. Furthermore, the skill facilitates rigorous quality control through automated endpoint evaluation, ensuring that your agent behaves reliably across varying levels of question complexity, from basic queries to advanced, out-of-scope scenarios.

Installation

To integrate this evaluation suite into your environment, use the OpenClaw command-line interface. Ensure you have your environment configured with a valid OpenRouter API key, as the skill requires external model access to perform its evaluation metrics. Install the skill using the following command:

clawhub install openclaw/skills/skills/brianhearn/expertpack-eval

Once installed, ensure your environment variables are correctly exported, or verify your OpenClaw authentication profile to allow the script to handle API requests automatically.

Use Cases

• Quantifying Pack Value: Use the EK ratio calculation during development to prove the efficacy of niche or proprietary data sets.
• Quality Assurance: Run baseline eval sets against your agent whenever you modify your pack's retrieval logic, instructions, or model configuration to prevent regressions.
• Optimizing Agent Training: Use the 'out-of-scope' question results to determine if your agent is over-fitting or hallucinating when it should correctly deny knowledge.
• Deployment Readiness: Provide the generated YAML report as proof of quality before deploying a pack-powered agent to a production environment.

Example Prompts

1. "Measure the EK ratio for the current expertpack in my local directory to see how much unique knowledge it provides."
2. "Run the blind probe eval on the new policy documentation pack using GPT-4-mini and Claude Sonnet as comparison models."
3. "Initiate an automated quality eval against the active endpoint using the questions.yaml test set."

Tips & Limitations

• Budget Awareness: Since this skill makes numerous calls to multiple frontier models for evaluation, keep an eye on your OpenRouter usage costs, especially when running large-scale eval sets with a high number of 'sample' iterations.
• Judge Reliability: Always prioritize Claude Sonnet for the 'judge' role, as smaller models often struggle with consistent scoring logic, which can skew your EK ratio accuracy.
• Data Privacy: Ensure that your eval test sets do not contain sensitive or PII data, as these queries are sent to external API endpoints for analysis during the evaluation process.