llm-evaluator

What This Skill Does

The llm-evaluator skill is a sophisticated LLM-as-a-Judge framework integrated directly into the OpenClaw agent ecosystem. It leverages Langfuse to provide a robust observability and evaluation layer for your AI outputs. By utilizing the cost-efficient GPT-5-nano model, it evaluates individual interaction traces across four critical dimensions: relevance, accuracy, hallucination detection, and overall helpfulness. This skill transforms raw agent logs into actionable quality metrics, allowing developers to maintain high standards of performance without excessive manual review. It is designed to act as an automated quality assurance gate, providing standardized 0-1 scores that allow you to track the performance of your AI agents over time.

Installation

To integrate this skill into your environment, use the OpenClaw command-line interface. Run the following command in your terminal:

clawhub install openclaw/skills/skills/aiwithabidi/llm-evaluator

Ensure that your environment variables for Langfuse and the required LLM API keys are correctly configured in your project settings to allow the evaluator to communicate with the trace management dashboard.

Use Cases

• Continuous Monitoring: Regularly audit production logs to ensure your agent remains helpful and accurate as you update system prompts or switch base models.
• Regression Testing: During development, run the evaluator against a set of 'gold standard' test cases to ensure new changes do not introduce hallucinations or relevance drops.
• Automated QA: Integrate this into your CI/CD pipeline to flag low-score outputs for manual human review before they reach end-users.
• Historical Analysis: Use the backfill functionality to gain insights into the historical performance of your agent after updating your evaluation criteria.

Example Prompts

1. "Evaluate the last 50 traces in our production log for hallucinations and give me a summary of the average accuracy score."
2. "Run the relevance evaluator on trace ID 8892 and output the detailed reasoning for the score."
3. "Backfill the scoring for the past 24 hours of activity to see if our recent model fine-tuning improved helpfulness scores."

Tips & Limitations

• Cost Efficiency: While GPT-5-nano is optimized for cost, running large-scale backfills on thousands of traces will still incur API costs. Use the --limit flag during batch operations to manage expenses.
• Scope: The quality of the evaluation is dependent on the clarity of your initial system prompts. If the agent's instructions were ambiguous, the evaluator may struggle to determine 'accuracy'.
• Integration: Ensure your Langfuse project is properly initialized, as the skill relies on valid trace IDs stored within that ecosystem to function correctly.