llm-evaluator

What This Skill Does

The llm-evaluator is a sophisticated LLM-as-a-Judge system designed to automate the quality assurance of AI agent outputs within the OpenClaw ecosystem. By leveraging the advanced analytical capabilities of GPT-5-nano via Langfuse, this skill provides a standardized framework for evaluating model performance. It systematically assesses agent traces across four critical metrics: relevance, accuracy, hallucination detection, and overall helpfulness. Whether you are running a production deployment or iterating on agent prompts, this tool provides the observability required to maintain high standards of AI reliability. It bridges the gap between raw execution and data-driven improvements by storing all evaluation results directly within the Langfuse dashboard.

Installation

To integrate the llm-evaluator into your environment, use the OpenClaw command-line interface. Ensure your system has the necessary credentials configured for the OpenClaw framework, then execute the following installation command:

clawhub install openclaw/skills/skills/aiwithabidi/llm-evaluator-pro

Once installed, verify the setup by running the test suite included in the script directory to ensure connectivity with your Langfuse project instance.

Use Cases

This skill is indispensable for developers and businesses building AI agents. Common use cases include:

• Post-Deployment Auditing: Automatically score traces from the last 24 hours to ensure that user queries are being handled correctly.
• A/B Testing: Evaluate different prompt versions against the same query set to determine which yields higher accuracy.
• Hallucination Mitigation: Automatically flag or score outputs that contain non-factual information, allowing you to intercept bad responses before they reach the end user.
• Continuous Integration: Use the batch backfill feature to monitor the performance of your agent over time as you update your base model or system instructions.

Example Prompts

1. "OpenClaw, please run the llm-evaluator to score the most recent trace ID 98765 for accuracy and hallucination."
2. "I need to perform a quality check on our system; backfill scores for the last 50 unscored traces using the evaluator tool."
3. "Evaluate the helpfulness of the last interaction with the customer support agent and save the metrics to Langfuse."

Tips & Limitations

• Cost Efficiency: Because this skill uses GPT-5-nano, it is optimized for high-volume scoring without excessive API costs. However, monitor your Langfuse token consumption when backfilling large datasets.
• Granularity: While you can score all metrics at once, scoring individual metrics (e.g., just 'relevance') can be faster when you are iterating on specific performance improvements.
• Context Windows: Ensure your traces contain sufficient context for the judge model to perform an accurate assessment; if the input prompt or retrieved context is missing, the judge may be unable to verify factual accuracy.