Autoresearch Loop Skill

Karpathy's autoresearch methodology applied to improving Claude skills, n8n workflows, system prompts, and business processes.

Core idea: Define what "better" means. Lock everything except the artifact being improved. Propose a change → test → measure → keep or discard → repeat until a stopping condition is met.

When NOT to use this loop:

• You can't define a single measurable metric (e.g. "improve my writing style" — too subjective)
• The artifact is too large to evaluate cheaply in a fixed budget
• There's no fixed eval set (or you can't create one) — without a stable yardstick, you're just guessing
• You need to improve two interdependent artifacts simultaneously — do them sequentially instead
• The artifact is a one-time document (a single client proposal, a one-off report) — the loop is for artifacts that will be reused and improved over time. A one-time deliverable has no future eval value; just write it well directly

If you can't answer "what number tells me if this experiment worked?", stop and define that first.

The methodology is format-agnostic: The loop works for any artifact type — code, prompts, documents, design systems, API configurations, process specs — as long as you can define an artifact, a metric, and a repeatable eval. For novel artifact types not covered by the examples below: walk through the setup phase (artifact → metric → eval → budget) and creatively define each. A Figma component library's metric could be a checklist pass rate (accessibility, consistency, coverage); its eval could be test scenarios ("render a data table", "create a form with validation states") scored against that checklist. Start with a small eval (5–10 test cases) to validate the metric produces meaningful signal before committing to a full campaign.

Setup Phase

Before the loop starts, establish these five things with the user:

1. The Artifact (What You're Improving)

The single file, document, workflow, or process being iteratively modified. Think of this as train.py in Karpathy's repo — the one thing the agent edits.

Examples:

• A SKILL.md file
• An n8n workflow JSON
• A system prompt
• An SOP document
• A business process description

Fixed files: Identify what must NOT change — the evaluation criteria, input test cases, external integrations. These are your prepare.py.