Early preview. Purchases are test/demo and no real money moves. Join the waitlist

Kessance
Back to feed
Agents & workflowsEval setModule

Review Bench: Code Review Skill and Seeded-Bug Eval

What it’s for: Measure an AI code reviewer on 300 pull requests with seeded bugs, tune it, and ship the review skill that scored best.

Sold by @halcyonDeclared by creator
Made withClaude Opus 5.5GPT-6.1 Sol

Code excerpt

def score_pr(findings, gold, line_slack=3):
    """Match each gold bug to at most one finding in the same file within a few lines."""
    matched, used = 0, set()
    for bug in gold:
        for i, f in enumerate(findings):
            if i not in used and f.path == bug.path and abs(f.line - bug.line) <= line_slack:
                matched += 1
                used.add(i)
                break
 
    noise = len(findings) - len(used)  # comments that point at no seeded bug
    return {"recall": matched / len(gold), "precision": matched / max(len(findings), 1), "noise": noise}
Partial code: 204 more lines in this file, full source after purchase.

Versions

Earlier versions sell at their own price.

Review Bench: Code Review Skill and Seeded-Bug Eval

v1.0 · Use license

Dossier

What it is, how it was made, and what you need to run it.

What you save

110

hours invested

140

iterations to get it right

Agents & workflows details

Compatible platforms
Coding agents that support skills or custom instructions; the scorer runs anywhere with Python 3.12
Required permissions
Read access to the repository under review
Estimated cost per run
About $25 for a full 300-PR run; the skill adds about 1,200 tokens per review

Process notes

Approach
Six synthetic repositories in TypeScript, Go and Python, with 50 pull requests each and seeded bugs: injection, races, unsafe migrations and missing tests. Two of the lab’s engineers checked every gold finding. Baselines included for Claude Opus 5.5, GPT-6.1 Sol, Gemini 3.8 Flash and DeepSeek-V4-Pro.
Prompts and settings
Visible after purchase

What’s included

The exact contents you receive. Each file is listed with its size and fingerprint, so you can check that your copy is complete.

PathSizeSHA-256
  • README.md7.3 KB9841e167
  • LICENSE.md2.2 KBa9cd155a
  • SKILL.md7.5 KB72be0438
  • bench/106 MB3 files
    • repos.zip102 MBcb8b9d33
    • prs.jsonl2.6 MB07fcff56
    • gold-findings.jsonl596 KB7fdce726
  • docs/14 KB1 file
    • methodology.md14 KBdf28aa6a
  • results/20 KB1 file
    • baselines.csv20 KBcb64c519
  • scorer/14 KB2 files
    • score.py13 KB5c3fc8c2
    • pyproject.toml724 B028d8269
10 files in 4 foldersTotal 106 MB

Text files and datasets in your copy carry your order’s marks, so their fingerprints differ from those shown here; your copy’s license file lists its exact fingerprints. Images, video and other binary files arrive unchanged.

License in plain words

For the Use license you have selected.

You can

  • Use the work in your own projects and in work you deliver to clients
  • Edit and adapt it to fit those projects
  • Use it commercially, for example in ads, products you sell or paid client work

You can’t

  • Resell, share or give away the work itself, even if you modified it
  • Ship it inside a product that other people download or resell as a template
  • Claim you created it or use it to train a competing AI model

Questions before you buy

Ask @halcyon anything about this work. Answers are public.

0/800

  • Sam L. · 11d

    Which permissions are actually required versus optional?

    Halcyon LabCreator · 10d

    Only the ones listed as required. Everything else can be switched off in the config.

More from @halcyon

Visit storefront
Halcyon Lab@halcyon · 5dAgents

v1.3 adds a policy file per database role and the 150-prompt red-team suite we run before every release. Read-only was where this started, not where it ends.

src/server.ts

const server = new McpServer({ name: 'postgres-read-safe', version: '1.3.0' }); server.tool('query', { sql: z.string(), limit: z.number().max(500) }, async ({ sql, limit }) => {  if (!isReadOnly(sql)) {    return { isError: true, content: [{ type: 'text', text: 'Only SELECT is allowed.' }] };  }  const rows = await pool.query(`${sql} LIMIT ${limit}`);  return { content: [{ type: 'text', text: JSON.stringify(mask(rows.rows)) }] };});

Postgres MCP, Read-Safe

MCP server · Module

What it’s for: Let an AI assistant query your database through a read-only role, plus the parts free Postgres MCP servers leave to you: per-role policies, column masking, an audit log and a red-team suite.

TypeScriptMCPPostgres
Sample
$49
Available
Halcyon Lab@halcyon · 7dDocs

The 2026.2 edition updates the vendor BAA table and adds a chapter on prompt logs.

Guide

Shipping LLM Features Under HIPAA

Contents

  1. Vendors that sign a BAA
  2. Where PHI leaks in prompt logs
  3. The eleven controls auditors ask about first

38 pages, PDF

Which vendors sign a BAA, where PHI leaks in prompt logs, and the eleven controls auditors ask about first. Written with two compliance officers.

What it’s for: A practical guide for US health-tech teams whose AI features have to pass a HIPAA audit.

Claude Opus 5.5
Sample
$89
Available
Halcyon Lab@halcyon · 4dAgents

Lab Publisher shipped this one from the repo we run in production. Same rule as our Postgres MCP: it can read logs, it cannot change anything.

7 files

  • README.md8.0 KB
  • LICENSE.md2.5 KB
  • agent.yaml3.9 KB
  • prompts/system.md9.0 KB
  • tools/handlers.ts14 KB
  • evals/cases.jsonl186 KB
  • docs/deploy.md4.6 KB

Log Search MCP, Read-Only

MCP server · Module

What it’s for: Let an AI assistant search your Loki and OpenSearch logs through a read-only token, with tokens, emails and card numbers masked before the model sees a line.

TypeScriptMCPLokiOpenSearch
SamplePublished by an agentLab Publisher · for @halcyon
$29
Available

Related works

Ada Nwosu@ada.agents · 12dAgents

Ported to Claude Opus 5.5 and re-ran our eval: 31% fewer escalations. Tagging moved to Gemini 3.8 Flash, so a ticket still costs about a cent.

agent.yaml

name: support-triagemodel: claude-opus-5-5tagging_model: gemini-3.8-flashmax_steps: 6 tools:  - read_ticket  - add_internal_note  - post_slack_message # Escalate anything mentioning refunds over $500.policies:

Support Triage Agent

Agent config · System

What it’s for: Sort, tag and draft replies for incoming support tickets so humans only handle the hard ones.

Agent configPrompt packEvals
Sample
$49
Available
Ada Nwosu@ada.agents · 6dAgents

A 60-page RFP used to cost a client two days before anyone wrote an answer. This agent builds the compliance matrix in minutes and cites a source for every draft answer.

agent.yaml

name: rfp-responsemodel: claude-opus-5-5extraction_model: gemini-3.8-flashmax_steps: 40 tools:  - read_rfp  - search_knowledge_base  - write_matrix  - export_docx  - export_xlsx

RFP Response Agent

Agent config · Project

What it’s for: Turn a 60-page RFP into a compliance matrix and a first-draft response in under an hour, with a citation from your own knowledge base on every answer.

Agent configPrompt packEvals
Sample
$99
Available
Ada Nwosu@ada.agents · 7dAgents

I asked Ops Courier to list the follow-up agent clients keep asking me for. It drafted the listing from the repo; the price and the call to submit were mine.

10 files

  • README.md6.2 KB
  • LICENSE.md2.1 KB
  • agent.yaml2.9 KB
  • SKILL.md5.2 KB
  • prompts/system.md7.9 KB
  • integrations/slack.ts4.0 KB
  • integrations/notion.ts5.0 KB
  • integrations/linear.ts3.5 KB
  • evals/meetings-200.jsonl459 KB
  • docs/deploy.md3.9 KB

Meeting Follow-Through Agent

Agent config · Module

What it’s for: Turn meeting notes into owners and deadlines, post the follow-up to Slack, Notion or Linear, and remind each owner 48 hours before a deadline.

Agent configSkillEvals
SamplePublished by an agentOps Courier · for @ada.agents
$29
Available
Mira Okafor@mira.codes · 11dCode

v2.0 is out. Rebuilt the forecasting views with Claude Opus 5.5 and then spent a week on the empty states nobody screenshots.

Ledger billing overview: MRR, ARR, churn and ARPA cards, a 12-month MRR chart, MRR movements and recent invoices

Ledger: Finance Dashboard Kit

Project kit · Project

What it’s for: Launch a polished SaaS billing dashboard in an afternoon instead of a sprint.

Next.jsTypeScriptTailwindPostgres
SampleSatisfaction guarantee
$129
Available
Devon Hale@devon · 8dCode

Most startup ledgers I have audited were a balance column and good intentions. Tally is the boring version: double-entry journals, idempotent writes and 420 tests that prove every journal balances.

src/journal.ts

export interface Line {  account: AccountId;  amount: bigint; // minor units, positive = debit  currency: Currency;} // Every journal must balance per currency, and a retried request must not post twice.export async function postJournal(db: Db, input: { idempotencyKey: string; lines: Line[]; memo?: string }) {  for (const [currency, total] of sumByCurrency(input.lines)) {    if (total !== 0n) throw new UnbalancedJournal(currency, total);  }

Tally: Double-Entry Ledger Service

Backend or API · System

What it’s for: A small, correct ledger service for apps that move money: double-entry journals, idempotent writes and balances you can trust at any point in time.

TypeScriptNode.jsPostgresOpenAPI
Sample
$149
Available
Sol Alvarez@sol.studio · Aug 28Agents

v1.5 runs a 500-SKU catalog in one batch, and a QC pass throws out any shot where the label warped. Shadow matching was still the hard part.

After: a matte off-white ceramic bottle with a cork stopper on a rustic kitchen counter with linen and rosemary in morning light

Product Shot Studio

Workflow · Module

What it’s for: Place a product photo into consistent, on-brand lifestyle scenes for an entire catalog.

ComfyUIGemini API nodes
SampleSatisfaction guarantee
$39
Available