Promptfoo Review

Overview

Promptfoo is an open source LLM evaluation toolkit, used by developers to test prompts and models and by marketing teams to track brand answers in AI search. The core is a CLI and a TypeScript library, both MIT licensed. You define prompt sets, providers and grading rules in YAML, run everything in one command, and get a comparison of how each model answered each prompt. Outputs are scored with assertions: exact match, contains, regex, similarity, and model-graded checks where one model grades another's answer. Results open in a local web viewer. For generative engine optimization, Promptfoo is not a dedicated GEO dashboard and does not pretend to be one. What teams do is write a fixed set of buyer questions, run that set across ChatGPT, Perplexity and other models on a schedule, and grade whether their brand appears in each answer and what the answer claims. That yields prompt-level brand-answer tracking with developer tooling: versioned, repeatable and diffable between runs. There is no share-of-voice index or rank chart out of the box; tracking is only as good as the prompt set and grading rules you write. The second major mode is red teaming. The CLI generates attack probes against an application and reports findings. Guardrails, an MCP proxy, model security and code scanning are separate products on the cloud side, and the hosted red team product shows a 10k probes per month limit. Everything so far runs locally at no cost. Promptfoo's cloud adds team and enterprise features, and as of September 2026 the pricing page lists no public price numbers.

AI Capabilities

• Model-graded evals where one LLM scores another's answers
• Automated red team probe generation against AI applications
• Prompt runs across ChatGPT, Perplexity and other models with side-by-side comparison

Key Integrations

Best for

Developer-led teams and technical marketers who want prompt-level brand-answer tracking, model comparisons and red teaming in one toolkit.

Not for

Marketers who want a no-code GEO dashboard with rank tracking, share-of-voice charts and scheduled reports.

Review notes

Promptfoo is a developer eval toolkit, and the workflow shows it. Prompt sets, providers and grading rules live in YAML. One command runs every prompt against every model and grades the outputs with assertions: exact match, contains, regex, similarity, and model-graded checks where a judge model scores the answer. Results open in a local web viewer where answers can be compared side by side.

The GEO use case is a repurposing rather than a packaged feature. A marketing team writes a fixed set of buyer questions, runs them across ChatGPT, Perplexity and other models on a schedule, and scores whether the brand appears in each answer and what the answer says. That is prompt-level brand-answer tracking built on the same primitives developers use for regression tests. It is not a dedicated GEO dashboard. There is no rank index or share-of-voice UI out of the box, and the tracking is only as good as the prompt set and grading rules you write. Assessed from vendor docs.

Red teaming is the other big mode. The CLI generates attack probes against an application and reports vulnerabilities. The hosted red team product shows a 10k probes per month limit, and guardrails, an MCP proxy, model security and code scanning sit as separate cloud products. The eval engine itself runs locally under the MIT license; the cloud at promptfoo.dev adds team and enterprise features with no public price numbers listed as of September 2026.

Verdict

An eval framework that can double as GEO prompt tracking for teams willing to write YAML and grading rules. Not a substitute for a visibility dashboard.

Similar Tools