AI visibility tracking usually fails in one of two ways. Either you collect screenshots that nobody acts on, or you build a dashboard that tells you your brand was mentioned 34 times without telling you whether any of those mentions were good.
The gap is not data collection. It is scoring. Someone has to look at each AI answer and decide: was our brand mentioned, was a competitor mentioned instead, was there a citation, was the framing accurate, and how urgent is this. Multiply that by a few hundred prompts across four platforms, and it becomes a job nobody has time for.
That scoring job is a fixed set of decisions. It is exactly the kind of work Jev handles well. This guide builds a scoring loop that turns raw AI answers into a prioritized worklist.
What you will finish with
A scored prompt log where every answer carries five judgments:
- Mention: did the answer name the brand?
- Competitor: which competitor appeared instead, if any?
- Citation: did the answer cite a source, and was it the brand's own page?
- Framing: was the brand described accurately and in the right category?
- Priority: how urgent is a fix?
Plus a page-level worklist that maps every low score to a specific action.
Who this is for: a GEO or SEO lead already tracking a prompt set, who wants a consistent way to score answers without reading every one by hand.
Prerequisites:
- A defined prompt set (start with 50 to 200 prompts, not 1,450).
- Captured answers from your target AI surfaces, saved in a structured format.
- A page inventory so you can map fixes back to URLs.
- A Jev API key, or an agent with the TypeSafe skill installed.
Time estimate: a few hours to set up capture and scoring, then a repeatable monthly loop.
Definition of done: every captured answer has five scores and a confidence value, low-confidence rows are flagged for human review, and the top-priority gaps have named owners.
Capture first, judge second
This is the hard prerequisite, and it is where most people try to take a shortcut that does not exist.
Jev has no live web access. It cannot query ChatGPT, cannot open Perplexity, cannot check what Gemini said this morning. Everything it judges has to be in the data you send it.
So the workflow has two distinct stages:
- Capture. You collect the raw answers from each AI surface and store them.
- Score. You send the captured answers to Jev and get structured judgments back.
If you try to skip capture and ask Jev "is my brand visible in AI search," you will get a confident answer built on nothing. That is the most common failure mode in this whole workflow.
A practical capture format:
{
"prompt_id": "p_014",
"prompt_text": "best project management software for small teams",
"surface": "chatgpt",
"captured_at": "2026-09-01T09:00:00Z",
"answer_text": "For small teams, the most commonly recommended options are Asana, Trello, and Monday.com. Asana is often cited for its flexible views...",
"cited_urls": ["https://example.com/best-pm-tools"]
}Store one row per prompt per surface. If you run the same prompt monthly, that is a new row, not an overwrite. The trend across time is where the useful signal lives.
Capture and scoring are separate stages. Jev cannot fetch an AI answer for you.
Define the prompt set and the answer record
Two decisions before you score anything.
Keep the prompt set focused. A few hundred well-chosen prompts across your real buyer questions will tell you more than a thousand random ones. Group them by intent: category discovery, comparison, alternatives, pricing, and problem-specific. If you want a starting structure, build the set from the questions your sales team actually gets asked.
Freeze the answer record schema. Every captured answer needs the same fields, or your scores will not be comparable across rows. The minimum set: prompt ID, prompt text, surface, capture date, answer text, and any cited URLs.
One more thing worth deciding early: what counts as a mention. A brand name appearing in a list is different from a brand being recommended. Decide your definition now and write it into the scoring criteria, or your scores will drift between runs.
Write the scoring questions
Here is the full scoring schema. Five questions, one request per answer, all evaluated in parallel against the same state.
{
"model": "jev-latest",
"state": {
"prompt": "best project management software for small teams",
"surface": "chatgpt",
"answer": "For small teams, the most commonly recommended options are Asana, Trello, and Monday.com. Asana is often cited for its flexible views...",
"brand": "Acme PM",
"cited_urls": ["https://example.com/best-pm-tools"]
},
"questions": {
"brand_mentioned": {
"type": "noul",
"instructions": "Does this answer mention the brand by name?",
"criteria": {
"true": "The brand name appears explicitly in the answer",
"false": "The brand name does not appear anywhere in the answer"
}
},
"mention_type": {
"type": "choice",
"instructions": "If the brand appears, how is it presented?",
"criteria": {
"recommended": "The brand is actively recommended as a good option",
"listed": "The brand appears in a list without a recommendation",
"referenced": "The brand is mentioned in passing or as a comparison point",
"absent": "The brand does not appear"
}
},
"competitor_present": {
"type": "choice",
"instructions": "Which competitor is most prominently featured in this answer?",
"criteria": {
"competitor_a": "Asana is the lead recommendation",
"competitor_b": "Trello is the lead recommendation",
"competitor_c": "Monday.com is the lead recommendation",
"none": "No competitor is prominently featured"
}
},
"own_citation": {
"type": "noul",
"instructions": "Does the answer cite a URL from the brand's own domain?",
"criteria": {
"true": "At least one cited URL belongs to the brand's domain",
"false": "No cited URL belongs to the brand's domain"
}
},
"framing_accuracy": {
"type": "score",
"instructions": "How accurately does the answer describe the brand's category and positioning?",
"criteria": [
"Wrong category or factually incorrect",
"Vague or generic description",
"Broadly accurate but missing key positioning",
"Accurate category and positioning"
]
},
"fix_priority": {
"type": "score",
"instructions": "How urgent is a content fix for this prompt?",
"criteria": ["No action needed", "Low priority", "Medium priority", "High priority"]
}
}
}Six questions in one request. Because they run in parallel, adding the last four costs you almost nothing in latency compared to asking just one.
A few notes on the design.
Separate mention from recommendation. A brand appearing in a list is not the same as being recommended. The mention_type question captures that distinction, and it is usually the difference between a vanity metric and a useful one.
Ask about citations separately from mentions. An answer can name your brand without citing your page. Those are different problems with different fixes. Mention problems point to positioning and authority work. Citation problems point to content structure and extractability.
Include a priority score. Without it, every gap looks equally urgent. The priority score is what turns a log into a worklist.
Run the batch and read the distributions
Execute the scoring across your captured answers. Practical notes from running this kind of loop:
Batch by surface. Group answers from the same platform together. It makes the output easier to compare and lets you spot platform-specific patterns.
Keep the full probability spread. For every choice question, store the probabilities across all options, not just the winner. A competitor_present result where competitor A is at 0.45 and competitor B is at 0.43 means two competitors are effectively tied, which is a different strategic situation than a clean lead.
Watch for flat distributions. If large numbers of answers come back with near-uniform probabilities, your criteria are probably too vague for the model to discriminate. Rewrite the criteria before you change anything else.
Do not treat a high confidence as a quality signal. A confident score tells you the model was certain about its judgment. It does not tell you the judgment was right. You still need the verification pass below.
Turn low scores into a page-level worklist
This is the step that makes the whole loop worth running. Scoring without action is just a more elaborate dashboard.
Map each failure pattern to a specific fix:
Signal | What it means | Typical fix |
|---|---|---|
No mention, competitor present | The brand is not in the consideration set for this query | Build or strengthen a page that directly answers the prompt |
Mentioned but not recommended | The brand is visible but not persuasive | Improve proof, differentiation, and comparison content |
Mentioned, no citation | The answer knows the brand but does not source the site | Improve extractability: clear answers, structured headings, citable facts |
Wrong framing | The answer describes the brand in the wrong category | Fix entity clarity: about page, category language, consistent descriptions |
Competitor-only citation | A competitor owns the source for this question | Build a stronger page on the same question with better evidence |
Then sort by priority score and assign owners. A high priority gap on a commercially important prompt with a competitor-only citation is a this-sprint item. A low priority gap on a peripheral question goes on the backlog.
Every failure signal maps to a specific page fix. A score without an action is just a dashboard.
Reported monitoring setups have run this kind of scoring across roughly 1,450 buyer questions and three AI engines, using the model to decide where the brand was cited, who was cited instead, and which page to fix first. Treat that scale as a reported figure. The important part is the structure: capture, score, map to a page, assign an owner.
Verify a sample by hand
Before you trust the trend, verify the scoring.
- Read twenty scored answers yourself. Confirm the mention and citation flags match what you see in the text.
- Check the framing scores against your own positioning. If the model calls an inaccurate description "accurate," your criteria are too lenient.
- Re-score a small sample twice. Run the same twenty answers again. If the scores move substantially, the criteria need tightening.
- Verify the citation extraction in code. Whether a URL belongs to your domain is a code check, not a model judgment. Do not let the model decide that.
- Confirm the priority ordering matches your commercial reality. If the top-priority gaps are all on low-value prompts, your priority criteria need to include business context.
Maintain a monthly cadence
AI answers change. A single snapshot tells you almost nothing.
- Capture monthly. Same prompt set, same surfaces, new answers.
- Score every capture. Keep the history.
- Compare month over month. Look for prompts where mention dropped, where a new competitor appeared, or where framing shifted.
- Re-examine the prompt set quarterly. Add questions your buyers have started asking and retire ones that no longer matter.
- Re-validate thresholds twice a year. What counts as a high-priority gap changes as your content and your market change.
The trend line is the asset. A single month of scores is a snapshot; six months of scores tells you whether your GEO work is actually moving anything.
FAQ
Can Jev check AI answers directly? No. It has no web access. You capture the answers first, then pass them in as the state. This is a hard constraint, not a configuration issue.
How many prompts do I need? Start with 50 to 200 well-chosen prompts across your real buyer questions. Expand only after the scoring criteria are stable and you trust the output.
Is a mention the same as being recommended? No, and conflating them is the most common measurement mistake. A brand listed among ten options is a very different result from a brand actively recommended. Score them separately.
How is this different from an AI visibility tool? Tools typically tell you whether your brand appeared. This workflow tells you how it appeared, whether it was cited, whether the framing was accurate, and what to fix. The scoring layer is the part most tools skip.
Can I automate the fixes too? Sometimes, for low-risk changes like adding a clear answer block to an existing page. Anything that changes positioning, category language, or a comparison page should go through an editor.
What if my brand is never mentioned? That is a valid and useful result. A consistent pattern of absence across a prompt cluster tells you the brand is not in the consideration set for that topic, which is a content and authority problem, not a measurement problem.
Which surfaces should I track? Start with the ones your buyers actually use. Tracking four surfaces well beats tracking eight badly, and each surface needs its own capture method.
What to do next
Pick fifty prompts. Capture the answers from two surfaces. Run the six-question scoring schema. Read twenty results by hand and check them against what you see in the text.
Then take the ten highest-priority gaps and map each one to a page. That mapping is the point. The score is only useful if it ends in a specific fix with a named owner.
Author: Ethan Marlowe, GEO Measurement Lead Across 500+ Prompts at Auspia. Ethan writes about prompt tracking, citation reporting, visibility dashboards, and AI answer quality checks.
