How to Set Up Codex for GEO: AI Visibility Scans (Full SKILL.md Included)

---
name: codex-seo-geo
description: Use when the user asks about AI Overviews, SGE, GEO, AI search, LLM optimization, Perplexity citations, ChatGPT search citations, AI visibility, llms.txt, AI crawler robots.txt rules, or Generative Engine Optimization for a page or domain. Works with any site; python3 only.
---
# GEO Readiness Scan & Optimization

Optimize content for AI search surfaces: Google AI Overviews, AI Mode,
ChatGPT web search, Perplexity, Bing Copilot. Frame findings as SEO
fundamentals applied to AI-search surfaces, not a separate discipline.
When community recommendations contradict Google's primary source, defer to
Google and note the contradiction.

## Primary Source

Google's AI optimization guide (Search Central, May 2026): "Optimizing for
generative AI search is still SEO from Google's perspective. AEO and GEO are
rebranded labels for the same work." Also explicitly: `llms.txt` and other
AI-text files are not needed for Google Search and do not help or hurt
visibility or rankings. Never recommend `llms.txt` as a Google ranking or
citation lever; it may still serve non-Google systems.

## Run

```bash
python3 ~/.codex/skills/codex-seo-geo/scripts/geo_scan.py <url> [--json]
```

The scanner measures five signal groups and reports a 0-100 GEO Readiness
score plus the evidence behind every deduction and a prioritized fix list.
It also fetches robots.txt (AI crawler allow/block), `/llms.txt` presence,
and follows 307/308 redirects manually (Python < 3.11 urllib does not).

## The five signals

Weights come from the adapted claude-seo framework; the underlying
thresholds track Google's official guidance and published studies.

### 1. Citability (25%)

- **Optimal passage length: 134-167 words** for AI citation; self-contained
  blocks that extract without context.
- ~44% of AI citations come from the first 30% of a page (SE Ranking);
  front-load your best answer block.
- Direct answer in the first 40-60 words of a section; a definition in a
  "X is..." pattern near the top.
- Claims attributed with specific sources; unique data points.

### 2. Structure (20%)

- Clean H1->H2->H3 hierarchy; one H1.
- Question-shaped H2s match how AI answers queries.
- Short paragraphs (2-4 sentences), tables for comparative data, lists for
  multi-item content, FAQ section.

### 3. Multimodal (15%)

- Text + relevant images; embedded video; infographics;
  structured data supporting media (Article/Organization/Person).

### 4. Authority & brand (20%)

- Author byline with credentials; publication date AND last-updated date.
- Recency: content under 3 months old is ~3x more likely to be cited;
  6+ months stale loses citation eligibility (SE Ranking, 1.3M-citation
  study). A scheduled refresh program is a top GEO play.
- Citations to primary sources; entity presence on Wikipedia, Wikidata.
- Brand mentions correlate ~3x more strongly with AI visibility than
  backlinks (Ahrefs, 75,000 brands): YouTube ~0.737, Wikipedia/Reddit high,
  Domain Rating ~0.266.

### 5. Technical accessibility (20%)

- AI crawlers do NOT execute JavaScript: server-side rendering matters.
- robots.txt allows key AI crawlers (see table).
- `/llms.txt` present (record only; no citation weight).

## AI crawler table

| Crawler | Owner | Obeys robots.txt? | Recommend |
|---------|-------|-------------------|-----------|
| GPTBot / OAI-SearchBot | OpenAI | yes | allow (ChatGPT search) |
| ClaudeBot | Anthropic | yes | allow |
| PerplexityBot | Perplexity | yes | allow |
| Google-Extended | Google | yes | allow (or block to opt out of Gemini training) |
| Google-CloudVertexBot | Google | yes | allow |
| anthropic-ai | Anthropic | yes | optional (training) |
| CCBot | Common Crawl | yes | block if training out |
| Bytespider | ByteDance | yes | block if training out |
| cohere-ai | Cohere | yes | optional |
| ChatGPT-User | OpenAI | **no (user-triggered)** | cannot be blocked by robots.txt |
| Google-Agent | Google | **no (user-triggered)** | cannot be blocked by robots.txt |
| Google-NotebookLM | Google | **no (user-triggered)** | cannot be blocked by robots.txt |

User-triggered fetchers ignore robots.txt by design; server-side access
controls are the only protection. There is no AI-feature opt-out file;
AI Overviews / AI Mode appearance is governed by standard preview and
indexing directives (nosnippet, data-nosnippet, max-snippet, noindex).

## Platform differences

| Platform | Citation sources | Focus |
|----------|------------------|-------|
| Google AI Overviews | Strongly ranking-correlated | Traditional SEO + passages |
| Google AI Mode | Weakly ranking-correlated, broader pool (~9 domains/query) | Freshness, entity authority, citable passages beyond position 5 |
| ChatGPT | Wikipedia (47.9%), Reddit (11.3%) | Entity presence, authoritative sources |
| Perplexity | Reddit (46.7%), Wikipedia | Community validation |
| Bing Copilot | Bing index | Bing SEO, IndexNow |

Only 11% of domains are cited by both ChatGPT and Google AI Overviews for
the same query - optimize per platform. AI Overviews and AI Mode cite the
same URLs only 13.7% of the time (Ahrefs, 540k query pairs); score both.

## Scoring rules

| Signal | Score | Deducted when |
|--------|-------|---------------|
| Citability | 25 | no 134-167w block (-15/8), none in first 30% (-5), no definition (-5), avg paragraph >120w (-2) |
| Structure | 20 | no single H1 or no H2 (-5), no question H2 (-5), no tables/lists (-5), no FAQ with 10+ paragraphs (-5) |
| Multimodal | 15 | 0 images (-5), <3 images (-2), no video (-4), no JSON-LD (-3) |
| Authority | 20 | no byline (-6), no published date (-6), no modified (-2), <2 source references (-4) |
| Technical | 20 | no SSR content (<250 raw words, -10), blocked AI crawlers (-5) |

## Output contract

For analysis: a report with the score, the per-signal breakdown with
evidence, platform-specific notes, and the 5 highest-impact changes sorted
by effort. For a page the user owns, add a scheduled refresh date for
content older than 3 months.

## Priority ladder

Quick wins: "X is..." definition in first 60 words; one 134-167 word
answer block in the first 30%; question-shaped H2s; specific statistics with
sources; publication + update dates; Person schema; allow GPTBot,
OAI-SearchBot, ClaudeBot, PerplexityBot.

Medium: author bio with credentials; SSR for key content; brand presence on
Reddit/YouTube; comparison tables with data; FAQ sections.

High impact: original research or surveys; Wikipedia presence; YouTube
mentions; comprehensive sameAs entity linking; tools or calculators.

## Errors

| Scenario | Action |
|----------|--------|
| URL unreachable | Report status code; verify URL; do not guess content |
| 307/308 redirect | Followed automatically (up to 4 hops); report final URL |
| JS-rendered shell | Report the SSR failure; AI crawlers will see the shell too |
| Blocked AI crawlers | List blocked+allowed; give exact robots.txt directives to add |
| No llms.txt | Note absence; Google ignores it, template offered for other systems |
| No structured data | Recommend Article, Organization, Person JSON-LD |

#!/usr/bin/env python3
"""GEO readiness scanner: measures the signals that drive AI citations -
structure, passage citability, authority, multimodal content and technical
accessibility - and reports a 0-100 score with the evidence behind it.
Standard library only. Usage:
  python3 geo_scan.py <url> [--json]
"""
import json
import re
import sys
import urllib.error
import urllib.parse
import urllib.request

TIMEOUT = 15
UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
      "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 "
      "codex-seo-geo/1.0")
AI_CRAWLERS = [
    "GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot",
    "Google-Extended", "CCBot", "anthropic-ai", "Bytespider",
    "cohere-ai", "Google-CloudVertexBot",
]
DANGER = ("Google-Agent", "Google-NotebookLM", "Google-Agent",
          "Google Messages", "ChatGPT-User")


def fetch(url, timeout=TIMEOUT, hops=0):
    if not url.startswith("http"):
        url = "https://" + url
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        resp = urllib.request.urlopen(req, timeout=timeout)
        return resp.status, resp.read(), resp.geturl()
    except urllib.error.HTTPError as e:
        # Python < 3.11 urllib does not follow 308; handle it manually
        loc = e.headers.get("Location") if e.code in (307, 308) else None
        if loc and hops < 4:
            return fetch(urllib.parse.urljoin(url, loc), timeout, hops + 1)
        return e.code, b"", url
    except Exception as e:
        return "ERR:" + str(e)[:80], b"", url


def clean_text(html):
    if isinstance(html, bytes):
        html = html.decode("utf-8", "replace")
    html = re.sub(r"<(script|style|noscript)[^>]*>.*?</\1>", " ", html,
                  flags=re.S | re.I)
    html = re.sub(r"", " ", html, flags=re.S)
    html = re.sub(r"<[^>]+>", " ", html)
    html = re.sub(r"&[a-z]+;|&#\d+;", " ", html)
    return re.sub(r"\s+", " ", html).strip()


def paragraphs(html):
    paras = re.findall(r"<p[^>]*>(.*?)</p>", html, re.S | re.I)
    out = []
    for p in paras:
        t = clean_text(p)
        if t:
            out.append(t)
    return out


def words(t):
    return len(t.split())


def main():
    url = sys.argv[1] if len(sys.argv) > 1 else ""
    if not url:
        print("usage: python3 geo_scan.py <url> [--json]")
        sys.exit(1)
    want_json = "--json" in sys.argv
    status, html, final = fetch(url)
    html = html.decode("utf-8", "replace")
    result = {"url": url, "status": status, "final_url": final}

    if status != 200:
        result["error"] = "fetch failed"
        print(json.dumps(result, default=str) if want_json else
              "GEO scan error: %s" % status)
        sys.exit(1)

    host = re.match(r"^(https?://[^/]+)", final)
    base = host.group(1) if host else final

    # --- technical accessibility (20) ---
    body = clean_text(html)
    total_words = words(body)
    # SSR check: real content in the raw HTML, not a JS shell
    n_h2 = len(re.findall(r"<h2[^>]*>", html, re.I))
    has_ssr = total_words >= 250
    # robots.txt AI crawlers
    r_status, r_body, _ = fetch(base + "/robots.txt")
    crawlers = {"allowed": [], "blocked": [], "absent": [], "user": []}
    if r_status == 200 and r_body:
        rtxt = r_body.decode("utf-8", "replace")
        for c in AI_CRAWLERS:
            dis = re.search(r"^User-agent:\s*%s\s*$.*?(?:^User-agent:|\Z)"
                            % re.escape(c), rtxt, re.S | re.I | re.M)
            if dis and "Disallow: /" in dis.group(0):
                crawlers["blocked"].append(c)
            elif dis:
                crawlers["allowed"].append(c)
            else:
                crawlers["absent"].append(c)
    lm_status, lm_body, _ = fetch(base + "/llms.txt")
    has_llms = lm_status == 200 and len(lm_body) > 20

    tech = {"has_ssr": has_ssr, "ssr_words": total_words,
            "crawlers": crawlers, "llms_txt": has_llms}
    tech_score = 20
    if not has_ssr:
        tech_score -= 10
    blocked = crawlers["blocked"]
    if blocked:
        tech_score -= 5
    result["technical"] = {"score": tech_score, "max": 20, **tech}

    # --- structural readability (20) ---
    h1 = len(re.findall(r"<h1[^>]*>", html, re.I))
    h2 = n_h2
    h3 = len(re.findall(r"<h3[^>]*>", html, re.I))
    q_h2 = 0
    for t in re.findall(r"<h2[^>]*>(.*?)</h2>", html, re.S | re.I):
        c = clean_text(t)
        if c and (c.endswith("?") or re.match(r"^(How|What|Why|When|Where|Who|Can|Is|Are)\b", c)):
            q_h2 += 1
    paras = paragraphs(html)
    n_tables = len(re.findall(r"<table[^>]*>", html, re.I))
    n_lists = len(re.findall(r"<ul[^>]*>|<ol[^>]*>", html, re.I))
    has_faq = bool(re.search(r"<h2[^>]*>[^<]*faq", html, re.I))
    struct = {"h1": h1, "h2": h2, "h3": h3, "question_h2": q_h2,
              "paragraphs": len(paras), "tables": n_tables,
              "lists": n_lists, "faq": has_faq}
    struct_score = 20
    if h1 != 1 or not h2:
        struct_score -= 5
    if q_h2 == 0 and h2 >= 6:
        struct_score -= 5
    if not (n_tables or n_lists):
        struct_score -= 5
    if not has_faq and len(paras) >= 10:
        struct_score -= 5
    result["structure"] = {"score": struct_score, "max": 20, **struct}

    # --- citability (25) ---
    # a self-contained answer block in the target band (134-167 words);
    # ~44% of AI citations come from the first 30% of a page
    total_chars = len(body)
    threshold = int(total_chars * 0.30)
    first_30 = []
    remaining = threshold
    for p in paras:
        if remaining <= 0:
            break
        first_30.append(p)
        remaining -= len(p) + 1
    bands = [[134, 167, "in-band"], [100, 133, "close"], [90, 300, "usable"]]
    in_band = sum(1 for p in paras if 134 <= words(p) <= 167)
    in_first = sum(1 for p in first_30 if 134 <= words(p) <= 167)
    define = 0
    for p in paras[:3]:
        if re.match(r"^(The |A |An )?[A-Z][a-zA-Z0-9' -]+ is (a|an|the|a way|a process|the process|software|tool|platform)\b", p):
            define += 1
    avg_para = round(sum(map(words, paras)) / len(paras)) if paras else 0
    cit = {"in_band_blocks": in_band, "in_first_30pct": in_first,
           "definition_head": define, "avg_para_words": avg_para,
           "front_answer_note": None}
    cit_score = 25
    if in_band >= 3:
        pass
    elif in_band >= 1:
        cit_score -= 8
    else:
        cit_score -= 15
    if in_first == 0 and in_band == 0:
        cit_score -= 5
    if define == 0:
        cit_score -= 5
    if avg_para > 120:
        cit_score -= 2
    result["citability"] = {"score": max(cit_score, 0), "max": 25, **cit}

    # --- authority (20) ---
    author = re.search(r'<meta[^>]+name=["\']author["\'][^>]+content=["\']([^"\']+)',
                       html, re.I)
    byline = re.search(r'(?:by\s+[A-Z][a-z]+\s+[A-Z][a-z]+)', body)
    pub = re.search(r'<meta[^>]+(?:property|name)=["\']'
                    r'(?:article:published_time|datePublished|date)["\'][^>]+'
                    r'content=["\']([^"\']+)', html, re.I)
    mod = re.search(r'<meta[^>]+(?:property|name)=["\']'
                    r'(?:article:modified_time|dateModified)["\'][^>]+'
                    r'content=["\']([^"\']+)', html, re.I)
    quote_sources = body.count("according to") + body.count("source:") + \
        body.count("study") + body.count("statistics")
    auth = {"author_meta": author.group(1) if author else None,
            "byline_found": bool(byline),
            "published": pub.group(1) if pub else None,
            "modified": mod.group(1) if mod else None,
            "source_hits": quote_sources}
    auth_score = 20
    if not (author and author.group(1)) and not byline:
        auth_score -= 6
    if not (pub and pub.group(1)):
        auth_score -= 6
    elif not (mod and mod.group(1)):
        auth_score -= 2
    if quote_sources < 2:
        auth_score -= 4
    result["authority"] = {"score": max(auth_score, 0), "max": 20, **auth}

    # --- multimodal (15) ---
    n_img = len(re.findall(r"<img[^>]*>", html, re.I))
    n_vid = len(re.findall(r"<video[^>]*>|youtube\.com|youtu\.be", html, re.I))
    n_schema = len(re.findall(r'application/ld\+json', html, re.I))
    multi = {"images": n_img, "video_embeds": n_vid, "schema_blocks": n_schema}
    multi_score = 15
    if n_img == 0:
        multi_score -= 5
    if n_img < 3:
        multi_score -= 2
    if n_vid == 0:
        multi_score -= 4
    if n_schema == 0:
        multi_score -= 3
    result["multimodal"] = {"score": max(multi_score, 0), "max": 15, **multi}

    total = (result["technical"]["score"] + result["structure"]["score"] +
             result["citability"]["score"] + result["authority"]["score"] +
             result["multimodal"]["score"])
    result["total"] = total

    if want_json:
        print(json.dumps(result, default=str, indent=2))
        return

    print("GEO readiness %s" % final)
    print("  total: %d/100" % total)
    print("  technical: %d/20  SSR:%s words=%d llms.txt:%s blocked=%s" % (
        result["technical"]["score"], has_ssr, total_words, has_llms, blocked))
    print("  structure: %d/20  h1=%d h2=%d h3=%d question-h2=%d tables=%d lists=%d faq=%s" % (
        result["structure"]["score"], h1, h2, h3, q_h2, n_tables, n_lists, has_faq))
    print("  citability: %d/25  134-167w blocks=%d in-first-30%%=%d definition=%d avg-para=%dw" % (
        result["citability"]["score"], in_band, in_first, define, avg_para))
    print("  authority: %d/20  author=%s published=%s modified=%s source-hits=%d" % (
        result["authority"]["score"],
        bool(author and author.group(1)), bool(pub and pub.group(1)),
        bool(mod and mod.group(1)), quote_sources))
    print("  multimodal: %d/15  images=%d video=%d schema=%d" % (
        result["multimodal"]["score"], n_img, n_vid, n_schema))
    print("  top fixes:")
    if not has_ssr:
        print("    - content may be JS-rendered; AI crawlers do not execute JS")
    if blocked:
        print("    - allow AI crawlers in robots.txt: %s" % ", ".join(blocked))
    if in_band == 0:
        print("    - add self-contained answer blocks of 134-167 words")
    if in_first == 0:
        print("    - move your best answer block into the first 30% of the page")
    if q_h2 == 0 and h2 >= 6:
        print("    - use question-shaped H2s to match how AI answers queries")
    if not (pub and pub.group(1)):
        print("    - add publication date (and ideally last modified)")
    if n_schema == 0:
        print("    - add Article/Organization/Person structured data")
    if not has_llms:
        print("    - llms.txt missing (optional: ignored by Google Search, may help other systems)")


if __name__ == "__main__":
    main()

mkdir -p ~/.codex/skills/codex-seo-geo/scripts
# save the two files above at:
#   ~/.codex/skills/codex-seo-geo/SKILL.md
#   ~/.codex/skills/codex-seo-geo/scripts/geo_scan.py
python3 ~/.codex/skills/codex-seo-geo/scripts/geo_scan.py https://example.com

GEO readiness https://auspia.ai/blog/codex-seo-sitemap
  total: 71/100
  technical: 20/20  SSR:True words=4880 llms.txt:True blocked=[]
  structure: 20/20  h1=1 h2=30 h3=0 question-h2=10 tables=4 lists=13 faq=True
  citability: 0/25  134-167w blocks=0 in-first-30%=0 definition=0 avg-para=30w
  authority: 16/20  author=False published=True modified=True source-hits=1
  multimodal: 15/15  images=16 video=1 schema=1
  top fixes:
    - add self-contained answer blocks of 134-167 words
    - move your best answer block into the first 30% of the page

GEO readiness https://felo.ai/search
  total: 37/100
  technical: 10/20  SSR:False words=43 llms.txt:False blocked=[]
  structure: 10/20  h1=1 h2=0 h3=0 question-h2=0 tables=0 lists=0 faq=False
  citability: 0/25  134-167w blocks=0 in-first-30%=0 definition=0 avg-para=18w
  authority: 4/20  author=False published=False modified=False source-hits=0
  multimodal: 13/15  images=2 video=3 schema=1
  top fixes:
    - content may be JS-rendered; AI crawlers do not execute JS
    - add self-contained answer blocks of 134-167 words

Read the two code blocks in the article above. Create
~/.codex/skills/codex-seo-geo/SKILL.md and
~/.codex/skills/codex-seo-geo/scripts/geo_scan.py from them exactly.
Then run python3 ~/.codex/skills/codex-seo-geo/scripts/geo_scan.py
https://example.com and report the output, plus what the three biggest
deductions mean for that site.