How to Audit Technical SEO with Codex (Full SKILL.md Included)

technical_scan https://auspia.ai
  final status: 200  (redirects: 0)  size: 410.9kB
  security headers: Strict-Transport-Security
  - Info: no CSP header
  - Warn: missing X-Content-Type-Options
  robots.txt: http 200  sitemaps declared: ['https://auspia.ai/sitemap.xml']
  sitemap url_count: 966
  title(46): Auspia - AI-powered SEO automation growth tool
  h1 x1, h2 x9, imgs 15 (missing alt 0), scripts 57
  schema: [['Organization'], ['WebSite']]

mkdir -p ~/.codex/skills/codex-seo-technical/scripts

---
name: codex-seo-technical
description: Use when the user asks for a technical SEO check of any website or URL - crawl problems, robots.txt, XML sitemap, canonical conflicts, duplicate content, noindex tags, security headers, HTTPS, mobile questions, Core Web Vitals, structured data presence, or JavaScript rendering concerns. Triggers on "technical SEO", "crawl issues", "robots.txt check", "why is my page not indexing", "site speed", "security headers".
---
# Technical SEO Audit

Run the scanner first, then evaluate the nine categories against this
checklist. Every recommendation carries an explicit "how would we know this
failed" check, so the audit is falsifiable rather than decorative.

## Run

```bash
python3 ~/.codex/skills/codex-seo-technical/scripts/technical_scan.py <url> [--json] [--psi]
```

- Without flags: human-readable summary.
- `--json`: machine-readable full report (use this when writing your findings).
- `--psi`: also queries the free PageSpeed Insights API for real lab + field
  data (no API key needed). It rate-limits without a key (HTTP 429); report
  "live CWV unavailable, low-traffic sites are common" and continue instead of
  treating it as a failure.

## The nine categories

### 1. Crawlability

| Check | Fail means | Fix |
|-------|-----------|-----|
| robots.txt | 404 or unreachable | Create one via a CMS or static file; keep mobile/non-search user agents in mind |
| Sitemap declared + reachable | No `Sitemap:` line, or declared URL 404s | Point at a real XML sitemap; check it after every deploy |
| Noindex tags | Robots meta or X-Robots-Tag on a page you want indexed | Remove the directive on canonical pages only; keep it on thin/gated pages |
| Crawl budget | Site >10k pages, many near-duplicate URLs | Canonicalize, block parameter URLs in robots.txt, trim dead pages |

AI crawler handling (robots.txt tokens): `GPTBot` (OpenAI training),
`ChatGPT-User` (OpenAI live browsing), `ClaudeBot` (Anthropic training),
`PerplexityBot` (Perplexity search + training), `Bytespider` (ByteDance),
`Google-Extended` (Gemini training only, not Google Search), `CCBot` (Common
Crawl). Blocking `Google-Extended` does not affect Google Search indexing or AI
Overviews. User-triggered fetchers - `Google-Agent`, `Google-NotebookLM`,
`Google Messages` - ignore robots.txt by design; block those server-side.

Hard facts to keep in audits: Googlebot reads only the first **2MB of HTML**
(and 64MB of a PDF), so keep critical content and JSON-LD in the opening
payload. Crawl rate auto-adjusts; there is no manual crawl-rate setting (the
legacy Search Console control was removed in January 2024). AMP gets no
ranking advantage since 2026 and Google sends users to publisher-hosted AMP
URLs; do not recommend AMP Cache unless the site already has it.

### 2. Indexability

| Check | Fail means | Fix |
|-------|-----------|-----|
| Canonical | Missing, not self-referencing, or points at a redirected URL | Self-referencing canonical on every page |
| Canonical vs noindex conflict | Canonical exists but robots says `noindex` | Either index it or point the canonical at the indexed twin |
| Thin content | <300 words on a page you want ranking | Expand or fix the intent match; thin pages have no ranking potential |
| Duplicate content | Same content at www/non-www, http/https, trailing-slash variants | One version per URL set; 301 the variants |
| Hreflang | `hreflang` links exist but pairs are wrong for your language set | Use bidirectional pairs, one canonical each |

Do not interpret an unchanged canonical immediately after a fix as failure:
Google may keep corrected pages in a duplicate cluster for up to two weeks,
then re-evaluate.

### 3. Security

Check headers in the scanner output: `Strict-Transport-Security`,
`Content-Security-Policy`, `X-Frame-Options`, `X-Content-Type-Options`,
`Referrer-Policy`. Missing HSTS and X-Content-Type-Options are warns; CSP
missing is information (not all sites need one).

**Critical:** back-button hijacking via `history.pushState`/`replaceState` (a
Google spam-policy violation, enforcement live since 2026-06-15); third-party
ad scripts doing it still counts.

### 4. URL structure

- Redirect chains: max one hop; 301 for permanent moves (never 302).
- Long query strings on content pages: flag >100-character URLs.
- Descriptive, hyphenated slugs; consistent trailing slashes.

### 5. Mobile and page experience

- Viewport meta present; no horizontal scroll (mobile).
- Mobile-first indexing: Googlebot Smartphone is the primary crawler. A mobile
  version is strongly recommended, not required; the real risk is content and
  parity loss, not de-indexing.
- Content parity (highest-value check): same primary content, matching robots
  meta, matching titles, equivalent structured data, crawlable resources.
- Interstitials: flag full-page popovers, consent-redirect pages, persistent
  blocking dialogs; allow small banners and standard CMS/legal dialogs.
- Keep key content visible on load - not behind tabs or accordions.

Page experience is guidance, not one ranking system. Only Core Web Vitals feed
ranking directly; HTTPS is a confirmed but lightweight signal.

### 6. Core Web Vitals

| Metric | Threshold (75th percentile) |
|--------|----------------------------|
| LCP | <=2.5s |
| INP | <=200ms |
| CLS | <=0.1 |

INP replaced FID in March 2024 - never report FID. Field data comes from CrUX
via `--psi`; when the API rate-limits, say so explicitly instead of inventing
lab numbers.

### 7. Structured data

Scanner reports JSON-LD blocks and their `@type`; flag invalid JSON (it is
invisible to Google). Full analysis = the codex-seo-schema skill; here, just
note presence and validity. Prefer JSON-LD over Microdata/RDFa.

### 8. JavaScript rendering

Signals from the scan: very low word count + high script count points to a
client-side-rendered shell. If so:

1. Check if canonical/robots/title appear in the raw HTML source (curl the
   URL). December 2025 Google guidance: a canonical injected only by JS may
   conflict, and Google MAY still honor a `noindex` that raw HTML served even
   if JS removes it.
2. Google doesn't render any page with a non-200 status; JS-injected meta on
   error pages is invisible.
3. Serve critical elements (title, canonical, robots, structured data) in the
   server HTML, and note the rendering limitation instead of faking results.

### 9. IndexNow

Recommend IndexNow for Bing, Yandex, and Naver (not Google). Detect support
via an `indexnow` token/endpoint or WordPress plugins; it is a suggestion, not
a defect.

## Scoring

Rate each category pass / warn / fail (pass=100, warn=60, fail=20), average to
a 0-100 Technical Score. Bucket the findings:

- **Critical** (fix today): blocks indexing, security violations, back-button
  hijacking, non-HTTPS.
- **High** (this week): canonical conflicts, missing HSTS, mobile parity loss,
  thin primary pages.
- **Medium** (this month): redirect chains, hreflang errors, missing CSP.
- **Low** (backlog): header details, target-size polish, IndexNow setup.

Each recommendation: the evidence line (what the scanner found), the fix, and
a falsify check so the user can verify next week - e.g. "add HSTS, then
`curl -I` shows Strict-Transport-Security; if Google still duplicates the
canonical for two weeks, that is normal, not failure".

## Errors

| Scenario | Action |
|----------|--------|
| URL unreachable / timeout | Report the error and the status code; ask the user to verify DNS/public access. Never guess site content from memory |
| robots.txt 404 | Note it and recommend creating one; continue the audit |
| Non-200 final status | If 4xx/5xx, stop on-page checks and report the status; crawling that URL is wasted budget |
| PSI 429 | Note live CWV unavailable; use lab proxies or a free API key; do not block the audit |
| JSON-LD invalid | Quote the block and point at schema.org validator |
| CSR-only page | Complete what the raw HTML proves, mark the rest as needs-browser, offer the Playwright note |

## Agent-friendly pages (optional deep-dive)

Recommend, not block: semantic HTML (real `<button>`/`<a>`, no
`<div onclick>`), label associations on inputs, minimum 48px touch targets,
stable layout. Vision models, raw DOM readers, and accessibility-tree readers
are all more reliable when the page is structurally plain.

#!/usr/bin/env python3
"""Nine-category technical SEO data collector. Standard library only."""

import json
import re
import ssl
import sys
import urllib.error
import urllib.parse
import urllib.request

TIMEOUT = 15

UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
      "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 "
      "codex-seo-technical/1.0")


def fetch(url):
    """Return (status, final_url, headers, body) following redirects."""
    chain = []
    seen = set()
    cur = url
    while cur not in seen and len(chain) < 8:
        seen.add(cur)
        chain.append(cur)
        req = urllib.request.Request(cur, headers={"User-Agent": UA})
        try:
            resp = urllib.request.urlopen(req, timeout=TIMEOUT)
            headers = dict(resp.headers)
            body = resp.read().decode("utf-8", "replace")
            return (resp.status, resp.geturl(), headers, body), chain
        except urllib.error.HTTPError as e:
            headers = dict(e.headers)
            loc = headers.get("Location")
            if e.code in (301, 302, 303, 307, 308) and loc:
                cur = urllib.parse.urljoin(cur, loc) if hasattr(
                    urllib, "parse") else loc
                continue
            return (e.code, cur, headers, ""), chain
        except urllib.error.URLError as e:
            return ("ERR:" + str(e.reason), cur, {}, ""), chain
    return ("LOOP", cur, {}, ""), chain


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        resp = urllib.request.urlopen(req, timeout=TIMEOUT)
        return resp.status, resp.geturl(), dict(resp.headers), resp.read()
    except urllib.error.HTTPError as e:
        return e.code, url, dict(e.headers), b""
    except Exception as e:
        return "ERR", url, {}, b""


def psi(url):
    """Try the free PageSpeed Insights API (no key). Returns CWV dict or None."""
    api = ("https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url="
           + urllib.parse.quote(url) + "&strategy=mobile")
    try:
        req = urllib.request.Request(api, headers={"User-Agent": UA})
        data = json.load(urllib.request.urlopen(req, timeout=45))
        lm = data.get("lighthouseResult", {}).get("audits", {})
        return {
            "LCP": lm.get("largest-contentful-paint", {}).get("displayValue"),
            "INP": lm.get("interaction-to-next-paint", {}).get("displayValue"),
            "CLS": lm.get("cumulative-layout-shift", {}).get("displayValue"),
            "TTFB": lm.get("server-response-time", {}).get("displayValue"),
            "crUX": "field data present" if data.get("loadingExperience") else
                    "no field data (low-traffic sites are common)",
        }
    except Exception as e:
        return {"crUX": "PSI API unavailable: " + str(e)}


def main():
    url = sys.argv[1] if len(sys.argv) > 1 else ""
    if not url:
        print("usage: python3 technical_scan.py <url> [--json] [--psi]")
        sys.exit(1)
    want_json = "--json" in sys.argv
    want_psi = "--psi" in sys.argv

    if not url.startswith("http"):
        url = "https://" + url

    (status, final, headers, html), chain = fetch(url)
    r = {"url": url, "status": status, "final_url": final,
         "redirect_chain": chain,
         "size_kb": round(len(html.encode()) / 1024, 1)}
    if not html:
        r["error"] = "no HTML body returned; status=" + str(status)
        print(json.dumps(r, indent=2) if want_json else r["error"])
        sys.exit(0 if status == 200 else 1)

    # check 1-3: HTTP redirect/HTTPS/security headers
    r["https_ok"] = final.startswith("https://")
    sec = {}
    for h in ["Strict-Transport-Security", "X-Frame-Options",
              "Content-Security-Policy", "X-Content-Type-Options",
              "Referrer-Policy"]:
        for k, v in headers.items():
            if k.lower() == h.lower():
                sec[h] = v[:120]
    r["security_headers"] = sec
    r["header_review"] = []
    if not r["https_ok"]:
        r["header_review"].append("Critical: page is not served over HTTPS")
    if "Strict-Transport-Security" not in sec:
        r["header_review"].append("Warn: missing HSTS header")
    if "Content-Security-Policy" not in sec:
        r["header_review"].append("Info: no CSP header")
    if "X-Content-Type-Options" not in sec:
        r["header_review"].append("Warn: missing X-Content-Type-Options")
    if len(chain) > 3:
        r["header_review"].append(
            "Warn: redirect chain is %d hops (keep under 3)" % len(chain))

    # check 4: robots.txt + AI crawler rules
    robots_url = "https://" + final.split("://", 1)[1].split("/", 1)[0] + "/robots.txt"
    sc, _, _, rb = get(robots_url)
    body = rb.decode("utf-8", "replace") if isinstance(rb, bytes) else ""
    r["robots"] = {"status": sc, "sitemap_declared":
                   [m for m in re.findall(r"(?i)^\s*Sitemap:\s*(\S+)", body)]}
    ai = {}
    for token in ["GPTBot", "ChatGPT-User", "ClaudeBot", "Claude-User",
                  "PerplexityBot", "Bytespider", "Google-Extended", "CCBot"]:
        m = re.search(r"(?im)^User-agent:\s*" + token + r"\s*$", body)
        if m:
            nxt = body[m.end():].splitlines()
            rules = [ln for ln in nxt[:12] if re.match(r"(?i)\s*(allow|disallow)", ln)]
            if rules and "disallow" in " ".join(rules).lower():
                ai[token] = "blocked in robots.txt"
            else:
                ai[token] = "mentioned, not blocked"
        else:
            ai[token] = "not mentioned (allowed by default)"
    r["ai_crawlers"] = ai
    if not body.strip():
        r["robots"]["note"] = "no robots.txt found (or non-200) - create one"

    # check 5: sitemap discovery
    sitemaps = r["robots"].get("sitemap_declared", [])
    if not sitemaps:
        root = "https://" + final.split("://", 1)[1].split("/", 1)[0]
        for guess in ["/sitemap.xml", "/sitemap_index.xml"]:
            gs, _, _, gb = get(root + guess)
            if gs == 200 and b"<urlset" in gb or b"<sitemapindex" in gb:
                sitemaps.append(root + guess)
                break
    locs = 0
    if sitemaps:
        _, _, _, sb = get(sitemaps[0])
        txt = sb.decode("utf-8", "replace") if isinstance(sb, bytes) else ""
        locs = len(re.findall(r"<loc>", txt))
        if "<sitemapindex" in txt:
            locs = "nested index (%d child sitemap entries)" % locs
    r["sitemap"] = {"sources": sitemaps, "url_count": locs}

    # checks 6-8: on-page HTML elements
    def m(p):
        return re.findall(p, html, re.I)

    title = m(r"<title[^>]*>([^<]+)</title>")
    desc = m(r'<meta[^>]+name=["\']description["\'][^>]+content=["\']([^"\']+)["\']')
    canon = m(r'<link[^>]+rel=["\']canonical["\'][^>]*>')
    robots_meta = m(r'<meta[^>]+name=["\']robots["\']')
    h1 = m(r"<h1[^>]*>")
    h2 = m(r"<h2[^>]*>")
    imgs = len(m(r"<img\b"))
    imgs_noalt = len(m(r"<img\b(?![^>]*alt=)[^>]*>"))
    scripts = len(m(r"<script\b"))
    jsonld = m(r'<script[^>]*type=["\']application/ld\+json["\'][^>]*>(.*?)</script>')
    links = len(m(r"<a\s[^>]*href="))
    lang = m(r'<html[^>]*\blang=["\']([^"\']+)["\']')
    hreflang = len(m(r'<link[^>]+hreflang=["\']'))
    text = re.sub(r"(?is)<script.*?</script>|<style.*?</style>|<[^>]+>", " ", html)
    words = len(text.split())
    page_ok = len(html) < 2_000_000

    r["onpage"] = {
        "title": title[0][:100] if title else None,
        "title_len": len(title[0]) if title else 0,
        "meta_description": (desc[0][:100] if desc else None),
        "canonical": canonical_href(canon[0]) if canon else None,
        "robots_meta": robots_meta[:3] if robots_meta else [],
        "h1_count": len(h1), "h2_count": len(h2),
        "images": imgs, "images_missing_alt": imgs_noalt,
        "scripts": scripts, "internal_links": links,
        "html_lang": lang[0] if lang else None,
        "hreflang_links": hreflang,
        "text_words": words,
        "under_2mb_html": page_ok,
    }

    # check 7: JSON-LD structure data
    ld = []
    for js in jsonld:
        payload = js[js.find("{"):]
        if not payload:
            ld.append({"type": "empty block", "ok": False})
            continue
        try:
            d = json.loads(payload)
            types = d.get("@type") if isinstance(d, dict) else None
            if isinstance(types, str):
                types = [types]
            ld.append({"type": types or ["(no @type)"], "ok": True})
        except Exception:
            ld.append({"type": "invalid JSON", "ok": False})
            r["schema_warning"] = "JSON-LD block did not parse; validate it on schema.org validator"
    r["structured_data"] = {"blocks": len(jsonld), "types": [x["type"] for x in ld if x.get("ok")],
                            "invalid": [x for x in ld if not x.get("ok")]}

    if want_psi:
        r["cwv"] = psi(final)
    else:
        r["cwv"] = {"note": "run with --psi or install the codex-seo-google data skill"}
    print(json.dumps(r, indent=2) if want_json else human(r))


def canonical_href(tag):
    m = re.search(r'href=["\']([^"\']+)["\']', tag)
    return m.group(1) if m else tag.strip()[:100]


def human(r):
    lines = ["technical_scan %s" % r["url"],
             "  final status: %s  (redirects: %d)  size: %skB" %
             (r["status"], len(r["redirect_chain"]) - 1, r["size_kb"]),
             "  security headers: %s" % (", ".join(sorted(r["security_headers"])) or "none")]
    for c in r["header_review"]:
        lines.append("  - " + c)
    if r.get("robots"):
        lines.append("  robots.txt: http %s  sitemaps declared: %s" %
                     (r["robots"]["status"], r["robots"].get("sitemap_declared", [])))
    lines.append("  sitemap url_count: %s" % r["sitemap"]["url_count"])
    o = r["onpage"]
    lines.append("  title(%d): %s" % (o["title_len"], o["title"]) if o["title"] else "  no title")
    lines.append("  h1 x%d, h2 x%d, imgs %d (missing alt %d), scripts %d" %
                 (o["h1_count"], o["h2_count"], o["images"], o["images_missing_alt"], o["scripts"]))
    lines.append("  schema: %s" % (r["structured_data"]["types"] or "none"))
    return "\n".join(lines)


if __name__ == "__main__":
    main()

python3 ~/.codex/skills/codex-seo-technical/scripts/technical_scan.py https://your-site.com --json