How to Set Up Codex for Programmatic SEO (Full SKILL.md Included)

---
name: codex-seo-programmatic
description: Use when the user asks about programmatic SEO, pages generated at scale from a data source, template pages, dynamic or data-driven pages, thin content at scale, scaled content abuse risk, index bloat, faceted navigation, auto-generated internal links, or a sitemap filled with many similar URLs. Audits a sitemap for repeating URL patterns and measures uniqueness, word counts, canonicals, and link density against the quality gates.
---
# Programmatic SEO: Pages at Scale

Programmatic SEO builds pages from a data source (a CSV, an API, a database)
through a template. The quality problem is not volume. One thousand pages
multiply whatever the template does - they multiply its weaknesses faster
than its strengths, and they multiply your risk if the template is thin.

## Core insight

A "mad-libs" template is invisible one page at a time. Swap the city name in
an identical 400-word paragraph and no single page looks wrong. Publish it
200 times and Google sees a scaled content abuse pattern, and the site - not
just those pages - starts losing trust. Every programmatic rollout has two
mechanical questions that a scanner can answer and one that only judgment
can:

1. Is each template family unique enough to exist on its own?
2. Are the URLs, canonicals and sitemap entries behaving themselves?
3. Does the data source actually differentiate the pages? (judgment)

Mechanics are the script's job. Judgment is yours.

## Commands

```
python3 pp_scan.py <sitemap-url-or-local-file> [--pattern /tools/]
                                                [--sample 8] [--json]
```

- `pp_scan.py` groups sitemap URLs into pattern families (e.g.
  `/tools/<slug>`), samples pages per family, and measures: unique content
  share (top-1 Jaccard), body word counts, canonical self-reference,
  noindex directives, internal link density, and URL rule violations.
- `--pattern` limits the scan to URLs containing that substring. Use it when
  you already know which pattern family you care about.
- `--json` prints machine-readable output for larger scripts.
- Local sitemap files work too: `python3 pp_scan.py sitemap.xml`, which
  makes the scan fully replayable.

## What the scanner measures

| Signal | How | Gate |
|--------|-----|------|
| Unique content | Top-1 Jaccard similarity between sampled pages of one pattern | PASS ≥40%; FLAG <40%; HARD STOP <30% |
| Body depth | Word count of visible body text (headers/nav/footer excluded) | FLAG <300 words |
| Canonical | `rel=canonical` present and self-referencing | Report % |
| Indexing | `noindex` inside sitemap-scoped pages | Report count |
| Internal links | Links to same host per page | Score vs 3-5/1000 words |
| URL rules | Query strings, >100 chars, uppercase, duplicate slugs | No-violation target |

The scanner excludes `<header>`, `<nav>`, `<footer>`, `<aside>`, `<script>`,
`<style>` from body text and link counting, so shared boilerplate does not
feed the similarity numbers.

## What the scanner cannot see (judgment tasks)

| Not observable from outside | Why |
|------------------------------|-----|
| Data source row quality | Rows, uniqueness, missing values live behind the CMS |
| Data freshness | Stale input produces stale pages |
| Review pipeline | Whether someone reads pages before publish |
| Standalone value | "Worth publishing if no sibling existed" |
| Locale duplication intent | hreflang mirrors vs duplicate content - use `codex-seo-hreflang` |

The output prints these as `n/a` on purpose: a score for a number you did
not measure is a lie.

## Quality gates (scaled content abuse, 2024-2026)

Google's Scaled Content Abuse policy came into force March 2024 and
enforcement escalated through 2025 (manual actions in June 2025, stronger
SpamBrain detection reported from August 2025). Google reported a 45%
reduction in low-quality, unoriginal content in its results after the
policy began. The gates below come from that context and from the
open-source methodology this skill adapts:

| Metric | Threshold | Action |
|--------|-----------|--------|
| Pages without content review | 100+ | WARNING: audit before relying on scale |
| Pages without justification | 500+ | HARD STOP: explicit approval required |
| Unique content per page | <40% | FLAG as thin content (penalty risk) |
| Word count per page | <300 | FLAG for review (may lack value) |

Hardened values for high-volume sites: treat <30% unique content as a HARD
STOP, keep 5-10% human review of generated pages, and publish in batches of
50-100 pages with 2-4 weeks of indexing/ranking observation before scaling
further.

### Safe patterns (work at scale)

- Integration pages with real setup docs, API details, screenshots
- Template/tool pages with downloadable content and usage instructions
- Glossary pages: 200+ word definitions with examples and related terms
- Product pages with unique specs, reviews, comparison data
- Data-driven pages with unique per-record statistics and analysis

### Penalty-risk patterns (avoid at scale)

- Location pages that only swap the city name in identical text
- "Best tool for [industry]" without industry-specific value
- "[Competitor] alternative" without real comparison data
- AI-generated pages without human review and unique value-add
- Pages where >60% of content is shared template boilerplate

## Uniqueness calculation

```
unique content % = (words unique to this page) / (total words on page) × 100
```

Measured against all other pages in the same template family. Shared
headers, footers and navigation are excluded. Template boilerplate text is
INCLUDED: identical paragraphs count against you even when they look
harmless.

## URL rules

- Lowercase, hyphenated slugs derived from the data
- No duplicate slugs; enforce uniqueness at generation time
- Keep URLs under 100 characters
- No query parameters for primary content URLs
- Consistent trailing slash usage (match the site's existing pattern)
- Every programmatic page carries a self-referencing canonical

## Internal linking

- Hub/spoke: category hub pages link to every page in their family
- Related items: auto-link 3-5 pages sharing a data attribute
- Breadcrumbs: BreadcrumbList schema from the URL hierarchy
- Anchor text: descriptive and varied, never the same exact-match string
- Density: 3-5 internal links per 1000 words

## Sitemap integration

- Split at 50,000 URLs or 50MB uncompressed per file, whichever comes first
- Use a sitemap index when the set needs multiple files
- `<lastmod>` reflects the real data update time, never generation time
- Exclude noindexed pages; register the sitemap in robots.txt
- Regenerate as records are added to the data source

## Index bloat prevention

- Noindex low-value pages instead of keeping them indexed
- Canonicalize true duplicates and low-value filtered views to their base
- For >10k programmatic URLs, monitor crawl stats in Search Console
- Merge records with insufficient data into aggregated pages instead of
  one page each
- Monthly check: indexed count vs intended count

## Scoring

The script scores five measurable categories, each 0-100:

| Category | How the score is computed |
|----------|---------------------------|
| URL Structure | 100 minus 30×query-fraction, 30×long-fraction, 20×uppercase, 20 if duplicate slugs |
| Template Uniqueness | = average top-1 unique % (0-100) |
| Thin Content Risk | 100 minus 35 per sampled-page share under 300 words, minus 45 if unique <40%, minus 60 if <30% |
| Internal Linking | min(100, actual links ÷ (words/1000 × 4) × 100) |
| Index Management | 100 minus 30 per non-self-canonical share, 30 per noindex share, 30 per lastmod-missing share |

Total = mean of the measured categories. Data Quality is always `n/a` -
the data source is not observable from the public web, and no score is
printed for it.

## Output contract

1. Pattern inventory: every URL family with 5+ pages, counts sorted.
2. Per-pattern measurements with gate notes (PASS / FLAG / HARD STOP /
   WARNING / NOTE).
3. URL rule check over the full set.
4. Six-category scorecard with TOTAL.
5. Recommendations, ranked CRITICAL / HIGH / MEDIUM / LOW, each with the
   evidence line it came from.

## Error handling

| Scenario | Action |
|----------|--------|
| Sitemap fetch fails | Report the error and URL; confirm robots.txt cites the sitemap |
| No repeating pattern found | The site has no generated page family; stop and say so |
| Pattern too small to sample | Report it in the inventory, skip measurement |
| Page fetch fails (SSL EOF, 403, timeout) | Record the error per URL, continue; report under-sampling |
| <30% unique content | Print the HARD STOP gate; require user approval before further analysis |
| Non-Latin script pages | Skip the word-count gate; similarity within the same language group is still valid |

#!/usr/bin/env python3
"""
pp_scan.py - programmatic page scanner (zero dependencies, stdlib only)

Reads a sitemap, groups its URLs into generated-page patterns (the repeated
URL shapes that hold thousands of pages), samples a few pages per pattern,
and measures the signals the programmatic-SEO quality gates depend on:

  - unique content share (Jaccard similarity between sampled pages)
  - word counts of the content body (boilerplate excluded)
  - canonical self-reference and noindex directives
  - internal link density
  - URL rule violations (long URLs, uppercase, query strings, duplicate slugs)

The scanner measures. The model judges: data-source quality, freshness, and
the rollout plan are not observable from the outside and stay human
decisions. The output lists what was measured, what it means against the
thresholds in the SKILL.md, and what needs a human.

Usage:
  python3 pp_scan.py https://example.com/sitemap.xml
  python3 pp_scan.py https://example.com/sitemap.xml --pattern /tools/
  python3 pp_scan.py smoke-sitemap.xml --pattern /tools/
  python3 pp_scan.py --json https://example.com/sitemap.xml
"""
import argparse
import gzip
import json
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
from html.parser import HTMLParser

UA = ("Mozilla/5.0 (compatible; codex-seo-programmatic/1.0; "
      "+https://auspia.ai/blog/codex-seo-programmatic)")
SKIP_TAGS = {"script", "style", "noscript", "svg", "header", "nav", "footer",
             "aside", "iframe", "form"}
CAP = 6_000_000  # per-document byte cap

# --- fetching ---------------------------------------------------------


def fetch(url, timeout=25):
    """Return decoded text of url. Local paths and file:// URLs work too."""
    if url.startswith(("http://", "https://")):
        pass
    else:
        path = url[len("file://"):] if url.startswith("file://") else url
        with open(urllib.request.url2pathname(path),
                  encoding="utf-8", errors="replace") as fh:
            return fh.read()
    req = urllib.request.Request(url, headers={
        "User-Agent": UA, "Accept-Encoding": "gzip"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        raw = r.read(CAP)
        if r.headers.get("Content-Encoding") == "gzip":
            raw = gzip.decompress(raw)
        return raw.decode("utf-8", "replace")


def sitemap_urls(xml):
    """Return list of (url, lastmod_or_None) from a sitemap document."""
    entries = re.findall(r"<url>(.*?)</url>", xml, re.S)
    if entries:
        out = []
        for block in entries:
            loc = re.search(r"<loc>(.*?)</loc>", block, re.S)
            mod = re.search(r"<lastmod>(.*?)</lastmod>", block, re.S)
            if loc:
                out.append((loc.group(1).strip(),
                            mod.group(1).strip() if mod else None))
        return out
    return []


def load_sitemap(target, seen=None):
    """Fetch a sitemap URL, expanding one level of sitemap index files."""
    seen = seen or set()
    if target in seen:
        return []
    seen.add(target)
    xml = fetch(target)
    if "<sitemapindex" in xml[:4000] or ("<sitemap>" in xml
                                         and "<urlset" not in xml[:4000]):
        urls = []
        for m in re.finditer(r"<sitemap>(.*?)</sitemap>", xml, re.S):
            loc = re.search(r"<loc>(.*?)</loc>", m.group(1), re.S)
            if loc and loc.group(1).strip().endswith(".xml"):
                urls += load_sitemap(loc.group(1).strip(), seen)
        return urls
    return sitemap_urls(xml)


# --- URL pattern derivation -------------------------------------------

UUID_RE = re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-"
                     r"[0-9a-f]{4}-[0-9a-f]{12}", re.I)
HEX_RE = re.compile(r"[0-9a-f]{16,}", re.I)
DIGITS_RE = re.compile(r"^\d+$")
# Known language prefixes stay literal so every locale gets its own
# (same language) pattern group instead of being merged by the slug rule.
LOCALES = {"en", "ja", "ko", "zh", "zh-hans", "zh-hant", "pt", "pt-br",
           "ru", "he", "ar", "th", "vi", "uk", "tr", "pl", "ms", "it", "id",
           "hi", "fr", "es", "de", "nl", "sv", "da", "fi", "nb", "no", "cs",
           "sk", "hu", "ro", "bg", "el", "fa", "et", "lv", "lt", "sl",
           "hr", "ca", "az", "sr", "sw"}


def segment_key(seg):
    """Classify one path segment: template word, number, or produced slug."""
    seg = seg.strip("/")
    if not seg:
        return ""
    if DIGITS_RE.match(seg) or UUID_RE.fullmatch(seg) or HEX_RE.fullmatch(seg):
        return "#"
    if seg.lower() in LOCALES:
        return seg.lower()
    if "-" in seg or len(seg) > 20:
        return "<slug>"
    return seg.lower()


def pattern_of(path):
    """Normalize a URL path into a pattern key like /tools/<slug>."""
    path = path.split("?")[0].rstrip("/")
    segs = [segment_key(s) for s in path.split("/") if s]
    return "/" + "/".join(s for s in segs if s)


def collapse_small_groups(raw_groups, min_count):
    """Re-key single-word-slug groups: /tools/xenonlens -> /tools/<slug>."""
    groups = {}
    small = {}
    for key, items in raw_groups.items():
        if len(items) >= min_count:
            groups[key] = items
        else:
            small[key] = items
    for key, items in small.items():
        if len(key.split("/")) < 2:
            continue
        rebuilt = key.split("/")
        rebuilt.pop()
        rebuilt.append("<slug>")
        groups.setdefault("/".join(rebuilt), []).extend(items)
    return groups


def group_patterns(entries):
    """group entries (url, lastmod) into {pattern: [(url, lastmod)]}."""
    raw = {}
    for url, mod in entries:
        parsed = urllib.parse.urlparse(url)
        key = pattern_of(parsed.path)
        if key:
            raw.setdefault(key, []).append((url, mod))
    return collapse_small_groups(raw, 5)


# --- page signals ------------------------------------------------------


def head_signals(html):
    """Regex-extract title, canonical and robots directives from <head>."""
    title = re.search(r"<title[^>]*>(.*?)</title>", html, re.S)
    canon = re.search(r'<link[^>]*rel=["\']?canonical["\']?[^>]*>', html, re.I)
    robots = re.findall(r'<meta[^>]*name=["\']robots["\'][^>]*>', html, re.I)
    noindex = False
    for tag in robots:
        m = re.search(r'content=["\']([^"\']*)["\']', tag, re.I)
        if m and "noindex" in m.group(1).lower():
            noindex = True
    href = None
    if canon:
        m = re.search(r'href=["\']([^"\']*)["\']', canon.group(0), re.I)
        if m:
            href = m.group(1)
    return (title.group(1).strip() if title else None,
            href,
            noindex)


class BodyParser(HTMLParser):
    """Visible text, h1 texts and internal link hrefs, boilerplate excluded."""

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.stack = []
        self.parts = []
        self.h1s = []
        self.links = []
        self._h1 = None

    def _visible(self):
        return not any(t in SKIP_TAGS for t in self.stack)

    def handle_starttag(self, tag, attrs):
        tag = tag.lower()
        if tag not in SKIP_TAGS:
            if tag == "a" and self._visible():
                href = dict(attrs).get("href", "")
                if href:
                    self.links.append(href)
            elif tag == "h1":
                self._h1 = []
        self.stack.append(tag)

    def handle_endtag(self, tag):
        tag = tag.lower()
        if tag == "h1" and self._h1 is not None:
            self.h1s.append(" ".join("".join(self._h1).split()))
            self._h1 = None
        if tag in self.stack:
            while self.stack:
                if self.stack.pop() == tag:
                    break

    def handle_data(self, data):
        if not self._visible():
            return
        self.parts.append(data)
        if self._h1 is not None:
            self._h1.append(data)

    @property
    def text(self):
        return " ".join(self.parts)


def page_signals(html, page_url):
    """Measure one page. Returns a dict of mechanical signals."""
    title, canon_href, noindex = head_signals(html)
    parser = BodyParser()
    parser.feed(html)
    text = parser.text
    tokens = re.findall(r"[0-9A-Za-z]+(?:['’][0-9A-Za-z]+)?", text.lower())
    cjk = len(re.findall(r"[぀-ヿ一-鿿가-힯]", text))
    canon_self = None
    if canon_href:
        pin = urllib.parse.urlsplit(page_url)
        cin = urllib.parse.urlsplit(urllib.parse.urljoin(page_url, canon_href))
        canon_self = (pin.netloc == cin.netloc and
                      pin.path.rstrip("/") == cin.path.rstrip("/"))
    internal = 0
    host = urllib.parse.urlsplit(page_url).netloc
    for h in parser.links:
        h = h.strip()
        if not h or h.startswith(("#", "mailto:", "tel:", "javascript:")):
            continue
        a = urllib.parse.urljoin(page_url, h)
        if urllib.parse.urlsplit(a).netloc == host:
            internal += 1
    return {
        "url": page_url,
        "status_ok": True,
        "title": (title or "")[:90],
        "words": len(tokens),
        "cjk": cjk,
        "h1": parser.h1s,
        "canonical": canon_href,
        "canonical_self": canon_self,
        "noindex": noindex,
        "internal_links": internal,
        "tokens": set(tokens),
    }


def similarity(a, b):
    """Jaccard similarity of two token sets."""
    if not a or not b:
        return 0.0
    return len(a & b) / len(a | b)


# --- measurement -------------------------------------------------------


def sample_indexes(total, n):
    """Evenly spaced sample indexes across a list."""
    if total <= n:
        return list(range(total))
    step = total / n
    return [min(int(i * step), total - 1) for i in range(n)]


def measure_pattern(pattern, items, args, errors):
    page = {}
    idxs = sample_indexes(len(items), args.sample)
    for i in idxs:
        url = items[i][0]
        try:
            page[i] = page_signals(fetch(url), url)
        except Exception as exc:  # noqa: BLE001 - report and move on
            errors.append(f"{url}: {type(exc).__name__}: {exc}")
    n = len(page)
    if n < 2:
        return None, page, errors
    max_sim = {}
    for i in page:
        sims = [similarity(page[i]["tokens"], page[j]["tokens"])
                for j in page if j != i]
        max_sim[i] = max(sims) if sims else 0.0
    unique = [100.0 * (1 - max_sim[i]) for i in page]
    words = [page[i]["words"] for i in page]
    links = [page[i]["internal_links"] for i in page]
    thin = [page[i] for i in page if page[i]["words"] < 300]
    canon = [page[i]["canonical_self"] for i in page if page[i]["canonical"]]
    met = {
        "pages_sampled": n,
        "avg_words": round(sum(words) / len(words)),
        "min_words": min(words),
        "pages_under_300_words": len(thin),
        "avg_top_1_unique_pct": round(sum(unique) / len(unique)),
        "min_unique_pct": round(min(unique)),
        "cjk_samples": sum(1 for i in page if page[i]["cjk"] > 200),
        "canonical_self_pct": (round(100 * sum(canon) / len(canon))
                               if canon else None),
        "noindex_in_sample": sum(1 for i in page if page[i]["noindex"]),
        "avg_internal_links": round(sum(links) / len(links), 1),
        "thin_pages": [p["url"] for p in thin],
        "low_uniqueness_pages": sorted(
            ({page[i]["url"]: round(max_sim[i] * 100) for i in page}
             .items()), key=lambda kv: -kv[1])[:2],
        "nonlatin_caveat": [i for i in page if page[i]["cjk"] > 200],
    }
    return met, page, errors


# --- url rules and scores ----------------------------------------------


def url_rules(groups, host):
    """URL rule violations, count and duplicate slugs judged per pattern."""
    q = lng = up = 0
    total = 0
    dups = 0
    trailing = set()
    for pattern, items in groups.items():
        slugs = {}
        for url, _ in items:
            parsed = urllib.parse.urlsplit(url)
            path = parsed.path
            total += 1
            if parsed.query:
                q += 1
            if len(url) > 100:
                lng += 1
            if any(c.isupper() for c in path):
                up += 1
            trailing.add(path.endswith("/"))
            last = path.rstrip("/").split("/")[-1].lower()
            slugs[last] = slugs.get(last, 0) + 1
        dups += sum(1 for v in slugs.values() if v > 1)
    return {
        "patterns_checked": len(groups),
        "with_query": q, "over_100_chars": lng, "uppercase_path": up,
        "duplicate_slugs": dups, "mixed_trailing_slash": len(trailing) > 1,
        "pct": {"query": 100 * q / total if total else 0,
                "long": 100 * lng / total if total else 0,
                "upper": 100 * up / total if total else 0},
    }


def score(met, rules, sitemap_no_lastmod_pct):
    """Convert measured signals to the six-category scorecard."""
    url = max(0, 100 - round(30 * (rules["pct"]["query"] / 100)
                             + 30 * (rules["pct"]["long"] / 100)
                             + 20 * (rules["pct"]["upper"] / 100)
                             + 20 * (rules["duplicate_slugs"] > 0)))
    if met is None:
        return {"Data Quality": "n/a", "URL Structure": url,
                "Template Uniqueness": "n/a", "Thin Content Risk": "n/a",
                "Internal Linking": "n/a", "Index Management": "n/a"}
    uni = met["avg_top_1_unique_pct"]
    thin = 100
    under = met["pages_under_300_words"]
    if met["nonlatin_caveat"]:
        under = 0  # word gate not applicable to non-Latin token counts
    thin -= 35 * (under / met["pages_sampled"])
    if uni < 40:
        thin -= 45
    if uni < 30:
        thin -= 60
    thin = max(0, round(thin))
    expected = 4 * met["avg_words"] / 1000 if met["avg_words"] else 1
    link = min(100, round(100 * met["avg_internal_links"] / max(1, expected)))
    idx = 100
    if met["canonical_self_pct"] is not None:
        idx -= round(30 * (100 - met["canonical_self_pct"]) / 100)
    idx -= 30 * (met["noindex_in_sample"] / met["pages_sampled"])
    if sitemap_no_lastmod_pct is not None:
        idx -= round(30 * sitemap_no_lastmod_pct / 100)
    idx = max(0, round(idx))
    return {"Data Quality": "n/a", "URL Structure": url,
            "Template Uniqueness": uni, "Thin Content Risk": thin,
            "Internal Linking": link, "Index Management": idx}


# --- gates -------------------------------------------------------------

GATE_UNIQUE_FLAG = 40    # original quality gate: below 40% -> thin
GATE_UNIQUE_STOP = 30    # hardening recommendation: below 30% -> hard stop
GATE_WORDS = 300         # original quality gate: below 300 words -> review


def gates_of(pattern, count, met):
    """Apply the quality gates in the SKILL.md. Returns list of gate notes."""
    out = []
    if count >= 100:
        out.append("WARNING pattern size %d pages - confirm a staged rollout "
                   "and a review sample before relying on this pattern"
                   % count)
    if met is None:
        out.append("PATTERN NOT SAMPLED (fetch errors) - treat all gates "
                   "as unresolved")
        return out
    if count >= 100 and met["pages_sampled"] < 5:
        out.append("WARNING sample smaller than 5 - inconclusive gates")
    if met["nonlatin_caveat"]:
        out.append("NOTE %d sampled pages show non-Latin script (CJK/kana/"
                   "hangul): word counts are token counts and undercount. "
                   "Words gate skipped; similarity within this language "
                   "group is still comparable."
                   % len(met["nonlatin_caveat"]))
    elif met["pages_under_300_words"]:
        out.append("FLAG %d/%d sampled pages under %d words (review list: %s)"
                   % (met["pages_under_300_words"], met["pages_sampled"],
                      GATE_WORDS, "; ".join(met["thin_pages"][:2])))
    if met["avg_top_1_unique_pct"] < GATE_UNIQUE_STOP:
        out.append("HARD STOP avg top-1 unique content %.0f%% < %d%% - the "
                   "template reproduces itself; do not publish more of this "
                   "pattern"
                   % (met["avg_top_1_unique_pct"], GATE_UNIQUE_STOP))
    elif met["avg_top_1_unique_pct"] < GATE_UNIQUE_FLAG:
        out.append("FLAG avg top-1 unique content %.0f%% < %d%% - thin "
                   "content risk; differentiate pages before scaling"
                   % (met["avg_top_1_unique_pct"], GATE_UNIQUE_FLAG))
    else:
        out.append("PASS unique content %.0f%% - gate satisfied"
                   % met["avg_top_1_unique_pct"])
    return out


# --- report ------------------------------------------------------------

SEPARATOR = "-" * 62


def cross_pattern_duplicates(patterns):
    """Slug names appearing in more than one pattern (locale mirrors)."""
    seen = {}
    for pattern, items in patterns:
        if len(items) < 5:
            continue
        for url, _ in items:
            last = urllib.parse.urlsplit(url).path.rstrip("/")
            last = last.split("/")[-1].lower()
            seen.setdefault(last, set()).add(pattern)
    return sum(1 for v in seen.values() if len(v) > 1)


def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("sitemap", help="sitemap URL or local .xml path")
    ap.add_argument("--pattern", default=None,
                    help="only URLs containing this substring (e.g. /tools/)")
    ap.add_argument("--sample", type=int, default=8,
                    help="pages fetched per pattern (default 8)")
    ap.add_argument("--max-fetch", type=int, default=24,
                    help="hard cap on fetched pages (default 24)")
    ap.add_argument("--json", action="store_true")
    args = ap.parse_args(argv)

    errors = []
    try:
        entries = load_sitemap(args.sitemap)
    except Exception as exc:  # noqa: BLE001
        print("SITEMAP FETCH FAILED: %s: %s" % (type(exc).__name__, exc))
        return 2
    if not entries:
        print("SITEMAP EMPTY or unparseable: %s" % args.sitemap)
        return 2
    if args.pattern:
        filtered = [(u, m) for u, m in entries if args.pattern in u]
        if not filtered:
            print("NO URLS MATCH --pattern %s (sitemap has %d URLs)"
                  % (args.pattern, len(entries)))
            return 2
        entries = filtered
    total = len(entries)
    groups = group_patterns(entries)
    if not groups:
        print("NO REPEATING URL PATTERN FOUND in %d URLs - nothing here "
              "looks generated at scale" % total)
        return 2

    tested = []
    fetched = 0
    patterns = sorted(groups.items(), key=lambda kv: -len(kv[1]))
    for pattern, items in patterns:
        if fetched >= args.max_fetch or len(items) < 5:
            continue
        met, pages, errs = measure_pattern(pattern, items, args, errors)
        fetched += len(pages)
        if met is None and not pages:
            continue
        tested.append((pattern, items, met, pages))

    sitemap_no_lastmod = sum(1 for _, m in entries if not m)
    sitemap_pct = 100 * sitemap_no_lastmod / total if total else None
    rules = url_rules(groups, urllib.parse.urlsplit(entries[0][0]).netloc)

    if args.json:
        return json_report(entries, total, groups, tested, rules,
                           sitemap_pct, errors)

    print("Programmatic scan: %s" % args.sitemap)
    if args.pattern:
        print("  filter --pattern %s | URLs in scope: %d" % (args.pattern,
                                                             total))
    print("  sitemap: %d URLs | %d%% lack <lastmod>"
          % (total, round(sitemap_pct or 0)))
    print(SEPARATOR)
    print("URL pattern inventory (count >= 5):")
    sampled_keys = {p for p, _, _, _ in tested}
    for pattern, items in patterns:
        if len(items) < 5:
            continue
        print("  %-34s %5d pages%s" % (pattern, len(items),
                                       "   (sampled)" if pattern in
                                       sampled_keys else ""))
    print(SEPARATOR)
    per_pattern = {}
    for pattern, items, met, _ in tested:
        per_pattern[pattern] = met
        print("Pattern: %s (%d pages)" % (pattern, len(items)))
        if met["nonlatin_caveat"]:
            print("  non-Latin script detected; word counts are token "
                  "counts, see note below")
        print("  sampled:      %d pages evenly spaced over %d"
              % (met["pages_sampled"], len(items)))
        print("  avg body words:      %d   (min %d, pages < %d words: %d)"
              % (met["avg_words"], met["min_words"], GATE_WORDS,
                 met["pages_under_300_words"]))
        print("  avg top-1 unique:  %d%%   (min %d%%)"
              % (met["avg_top_1_unique_pct"], met["min_unique_pct"]))
        print("  canonical self-ref:  %s"
              % (str(met["canonical_self_pct"]) + "%"
                 if met["canonical_self_pct"] is not None else "no canonicals"))
        print("  noindex in sample:   %d" % met["noindex_in_sample"])
        print("  avg internal links:  %.1f" % met["avg_internal_links"])
        for gate in gates_of(pattern, len(items), met):
            print("  [GATE] %s" % gate)
        print()
    print(SEPARATOR)
    print("URL rule check over all %d URLs:" % total)
    print("  query strings: %d | >100 chars: %d | uppercase in path: %d"
          % (rules["with_query"], rules["over_100_chars"],
             rules["uppercase_path"]))
    print("  duplicate end-slugs across %d patterns (within-pattern): %d"
          % (rules["patterns_checked"], rules["duplicate_slugs"]))
    print("  mixed trailing slash: %s" % rules["mixed_trailing_slash"])
    cross = cross_pattern_duplicates(patterns)
    if cross:
        print("  cross-pattern duplicate slugs: %d (locale mirrors of the "
              "same page count here; judge, don't penalize)"
              % cross)
    print(SEPARATOR)
    print("Programmatic score (mechanics only; Data Quality is a human "
          "call - the data source lives behind the CMS):")
    for i, (pattern, items, met, _) in enumerate(tested):
        if i:
            print()
        sc = score(met, rules, sitemap_pct)
        print("  %s" % pattern)
        for k, v in sc.items():
            print("    %-22s %s" % (k, v))
        vals = [v for v in sc.values() if isinstance(v, (int, float))]
        if vals:
            print("    TOTAL (mean of measured): %d/100"
                  % round(sum(vals) / len(vals)))
    print(SEPARATOR)
    if errors:
        print("Fetch errors (%d):" % len(errors))
        for e in errors[:6]:
            print("  " + e)
    return 0


def json_report(entries, total, groups, tested, rules, sitemap_pct, errors):
    payload = {
        "sitemap_url": None,
        "total_urls": total,
        "url_rules": rules,
        "sitemap_no_lastmod_pct": sitemap_pct,
        "patterns": {},
        "fetch_errors": errors,
    }
    for pattern, items, met, _ in tested:
        payload["patterns"][pattern] = {
            "count": len(items),
            "gate": gates_of(pattern, len(items), met),
            "metrics": met,
            "score": score(met, rules, sitemap_pct),
        }
    json.dump(payload, sys.stdout, indent=1, ensure_ascii=False)
    return 0


if __name__ == "__main__":
    sys.exit(main())

mkdir -p ~/.codex/skills/codex-seo-programmatic/scripts \
         ~/.codex/skills/codex-seo-programmatic/fixtures

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://auspia.ai/tools/landing-page-diagnostic</loc></url>
<url><loc>https://auspia.ai/tools/website-seo-score-checker</loc></url>
<url><loc>https://auspia.ai/tools/ai-search-visibility-checker</loc></url>
<url><loc>https://auspia.ai/tools/llms-txt-generator-checker</loc></url>
<url><loc>https://auspia.ai/tools/robots-txt-ai-crawler-checker</loc></url>
<url><loc>https://auspia.ai/tools/ai_overview</loc></url>
<url><loc>https://auspia.ai/tools/one-page-geo-citation-audit</loc></url>
<url><loc>https://auspia.ai/tools/best-tools-list-debiaser</loc></url>
<url><loc>https://auspia.ai/tools/google-index-checker</loc></url>
<url><loc>https://auspia.ai/tools/http-status-code-checker</loc></url>
<url><loc>https://auspia.ai/tools/on-page-seo-audit</loc></url>
<url><loc>https://auspia.ai/tools/googlebot-spider-simulator</loc></url>
</urlset>

python3 ~/.codex/skills/codex-seo-programmatic/scripts/pp_scan.py \
  ~/.codex/skills/codex-seo-programmatic/fixtures/smoke.xml

python3 ~/.codex/skills/codex-seo-programmatic/scripts/pp_scan.py https://yourdomain.com/sitemap.xml

Programmatic scan: https://auspia.ai/sitemap.xml
  filter --pattern /tools/ | URLs in scope: 441
  sitemap: 441 URLs | 0% lack <lastmod>
--------------------------------------------------------------
URL pattern inventory (count >= 5):
  /tools/<slug>                         21 pages   (sampled)
  /ja/tools/<slug>                      21 pages   (sampled)
  /ko/tools/<slug>                      21 pages   (sampled)
  /zh-hant/tools/<slug>                 21 pages
  /zh-hans/tools/<slug>                 21 pages
  /ar/tools/<slug>                      21 pages
  /he/tools/<slug>                      21 pages
  /ru/tools/<slug>                      21 pages
  /th/tools/<slug>                      21 pages
  /pt-br/tools/<slug>                   21 pages
  /es/tools/<slug>                      21 pages
  /de/tools/<slug>                      21 pages
  /fr/tools/<slug>                      21 pages
  /id/tools/<slug>                      21 pages
  /it/tools/<slug>                      21 pages
  /tr/tools/<slug>                      21 pages
  /uk/tools/<slug>                      21 pages
  /pl/tools/<slug>                      21 pages
  /hi/tools/<slug>                      21 pages
  /vi/tools/<slug>                      21 pages
  /ms/tools/<slug>                      21 pages
--------------------------------------------------------------
Pattern: /tools/<slug> (21 pages)
  sampled:      8 pages evenly spaced over 21
  avg body words:      741   (min 316, pages < 300 words: 0)
  avg top-1 unique:  72%   (min 62%)
  canonical self-ref:  100%
  noindex in sample:   0
  avg internal links:  2.1
  [GATE] PASS unique content 72% - gate satisfied

Pattern: /ja/tools/<slug> (21 pages)
  non-Latin script detected; word counts are token counts, see note below
  sampled:      8 pages evenly spaced over 21
  avg body words:      94   (min 44, pages < 300 words: 8)
  avg top-1 unique:  68%   (min 50%)
  canonical self-ref:  100%
  noindex in sample:   0
  avg internal links:  2.1
  [GATE] NOTE 8 sampled pages show non-Latin script (CJK/kana/hangul): word counts are token counts and undercount. Words gate skipped; similarity within this language group is still comparable.
  [GATE] PASS unique content 68% - gate satisfied

Pattern: /ko/tools/<slug> (21 pages)
  non-Latin script detected; word counts are token counts, see note below
  sampled:      8 pages evenly spaced over 21
  avg body words:      96   (min 42, pages < 300 words: 8)
  avg top-1 unique:  68%   (min 48%)
  canonical self-ref:  100%
  noindex in sample:   0
  avg internal links:  2.1
  [GATE] NOTE 8 sampled pages show non-Latin script (CJK/kana/hangul): word counts are token counts and undercount. Words gate skipped; similarity within this language group is still comparable.
  [GATE] PASS unique content 68% - gate satisfied

--------------------------------------------------------------
URL rule check over all 441 URLs:
  query strings: 0 | >100 chars: 0 | uppercase in path: 0
  duplicate end-slugs across 21 patterns (within-pattern): 0
  mixed trailing slash: False
  cross-pattern duplicate slugs: 21 (locale mirrors of the same page count here; judge, don't penalize)
--------------------------------------------------------------
Programmatic score (mechanics only; Data Quality is a human call - the data source lives behind the CMS):
  /tools/<slug>
    Data Quality           n/a
    URL Structure          100
    Template Uniqueness    72
    Thin Content Risk      100
    Internal Linking       71
    Index Management       100
    TOTAL (mean of measured): 89/100

  /ja/tools/<slug>
    Data Quality           n/a
    URL Structure          100
    Template Uniqueness    68
    Thin Content Risk      100
    Internal Linking       100
    Index Management       100
    TOTAL (mean of measured): 94/100

  /ko/tools/<slug>
    Data Quality           n/a
    URL Structure          100
    Template Uniqueness    68
    Thin Content Risk      100
    Internal Linking       100
    Index Management       100
    TOTAL (mean of measured): 94/100
--------------------------------------------------------------

Programmatic scan: https://auspia.ai/blog/blog-0.xml
  sitemap: 4354 URLs | 100% lack <lastmod>
--------------------------------------------------------------
URL pattern inventory (count >= 5):
  /blog/<slug>                         932 pages   (sampled)
  /ja/blog/<slug>                      266 pages   (sampled)
  /zh-hant/blog/<slug>                 256 pages   (sampled)
  /zh-hans/blog/<slug>                 249 pages
  /ko/blog/<slug>                      194 pages
  /pt-br/blog/<slug>                   179 pages
  /es/blog/<slug>                      161 pages
  /ru/blog/<slug>                      161 pages
  /de/blog/<slug>                      159 pages
  /fr/blog/<slug>                      158 pages
  /th/blog/<slug>                      151 pages
  /id/blog/<slug>                      150 pages
  /it/blog/<slug>                      150 pages
  /ar/blog/<slug>                      149 pages
  /tr/blog/<slug>                      149 pages
  /he/blog/<slug>                      147 pages
  /pl/blog/<slug>                      145 pages
  /uk/blog/<slug>                      145 pages
  /hi/blog/<slug>                      144 pages
  /ms/blog/<slug>                      144 pages
  /vi/blog/<slug>                      144 pages
--------------------------------------------------------------
Pattern: /blog/<slug> (932 pages)
  sampled:      8 pages evenly spaced over 932
  avg body words:      2243   (min 634, pages < 300 words: 0)
  avg top-1 unique:  66%   (min 61%)
  canonical self-ref:  100%
  noindex in sample:   0
  avg internal links:  10.9
  [GATE] WARNING pattern size 932 pages - confirm a staged rollout and a review sample before relying on this pattern
  [GATE] PASS unique content 66% - gate satisfied

Pattern: /ja/blog/<slug> (266 pages)
  non-Latin script detected; word counts are token counts, see note below
  sampled:      8 pages evenly spaced over 266
  avg body words:      355   (min 140, pages < 300 words: 5)
  avg top-1 unique:  58%   (min 44%)
  canonical self-ref:  100%
  noindex in sample:   0
  avg internal links:  8.0
  [GATE] WARNING pattern size 266 pages - confirm a staged rollout and a review sample before relying on this pattern
  [GATE] NOTE 8 sampled pages show non-Latin script (CJK/kana/hangul): word counts are token counts and undercount. Words gate skipped; similarity within this language group is still comparable.
  [GATE] PASS unique content 58% - gate satisfied

Pattern: /zh-hant/blog/<slug> (256 pages)
  non-Latin script detected; word counts are token counts, see note below
  sampled:      8 pages evenly spaced over 256
  avg body words:      572   (min 167, pages < 300 words: 3)
  avg top-1 unique:  68%   (min 59%)
  canonical self-ref:  100%
  noindex in sample:   0
  avg internal links:  9.2
  [GATE] WARNING pattern size 256 pages - confirm a staged rollout and a review sample before relying on this pattern
  [GATE] NOTE 8 sampled pages show non-Latin script (CJK/kana/hangul): word counts are token counts and undercount. Words gate skipped; similarity within this language group is still comparable.
  [GATE] PASS unique content 68% - gate satisfied

--------------------------------------------------------------
URL rule check over all 4354 URLs:
  query strings: 0 | >100 chars: 4 | uppercase in path: 0
  duplicate end-slugs across 42 patterns (within-pattern): 0
  mixed trailing slash: False
  cross-pattern duplicate slugs: 274 (locale mirrors of the same page count here; judge, don't penalize)
--------------------------------------------------------------
Programmatic score (mechanics only; Data Quality is a human call - the data source lives behind the CMS):
  /blog/<slug>
    Data Quality           n/a
    URL Structure          100
    Template Uniqueness    66
    Thin Content Risk      100
    Internal Linking       100
    Index Management       70
    TOTAL (mean of measured): 87/100

  /ja/blog/<slug>
    Data Quality           n/a
    URL Structure          100
    Template Uniqueness    58
    Thin Content Risk      100
    Internal Linking       100
    Index Management       70
    TOTAL (mean of measured): 86/100

  /zh-hant/blog/<slug>
    Data Quality           n/a
    URL Structure          100
    Template Uniqueness    68
    Thin Content Risk      100
    Internal Linking       100
    Index Management       70
    TOTAL (mean of measured): 88/100
--------------------------------------------------------------

My task: install the codex-seo-programmatic skill the easy way.
Read the three code blocks below, one at a time.

A. Save this as ~/.codex/skills/codex-seo-programmatic/SKILL.md:

~~~~markdown
---
name: codex-seo-programmatic
description: Use when the user asks about programmatic SEO, pages generated at scale from a data source, template pages, dynamic or data-driven pages, thin content at scale, scaled content abuse risk, index bloat, faceted navigation, auto-generated internal links, or a sitemap filled with many similar URLs. Audits a sitemap for repeating URL patterns and measures uniqueness, word counts, canonicals, and link density against the quality gates.
---
# Programmatic SEO: Pages at Scale

Programmatic SEO builds pages from a data source (a CSV, an API, a database)
through a template. The quality problem is not volume. One thousand pages
multiply whatever the template does - they multiply its weaknesses faster
than its strengths, and they multiply your risk if the template is thin.

## Core insight

A "mad-libs" template is invisible one page at a time. Swap the city name in
an identical 400-word paragraph and no single page looks wrong. Publish it
200 times and Google sees a scaled content abuse pattern, and the site - not
just those pages - starts losing trust. Every programmatic rollout has two
mechanical questions that a scanner can answer and one that only judgment
can:

1. Is each template family unique enough to exist on its own?
2. Are the URLs, canonicals and sitemap entries behaving themselves?
3. Does the data source actually differentiate the pages? (judgment)

Mechanics are the script's job. Judgment is yours.

## Commands

- `pp_scan.py` groups sitemap URLs into pattern families (e.g.
  `/tools/<slug>`), samples pages per family, and measures: unique content
  share (top-1 Jaccard), body word counts, canonical self-reference,
  noindex directives, internal link density, and URL rule violations.
- `--pattern` limits the scan to URLs containing that substring. Use it when
  you already know which pattern family you care about.
- `--json` prints machine-readable output for larger scripts.
- Local sitemap files work too: `python3 pp_scan.py sitemap.xml`, which
  makes the scan fully replayable.

## What the scanner measures

| Signal | How | Gate |
|--------|-----|------|
| Unique content | Top-1 Jaccard similarity between sampled pages of one pattern | PASS ≥40%; FLAG <40%; HARD STOP <30% |
| Body depth | Word count of visible body text (headers/nav/footer excluded) | FLAG <300 words |
| Canonical | `rel=canonical` present and self-referencing | Report % |
| Indexing | `noindex` inside sitemap-scoped pages | Report count |
| Internal links | Links to same host per page | Score vs 3-5/1000 words |
| URL rules | Query strings, >100 chars, uppercase, duplicate slugs | No-violation target |

The scanner excludes `<header>`, `<nav>`, `<footer>`, `<aside>`, `<script>`,
`<style>` from body text and link counting, so shared boilerplate does not
feed the similarity numbers.

## What the scanner cannot see (judgment tasks)

| Not observable from outside | Why |
|------------------------------|-----|
| Data source row quality | Rows, uniqueness, missing values live behind the CMS |
| Data freshness | Stale input produces stale pages |
| Review pipeline | Whether someone reads pages before publish |
| Standalone value | "Worth publishing if no sibling existed" |
| Locale duplication intent | hreflang mirrors vs duplicate content - use `codex-seo-hreflang` |

The output prints these as `n/a` on purpose: a score for a number you did
not measure is a lie.

## Quality gates (scaled content abuse, 2024-2026)

Google's Scaled Content Abuse policy came into force March 2024 and
enforcement escalated through 2025 (manual actions in June 2025, stronger
SpamBrain detection reported from August 2025). Google reported a 45%
reduction in low-quality, unoriginal content in its results after the
policy began. The gates below come from that context and from the
open-source methodology this skill adapts:

| Metric | Threshold | Action |
|--------|-----------|--------|
| Pages without content review | 100+ | WARNING: audit before relying on scale |
| Pages without justification | 500+ | HARD STOP: explicit approval required |
| Unique content per page | <40% | FLAG as thin content (penalty risk) |
| Word count per page | <300 | FLAG for review (may lack value) |

Hardened values for high-volume sites: treat <30% unique content as a HARD
STOP, keep 5-10% human review of generated pages, and publish in batches of
50-100 pages with 2-4 weeks of indexing/ranking observation before scaling
further.

### Safe patterns (work at scale)

- Integration pages with real setup docs, API details, screenshots
- Template/tool pages with downloadable content and usage instructions
- Glossary pages: 200+ word definitions with examples and related terms
- Product pages with unique specs, reviews, comparison data
- Data-driven pages with unique per-record statistics and analysis

### Penalty-risk patterns (avoid at scale)

- Location pages that only swap the city name in identical text
- "Best tool for [industry]" without industry-specific value
- "[Competitor] alternative" without real comparison data
- AI-generated pages without human review and unique value-add
- Pages where >60% of content is shared template boilerplate

## Uniqueness calculation

Measured against all other pages in the same template family. Shared
headers, footers and navigation are excluded. Template boilerplate text is
INCLUDED: identical paragraphs count against you even when they look
harmless.

## URL rules

- Lowercase, hyphenated slugs derived from the data
- No duplicate slugs; enforce uniqueness at generation time
- Keep URLs under 100 characters
- No query parameters for primary content URLs
- Consistent trailing slash usage (match the site's existing pattern)
- Every programmatic page carries a self-referencing canonical

## Internal linking

- Hub/spoke: category hub pages link to every page in their family
- Related items: auto-link 3-5 pages sharing a data attribute
- Breadcrumbs: BreadcrumbList schema from the URL hierarchy
- Anchor text: descriptive and varied, never the same exact-match string
- Density: 3-5 internal links per 1000 words

## Sitemap integration

- Split at 50,000 URLs or 50MB uncompressed per file, whichever comes first
- Use a sitemap index when the set needs multiple files
- `<lastmod>` reflects the real data update time, never generation time
- Exclude noindexed pages; register the sitemap in robots.txt
- Regenerate as records are added to the data source

## Index bloat prevention

- Noindex low-value pages instead of keeping them indexed
- Canonicalize true duplicates and low-value filtered views to their base
- For >10k programmatic URLs, monitor crawl stats in Search Console
- Merge records with insufficient data into aggregated pages instead of
  one page each
- Monthly check: indexed count vs intended count

## Scoring

The script scores five measurable categories, each 0-100:

| Category | How the score is computed |
|----------|---------------------------|
| URL Structure | 100 minus 30×query-fraction, 30×long-fraction, 20×uppercase, 20 if duplicate slugs |
| Template Uniqueness | = average top-1 unique % (0-100) |
| Thin Content Risk | 100 minus 35 per sampled-page share under 300 words, minus 45 if unique <40%, minus 60 if <30% |
| Internal Linking | min(100, actual links ÷ (words/1000 × 4) × 100) |
| Index Management | 100 minus 30 per non-self-canonical share, 30 per noindex share, 30 per lastmod-missing share |

Total = mean of the measured categories. Data Quality is always `n/a` -
the data source is not observable from the public web, and no score is
printed for it.

## Output contract

1. Pattern inventory: every URL family with 5+ pages, counts sorted.
2. Per-pattern measurements with gate notes (PASS / FLAG / HARD STOP /
   WARNING / NOTE).
3. URL rule check over the full set.
4. Six-category scorecard with TOTAL.
5. Recommendations, ranked CRITICAL / HIGH / MEDIUM / LOW, each with the
   evidence line it came from.

## Error handling

| Scenario | Action |
|----------|--------|
| Sitemap fetch fails | Report the error and URL; confirm robots.txt cites the sitemap |
| No repeating pattern found | The site has no generated page family; stop and say so |
| Pattern too small to sample | Report it in the inventory, skip measurement |
| Page fetch fails (SSL EOF, 403, timeout) | Record the error per URL, continue; report under-sampling |
| <30% unique content | Print the HARD STOP gate; require user approval before further analysis |
| Non-Latin script pages | Skip the word-count gate; similarity within the same language group is still valid |

~~~~

B. Save this as ~/.codex/skills/codex-seo-programmatic/scripts/pp_scan.py:

~~~~python
#!/usr/bin/env python3
"""
pp_scan.py - programmatic page scanner (zero dependencies, stdlib only)

Reads a sitemap, groups its URLs into generated-page patterns (the repeated
URL shapes that hold thousands of pages), samples a few pages per pattern,
and measures the signals the programmatic-SEO quality gates depend on:

  - unique content share (Jaccard similarity between sampled pages)
  - word counts of the content body (boilerplate excluded)
  - canonical self-reference and noindex directives
  - internal link density
  - URL rule violations (long URLs, uppercase, query strings, duplicate slugs)

The scanner measures. The model judges: data-source quality, freshness, and
the rollout plan are not observable from the outside and stay human
decisions. The output lists what was measured, what it means against the
thresholds in the SKILL.md, and what needs a human.

Usage:
  python3 pp_scan.py https://example.com/sitemap.xml
  python3 pp_scan.py https://example.com/sitemap.xml --pattern /tools/
  python3 pp_scan.py smoke-sitemap.xml --pattern /tools/
  python3 pp_scan.py --json https://example.com/sitemap.xml
"""
import argparse
import gzip
import json
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
from html.parser import HTMLParser

UA = ("Mozilla/5.0 (compatible; codex-seo-programmatic/1.0; "
      "+https://auspia.ai/blog/codex-seo-programmatic)")
SKIP_TAGS = {"script", "style", "noscript", "svg", "header", "nav", "footer",
             "aside", "iframe", "form"}
CAP = 6_000_000  # per-document byte cap

# --- fetching ---------------------------------------------------------


def fetch(url, timeout=25):
    """Return decoded text of url. Local paths and file:// URLs work too."""
    if url.startswith(("http://", "https://")):
        pass
    else:
        path = url[len("file://"):] if url.startswith("file://") else url
        with open(urllib.request.url2pathname(path),
                  encoding="utf-8", errors="replace") as fh:
            return fh.read()
    req = urllib.request.Request(url, headers={
        "User-Agent": UA, "Accept-Encoding": "gzip"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        raw = r.read(CAP)
        if r.headers.get("Content-Encoding") == "gzip":
            raw = gzip.decompress(raw)
        return raw.decode("utf-8", "replace")


def sitemap_urls(xml):
    """Return list of (url, lastmod_or_None) from a sitemap document."""
    entries = re.findall(r"<url>(.*?)</url>", xml, re.S)
    if entries:
        out = []
        for block in entries:
            loc = re.search(r"<loc>(.*?)</loc>", block, re.S)
            mod = re.search(r"<lastmod>(.*?)</lastmod>", block, re.S)
            if loc:
                out.append((loc.group(1).strip(),
                            mod.group(1).strip() if mod else None))
        return out
    return []


def load_sitemap(target, seen=None):
    """Fetch a sitemap URL, expanding one level of sitemap index files."""
    seen = seen or set()
    if target in seen:
        return []
    seen.add(target)
    xml = fetch(target)
    if "<sitemapindex" in xml[:4000] or ("<sitemap>" in xml
                                         and "<urlset" not in xml[:4000]):
        urls = []
        for m in re.finditer(r"<sitemap>(.*?)</sitemap>", xml, re.S):
            loc = re.search(r"<loc>(.*?)</loc>", m.group(1), re.S)
            if loc and loc.group(1).strip().endswith(".xml"):
                urls += load_sitemap(loc.group(1).strip(), seen)
        return urls
    return sitemap_urls(xml)


# --- URL pattern derivation -------------------------------------------

UUID_RE = re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-"
                     r"[0-9a-f]{4}-[0-9a-f]{12}", re.I)
HEX_RE = re.compile(r"[0-9a-f]{16,}", re.I)
DIGITS_RE = re.compile(r"^\d+$")
# Known language prefixes stay literal so every locale gets its own
# (same language) pattern group instead of being merged by the slug rule.
LOCALES = {"en", "ja", "ko", "zh", "zh-hans", "zh-hant", "pt", "pt-br",
           "ru", "he", "ar", "th", "vi", "uk", "tr", "pl", "ms", "it", "id",
           "hi", "fr", "es", "de", "nl", "sv", "da", "fi", "nb", "no", "cs",
           "sk", "hu", "ro", "bg", "el", "fa", "et", "lv", "lt", "sl",
           "hr", "ca", "az", "sr", "sw"}


def segment_key(seg):
    """Classify one path segment: template word, number, or produced slug."""
    seg = seg.strip("/")
    if not seg:
        return ""
    if DIGITS_RE.match(seg) or UUID_RE.fullmatch(seg) or HEX_RE.fullmatch(seg):
        return "#"
    if seg.lower() in LOCALES:
        return seg.lower()
    if "-" in seg or len(seg) > 20:
        return "<slug>"
    return seg.lower()


def pattern_of(path):
    """Normalize a URL path into a pattern key like /tools/<slug>."""
    path = path.split("?")[0].rstrip("/")
    segs = [segment_key(s) for s in path.split("/") if s]
    return "/" + "/".join(s for s in segs if s)


def collapse_small_groups(raw_groups, min_count):
    """Re-key single-word-slug groups: /tools/xenonlens -> /tools/<slug>."""
    groups = {}
    small = {}
    for key, items in raw_groups.items():
        if len(items) >= min_count:
            groups[key] = items
        else:
            small[key] = items
    for key, items in small.items():
        if len(key.split("/")) < 2:
            continue
        rebuilt = key.split("/")
        rebuilt.pop()
        rebuilt.append("<slug>")
        groups.setdefault("/".join(rebuilt), []).extend(items)
    return groups


def group_patterns(entries):
    """group entries (url, lastmod) into {pattern: [(url, lastmod)]}."""
    raw = {}
    for url, mod in entries:
        parsed = urllib.parse.urlparse(url)
        key = pattern_of(parsed.path)
        if key:
            raw.setdefault(key, []).append((url, mod))
    return collapse_small_groups(raw, 5)


# --- page signals ------------------------------------------------------


def head_signals(html):
    """Regex-extract title, canonical and robots directives from <head>."""
    title = re.search(r"<title[^>]*>(.*?)</title>", html, re.S)
    canon = re.search(r'<link[^>]*rel=["\']?canonical["\']?[^>]*>', html, re.I)
    robots = re.findall(r'<meta[^>]*name=["\']robots["\'][^>]*>', html, re.I)
    noindex = False
    for tag in robots:
        m = re.search(r'content=["\']([^"\']*)["\']', tag, re.I)
        if m and "noindex" in m.group(1).lower():
            noindex = True
    href = None
    if canon:
        m = re.search(r'href=["\']([^"\']*)["\']', canon.group(0), re.I)
        if m:
            href = m.group(1)
    return (title.group(1).strip() if title else None,
            href,
            noindex)


class BodyParser(HTMLParser):
    """Visible text, h1 texts and internal link hrefs, boilerplate excluded."""

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.stack = []
        self.parts = []
        self.h1s = []
        self.links = []
        self._h1 = None

    def _visible(self):
        return not any(t in SKIP_TAGS for t in self.stack)

    def handle_starttag(self, tag, attrs):
        tag = tag.lower()
        if tag not in SKIP_TAGS:
            if tag == "a" and self._visible():
                href = dict(attrs).get("href", "")
                if href:
                    self.links.append(href)
            elif tag == "h1":
                self._h1 = []
        self.stack.append(tag)

    def handle_endtag(self, tag):
        tag = tag.lower()
        if tag == "h1" and self._h1 is not None:
            self.h1s.append(" ".join("".join(self._h1).split()))
            self._h1 = None
        if tag in self.stack:
            while self.stack:
                if self.stack.pop() == tag:
                    break

    def handle_data(self, data):
        if not self._visible():
            return
        self.parts.append(data)
        if self._h1 is not None:
            self._h1.append(data)

    @property
    def text(self):
        return " ".join(self.parts)


def page_signals(html, page_url):
    """Measure one page. Returns a dict of mechanical signals."""
    title, canon_href, noindex = head_signals(html)
    parser = BodyParser()
    parser.feed(html)
    text = parser.text
    tokens = re.findall(r"[0-9A-Za-z]+(?:['’][0-9A-Za-z]+)?", text.lower())
    cjk = len(re.findall(r"[぀-ヿ一-鿿가-힯]", text))
    canon_self = None
    if canon_href:
        pin = urllib.parse.urlsplit(page_url)
        cin = urllib.parse.urlsplit(urllib.parse.urljoin(page_url, canon_href))
        canon_self = (pin.netloc == cin.netloc and
                      pin.path.rstrip("/") == cin.path.rstrip("/"))
    internal = 0
    host = urllib.parse.urlsplit(page_url).netloc
    for h in parser.links:
        h = h.strip()
        if not h or h.startswith(("#", "mailto:", "tel:", "javascript:")):
            continue
        a = urllib.parse.urljoin(page_url, h)
        if urllib.parse.urlsplit(a).netloc == host:
            internal += 1
    return {
        "url": page_url,
        "status_ok": True,
        "title": (title or "")[:90],
        "words": len(tokens),
        "cjk": cjk,
        "h1": parser.h1s,
        "canonical": canon_href,
        "canonical_self": canon_self,
        "noindex": noindex,
        "internal_links": internal,
        "tokens": set(tokens),
    }


def similarity(a, b):
    """Jaccard similarity of two token sets."""
    if not a or not b:
        return 0.0
    return len(a & b) / len(a | b)


# --- measurement -------------------------------------------------------


def sample_indexes(total, n):
    """Evenly spaced sample indexes across a list."""
    if total <= n:
        return list(range(total))
    step = total / n
    return [min(int(i * step), total - 1) for i in range(n)]


def measure_pattern(pattern, items, args, errors):
    page = {}
    idxs = sample_indexes(len(items), args.sample)
    for i in idxs:
        url = items[i][0]
        try:
            page[i] = page_signals(fetch(url), url)
        except Exception as exc:  # noqa: BLE001 - report and move on
            errors.append(f"{url}: {type(exc).__name__}: {exc}")
    n = len(page)
    if n < 2:
        return None, page, errors
    max_sim = {}
    for i in page:
        sims = [similarity(page[i]["tokens"], page[j]["tokens"])
                for j in page if j != i]
        max_sim[i] = max(sims) if sims else 0.0
    unique = [100.0 * (1 - max_sim[i]) for i in page]
    words = [page[i]["words"] for i in page]
    links = [page[i]["internal_links"] for i in page]
    thin = [page[i] for i in page if page[i]["words"] < 300]
    canon = [page[i]["canonical_self"] for i in page if page[i]["canonical"]]
    met = {
        "pages_sampled": n,
        "avg_words": round(sum(words) / len(words)),
        "min_words": min(words),
        "pages_under_300_words": len(thin),
        "avg_top_1_unique_pct": round(sum(unique) / len(unique)),
        "min_unique_pct": round(min(unique)),
        "cjk_samples": sum(1 for i in page if page[i]["cjk"] > 200),
        "canonical_self_pct": (round(100 * sum(canon) / len(canon))
                               if canon else None),
        "noindex_in_sample": sum(1 for i in page if page[i]["noindex"]),
        "avg_internal_links": round(sum(links) / len(links), 1),
        "thin_pages": [p["url"] for p in thin],
        "low_uniqueness_pages": sorted(
            ({page[i]["url"]: round(max_sim[i] * 100) for i in page}
             .items()), key=lambda kv: -kv[1])[:2],
        "nonlatin_caveat": [i for i in page if page[i]["cjk"] > 200],
    }
    return met, page, errors


# --- url rules and scores ----------------------------------------------


def url_rules(groups, host):
    """URL rule violations, count and duplicate slugs judged per pattern."""
    q = lng = up = 0
    total = 0
    dups = 0
    trailing = set()
    for pattern, items in groups.items():
        slugs = {}
        for url, _ in items:
            parsed = urllib.parse.urlsplit(url)
            path = parsed.path
            total += 1
            if parsed.query:
                q += 1
            if len(url) > 100:
                lng += 1
            if any(c.isupper() for c in path):
                up += 1
            trailing.add(path.endswith("/"))
            last = path.rstrip("/").split("/")[-1].lower()
            slugs[last] = slugs.get(last, 0) + 1
        dups += sum(1 for v in slugs.values() if v > 1)
    return {
        "patterns_checked": len(groups),
        "with_query": q, "over_100_chars": lng, "uppercase_path": up,
        "duplicate_slugs": dups, "mixed_trailing_slash": len(trailing) > 1,
        "pct": {"query": 100 * q / total if total else 0,
                "long": 100 * lng / total if total else 0,
                "upper": 100 * up / total if total else 0},
    }


def score(met, rules, sitemap_no_lastmod_pct):
    """Convert measured signals to the six-category scorecard."""
    url = max(0, 100 - round(30 * (rules["pct"]["query"] / 100)
                             + 30 * (rules["pct"]["long"] / 100)
                             + 20 * (rules["pct"]["upper"] / 100)
                             + 20 * (rules["duplicate_slugs"] > 0)))
    if met is None:
        return {"Data Quality": "n/a", "URL Structure": url,
                "Template Uniqueness": "n/a", "Thin Content Risk": "n/a",
                "Internal Linking": "n/a", "Index Management": "n/a"}
    uni = met["avg_top_1_unique_pct"]
    thin = 100
    under = met["pages_under_300_words"]
    if met["nonlatin_caveat"]:
        under = 0  # word gate not applicable to non-Latin token counts
    thin -= 35 * (under / met["pages_sampled"])
    if uni < 40:
        thin -= 45
    if uni < 30:
        thin -= 60
    thin = max(0, round(thin))
    expected = 4 * met["avg_words"] / 1000 if met["avg_words"] else 1
    link = min(100, round(100 * met["avg_internal_links"] / max(1, expected)))
    idx = 100
    if met["canonical_self_pct"] is not None:
        idx -= round(30 * (100 - met["canonical_self_pct"]) / 100)
    idx -= 30 * (met["noindex_in_sample"] / met["pages_sampled"])
    if sitemap_no_lastmod_pct is not None:
        idx -= round(30 * sitemap_no_lastmod_pct / 100)
    idx = max(0, round(idx))
    return {"Data Quality": "n/a", "URL Structure": url,
            "Template Uniqueness": uni, "Thin Content Risk": thin,
            "Internal Linking": link, "Index Management": idx}


# --- gates -------------------------------------------------------------

GATE_UNIQUE_FLAG = 40    # original quality gate: below 40% -> thin
GATE_UNIQUE_STOP = 30    # hardening recommendation: below 30% -> hard stop
GATE_WORDS = 300         # original quality gate: below 300 words -> review


def gates_of(pattern, count, met):
    """Apply the quality gates in the SKILL.md. Returns list of gate notes."""
    out = []
    if count >= 100:
        out.append("WARNING pattern size %d pages - confirm a staged rollout "
                   "and a review sample before relying on this pattern"
                   % count)
    if met is None:
        out.append("PATTERN NOT SAMPLED (fetch errors) - treat all gates "
                   "as unresolved")
        return out
    if count >= 100 and met["pages_sampled"] < 5:
        out.append("WARNING sample smaller than 5 - inconclusive gates")
    if met["nonlatin_caveat"]:
        out.append("NOTE %d sampled pages show non-Latin script (CJK/kana/"
                   "hangul): word counts are token counts and undercount. "
                   "Words gate skipped; similarity within this language "
                   "group is still comparable."
                   % len(met["nonlatin_caveat"]))
    elif met["pages_under_300_words"]:
        out.append("FLAG %d/%d sampled pages under %d words (review list: %s)"
                   % (met["pages_under_300_words"], met["pages_sampled"],
                      GATE_WORDS, "; ".join(met["thin_pages"][:2])))
    if met["avg_top_1_unique_pct"] < GATE_UNIQUE_STOP:
        out.append("HARD STOP avg top-1 unique content %.0f%% < %d%% - the "
                   "template reproduces itself; do not publish more of this "
                   "pattern"
                   % (met["avg_top_1_unique_pct"], GATE_UNIQUE_STOP))
    elif met["avg_top_1_unique_pct"] < GATE_UNIQUE_FLAG:
        out.append("FLAG avg top-1 unique content %.0f%% < %d%% - thin "
                   "content risk; differentiate pages before scaling"
                   % (met["avg_top_1_unique_pct"], GATE_UNIQUE_FLAG))
    else:
        out.append("PASS unique content %.0f%% - gate satisfied"
                   % met["avg_top_1_unique_pct"])
    return out


# --- report ------------------------------------------------------------

SEPARATOR = "-" * 62


def cross_pattern_duplicates(patterns):
    """Slug names appearing in more than one pattern (locale mirrors)."""
    seen = {}
    for pattern, items in patterns:
        if len(items) < 5:
            continue
        for url, _ in items:
            last = urllib.parse.urlsplit(url).path.rstrip("/")
            last = last.split("/")[-1].lower()
            seen.setdefault(last, set()).add(pattern)
    return sum(1 for v in seen.values() if len(v) > 1)


def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("sitemap", help="sitemap URL or local .xml path")
    ap.add_argument("--pattern", default=None,
                    help="only URLs containing this substring (e.g. /tools/)")
    ap.add_argument("--sample", type=int, default=8,
                    help="pages fetched per pattern (default 8)")
    ap.add_argument("--max-fetch", type=int, default=24,
                    help="hard cap on fetched pages (default 24)")
    ap.add_argument("--json", action="store_true")
    args = ap.parse_args(argv)

    errors = []
    try:
        entries = load_sitemap(args.sitemap)
    except Exception as exc:  # noqa: BLE001
        print("SITEMAP FETCH FAILED: %s: %s" % (type(exc).__name__, exc))
        return 2
    if not entries:
        print("SITEMAP EMPTY or unparseable: %s" % args.sitemap)
        return 2
    if args.pattern:
        filtered = [(u, m) for u, m in entries if args.pattern in u]
        if not filtered:
            print("NO URLS MATCH --pattern %s (sitemap has %d URLs)"
                  % (args.pattern, len(entries)))
            return 2
        entries = filtered
    total = len(entries)
    groups = group_patterns(entries)
    if not groups:
        print("NO REPEATING URL PATTERN FOUND in %d URLs - nothing here "
              "looks generated at scale" % total)
        return 2

    tested = []
    fetched = 0
    patterns = sorted(groups.items(), key=lambda kv: -len(kv[1]))
    for pattern, items in patterns:
        if fetched >= args.max_fetch or len(items) < 5:
            continue
        met, pages, errs = measure_pattern(pattern, items, args, errors)
        fetched += len(pages)
        if met is None and not pages:
            continue
        tested.append((pattern, items, met, pages))

    sitemap_no_lastmod = sum(1 for _, m in entries if not m)
    sitemap_pct = 100 * sitemap_no_lastmod / total if total else None
    rules = url_rules(groups, urllib.parse.urlsplit(entries[0][0]).netloc)

    if args.json:
        return json_report(entries, total, groups, tested, rules,
                           sitemap_pct, errors)

    print("Programmatic scan: %s" % args.sitemap)
    if args.pattern:
        print("  filter --pattern %s | URLs in scope: %d" % (args.pattern,
                                                             total))
    print("  sitemap: %d URLs | %d%% lack <lastmod>"
          % (total, round(sitemap_pct or 0)))
    print(SEPARATOR)
    print("URL pattern inventory (count >= 5):")
    sampled_keys = {p for p, _, _, _ in tested}
    for pattern, items in patterns:
        if len(items) < 5:
            continue
        print("  %-34s %5d pages%s" % (pattern, len(items),
                                       "   (sampled)" if pattern in
                                       sampled_keys else ""))
    print(SEPARATOR)
    per_pattern = {}
    for pattern, items, met, _ in tested:
        per_pattern[pattern] = met
        print("Pattern: %s (%d pages)" % (pattern, len(items)))
        if met["nonlatin_caveat"]:
            print("  non-Latin script detected; word counts are token "
                  "counts, see note below")
        print("  sampled:      %d pages evenly spaced over %d"
              % (met["pages_sampled"], len(items)))
        print("  avg body words:      %d   (min %d, pages < %d words: %d)"
              % (met["avg_words"], met["min_words"], GATE_WORDS,
                 met["pages_under_300_words"]))
        print("  avg top-1 unique:  %d%%   (min %d%%)"
              % (met["avg_top_1_unique_pct"], met["min_unique_pct"]))
        print("  canonical self-ref:  %s"
              % (str(met["canonical_self_pct"]) + "%"
                 if met["canonical_self_pct"] is not None else "no canonicals"))
        print("  noindex in sample:   %d" % met["noindex_in_sample"])
        print("  avg internal links:  %.1f" % met["avg_internal_links"])
        for gate in gates_of(pattern, len(items), met):
            print("  [GATE] %s" % gate)
        print()
    print(SEPARATOR)
    print("URL rule check over all %d URLs:" % total)
    print("  query strings: %d | >100 chars: %d | uppercase in path: %d"
          % (rules["with_query"], rules["over_100_chars"],
             rules["uppercase_path"]))
    print("  duplicate end-slugs across %d patterns (within-pattern): %d"
          % (rules["patterns_checked"], rules["duplicate_slugs"]))
    print("  mixed trailing slash: %s" % rules["mixed_trailing_slash"])
    cross = cross_pattern_duplicates(patterns)
    if cross:
        print("  cross-pattern duplicate slugs: %d (locale mirrors of the "
              "same page count here; judge, don't penalize)"
              % cross)
    print(SEPARATOR)
    print("Programmatic score (mechanics only; Data Quality is a human "
          "call - the data source lives behind the CMS):")
    for i, (pattern, items, met, _) in enumerate(tested):
        if i:
            print()
        sc = score(met, rules, sitemap_pct)
        print("  %s" % pattern)
        for k, v in sc.items():
            print("    %-22s %s" % (k, v))
        vals = [v for v in sc.values() if isinstance(v, (int, float))]
        if vals:
            print("    TOTAL (mean of measured): %d/100"
                  % round(sum(vals) / len(vals)))
    print(SEPARATOR)
    if errors:
        print("Fetch errors (%d):" % len(errors))
        for e in errors[:6]:
            print("  " + e)
    return 0


def json_report(entries, total, groups, tested, rules, sitemap_pct, errors):
    payload = {
        "sitemap_url": None,
        "total_urls": total,
        "url_rules": rules,
        "sitemap_no_lastmod_pct": sitemap_pct,
        "patterns": {},
        "fetch_errors": errors,
    }
    for pattern, items, met, _ in tested:
        payload["patterns"][pattern] = {
            "count": len(items),
            "gate": gates_of(pattern, len(items), met),
            "metrics": met,
            "score": score(met, rules, sitemap_pct),
        }
    json.dump(payload, sys.stdout, indent=1, ensure_ascii=False)
    return 0


if __name__ == "__main__":
    sys.exit(main())

~~~~

C. Save this as ~/.codex/skills/codex-seo-programmatic/fixtures/smoke.xml:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://auspia.ai/tools/landing-page-diagnostic</loc></url>
<url><loc>https://auspia.ai/tools/website-seo-score-checker</loc></url>
<url><loc>https://auspia.ai/tools/ai-search-visibility-checker</loc></url>
<url><loc>https://auspia.ai/tools/llms-txt-generator-checker</loc></url>
<url><loc>https://auspia.ai/tools/robots-txt-ai-crawler-checker</loc></url>
<url><loc>https://auspia.ai/tools/ai_overview</loc></url>
<url><loc>https://auspia.ai/tools/one-page-geo-citation-audit</loc></url>
<url><loc>https://auspia.ai/tools/best-tools-list-debiaser</loc></url>
<url><loc>https://auspia.ai/tools/google-index-checker</loc></url>
<url><loc>https://auspia.ai/tools/http-status-code-checker</loc></url>
<url><loc>https://auspia.ai/tools/on-page-seo-audit</loc></url>
<url><loc>https://auspia.ai/tools/googlebot-spider-simulator</loc></url>
</urlset>

Then run:
  mkdir -p ~/.codex/skills/codex-seo-programmatic/scripts
  mkdir -p ~/.codex/skills/codex-seo-programmatic/fixtures
  python3 --version
  # place the files as above, then:
  python3 ~/.codex/skills/codex-seo-programmatic/scripts/pp_scan.py \
    ~/.codex/skills/codex-seo-programmatic/fixtures/smoke.xml

Answers I need from you:
1. Did you place all three files? Report the byte counts of each.
2. Does the self-test output show one pattern, 12 pages, and URL Structure 100?
3. If the smoke test fails, report the exact error, do not try to fix it yourself.
4. Keep the skill in ~/.codex/skills/codex-seo-programmatic. Do not touch anything else in my skills directories.