How to Set Up Codex for XML Sitemap Checks (Full SKILL.md Included)

robots.txt declarations  ->  probe common paths  ->  parse each sitemap
                          ->  validate (XML, limits, lastmod, deprecations)
                          ->  sample target URL statuses
                          ->  prioritized issue list
                        (if sitemap index -> repeat the middle on each child)

---
name: codex-seo-sitemap
description: Use when the user asks about XML sitemaps, sitemap issues, missing sitemaps, sitemap validation, sitemap generation, or sitemap index files. Triggers on "sitemap", "check my sitemap", "generate sitemap", "sitemap not found". Works with any site; python3 only.
---
# Sitemap Analysis & Generation

Analyze existing XML sitemaps or generate new ones. Analysis mode validates
format, URL counts, sizes, lastmod quality, deprecations, and samples target
URLs. Generation mode applies industry quality gates before emitting XML.

## Run

```bash
python3 ~/.codex/skills/codex-seo-sitemap/scripts/sitemap_check.py <url> [--json]
```

The checker reads robots.txt `Sitemap:` lines, probes common paths when nothing
is declared, and reports per-sitemap: status, XML validity, URL count, size,
lastmod validity and distinctness, deprecated tags, and a status sample of
the first 20 target URLs. `--json` gives the full structure.

## Mode 1: Analyze Existing Sitemap

### Discovery rules

- Read robots.txt first; a declared sitemap is a candidate, not proof it works.
- If nothing is declared or a declared sitemap is stale/invalid, probe common
  paths: /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap1.xml,
  /wp-sitemap.xml, /seo-sitemap.xml.
- Report "no sitemap found" only after declared and common candidates are
  checked and all fail.

### Validation checks

- Valid XML parse
- Per-file limit: 50,000 URLs AND 50MB uncompressed, whichever is hit first
- All URLs return 200 (the checker samples the first 20; crawl the rest for a
  full audit)
- `<lastmod>` valid W3C Datetime and plausibly the real last content change;
  Google only honors it when consistently accurate, so warn on uniformly
  identical values or values newer than the page's real content
- No deprecated tags: `<priority>` and `<changefreq>` are ignored by Google
- Sitemap referenced in robots.txt
- Compare crawled pages vs sitemap and flag missing pages (in a full crawl)

### Quality signals

- Sitemap index file if >50k URLs
- Split by content type (pages, posts, images, videos)
- No non-canonical URLs
- No noindexed URLs
- No redirected URLs
- HTTPS URLs only

### Common issues

| Issue | Severity | Fix |
|-------|----------|-----|
| >50k URLs in single file | Critical | Split with sitemap index |
| >50MB uncompressed single file | Critical | Split with sitemap index |
| Non-200 URLs | High | Remove or fix broken URLs |
| Noindexed URLs included | High | Remove from sitemap |
| Redirected URLs included | Medium | Update to final URLs |
| All identical lastmod | Low | Use actual modification dates |
| Priority/changefreq used | Info | Can remove (ignored by Google) |

### Extension sitemaps (image / video / news)

Image: namespace google.com/schemas/sitemap-image/1.1. Only two valid tags
remain, `<image:image>` and `<image:loc>` (max 1,000 `<image:image>` per
`<url>`). `<image:caption>`, `<image:geo_location>`, `<image:title>`,
`<image:license>` were deprecated (2022) - info-level removable.

Video: required `<video:video>` with `<video:thumbnail_loc>`,
`<video:title>`, `<video:description>`, plus `<video:content_loc>` or
`<video:player_loc>`; mRSS also supported. Deprecated/removed tags
(`<video:category>`, `<video:gallery_loc>`, `<video:price>`, `<video:tvshow>`,
player autoplay/allow_embed) are info-level removable; recheck Google docs
before citing a removal date.

News: max 1,000 `<news:news>` per file (not 50,000); only articles from the
last 2 days; required `<news:publication>`/`<news:name>`/`<news:language>`/
`<news:publication_date>`/`<news:title>`; submitted through Search Console or
a robots/sitemap-index reference. When the `news:` namespace is present,
override the 50k check with the 1,000 cap.

## Mode 2: Generate New Sitemap

### Process

1. Ask for business type (or auto-detect from the existing site)
2. Load the industry architecture template
3. Plan structure interactively with the user
4. Apply quality gates:
   - WARNING at 30+ location pages (require 60%+ unique content)
   - HARD STOP at 50+ location pages (require justification)
5. Generate valid XML
6. Split at whichever comes first: 50,000 URLs or 50MB uncompressed, using a
   sitemap index
7. Generate STRUCTURE.md site-architecture documentation

### Safe programmatic pages (OK at scale)

- Integration pages with real setup docs
- Template/tool pages with downloadable content
- Glossary pages with 200+ word definitions
- Product pages with unique specs and reviews
- User profile pages with user-generated content

### Penalty risk (avoid at scale)

- Location pages with only the city name swapped
- "Best [tool] for [industry]" without industry-specific value
- "[Competitor] alternative" without real comparison data
- AI-generated pages without human review and unique value

## Sitemap format

Standard:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/page</loc>
    <lastmod>2026-02-07</lastmod>
  </url>
</urlset>
```

Index (for >50k URLs or split-by-type):

```xml
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages.xml</loc>
    <lastmod>2026-02-07</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-posts.xml</loc>
    <lastmod>2026-02-07</lastmod>
  </sitemap>
</sitemapindex>
```

## Reading the checker output

Each sitemap prints status, XML validity, root tag (urlset or sitemapindex),
URL count, size, lastmod statistics, deprecated-tag counts, and for a plain
urlset a sample of target URL statuses (ok / redirected / missing / other).
The flags out:

| Output | Meaning |
|--------|---------|
| `urls > 50,000 limit` | split into an index |
| `size > 50MB` | split into an index |
| `lastmod distinct: 1` | all identical; likely boilerplate, Google ignores it |
| `deprecated tags` | priority/changefreq present, removable |
| `sample_redirected/missing` | stale sitemap entries |
| `index file with N children` | re-check each child sitemap |

## Errors

| Scenario | Action |
|----------|--------|
| URL unreachable | Report status code; check if the site is live |
| No sitemap found | Report that, after robots.txt and common paths, nothing was found |
| Invalid XML | Report the parse error and line context; fetch fresh |
| Sitemap served with HTML status 200 | Likely a 404 page; treat as invalid, re-check the sitemap URL |
| HEAD blocked by WAF | The checker falls back to ranged GET; sizes may be approximate |
| Rate limiting | Back off; report partial results with retry timing |

## Output

### For analysis

- A report listing each sitemap, its health, and the prioritized issues
- Issues with severity from the table above
- Recommendations per issue

### For generation

- sitemap.xml (or split files plus index)
- STRUCTURE.md: site architecture documentation
- URL count and organization summary

#!/usr/bin/env python3
"""Check a site's XML sitemaps: discovery, XML validity, size, URL count,
deprecated tags, lastmod format, and a sample of target URL statuses.
Standard library only. Usage:
  python3 sitemap_check.py <url> [--json]
"""
import json
import re
import sys
import urllib.error
import urllib.request
import urllib.parse
import xml.etree.ElementTree as ET

TIMEOUT = 12
MAX_URL_COUNT = 50000
MAX_SIZE_BYTES = 50 * 1024 * 1024
SAMPLE = 20
UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
      "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 "
      "codex-seo-sitemap/1.0")
COMMON_PATHS = [
    "/sitemap.xml", "/sitemap_index.xml", "/sitemap-index.xml",
    "/sitemap1.xml", "/wp-sitemap.xml", "/seo-sitemap.xml",
]
DATETIME_RE = re.compile(
    r"^\d{4}-\d{2}-\d{2}(T\d{2}:\d{2}:\d{2}(\.\d+)?"
    r"(Z|[+-]\d{2}:\d{2})?)?\d*$")
DEPRECATED_TAGS = ("priority", "changefreq")


def fetch(url, method="GET"):
    req = urllib.request.Request(url, method=method, headers={"User-Agent": UA})
    try:
        resp = urllib.request.urlopen(req, timeout=TIMEOUT)
        data = resp.read()
        return resp.status, resp.geturl(), data, resp.headers
    except urllib.error.HTTPError as e:
        return e.code, url, b"", e.headers
    except Exception as e:
        return "ERR:" + str(e)[:100], url, b"", {}


def local(tag):
    return tag.split("}")[-1]


def status_of(url):
    """Work out a target URL's status; fall back to ranged GET when the
    server refuses HEAD (405/403/501) since sitemap targets often use WAFs."""
    status, _, _, _ = fetch(url, method="HEAD")
    if isinstance(status, int) and status in (405, 403, 501, 400, 404):
        if status == 404:
            return 404
        req = urllib.request.Request(url, headers={
            "User-Agent": UA, "Range": "bytes=0-99"})
        try:
            resp = urllib.request.urlopen(req, timeout=TIMEOUT)
            return resp.status
        except urllib.error.HTTPError as e:
            return e.code
        except Exception:
            return status
    return status


def parse_robots(base):
    """Return declared sitemap URLs from robots.txt (bounded: one host only)."""
    url = urllib.parse.urljoin(base, "/robots.txt")
    status, _, data, _ = fetch(url)
    if status != 200 or not data:
        return []
    out = []
    for line in data.decode("utf-8", "replace").splitlines():
        m = re.match(r"^\s*Sitemap:\s*(\S+)", line, re.I)
        if m:
            out.append(m.group(1))
    return out


def probe_common(base):
    out = []
    for p in COMMON_PATHS:
        url = urllib.parse.urljoin(base, p)
        status, _, data, _ = fetch(url)
        if status == 200:
            # sitemaps return application/xml or text/xml; also accept .xml
            first = data[:256].decode("utf-8", "replace")
            if "<urlset" in first or "<sitemapindex" in first or \
                    "<urlset" in first.lower():
                out.append(url)
    return out


def analyze_sitemap(url, depth=0):
    info = {"url": url, "depth": depth, "children": []}
    status, final, data, headers = fetch(url)
    info["status"] = status
    info["final_url"] = final
    if status not in (200, "200"):
        info["error"] = "fetch failed"
        return info
    info["size_bytes"] = len(data)
    info["size_mb"] = round(len(data) / 1024 / 1024, 2)
    try:
        root = ET.fromstring(data.decode("utf-8", "replace"))
    except ET.ParseError as e:
        info["ok_xml"] = False
        info["xml_error"] = str(e)[:160]
        return info
    info["ok_xml"] = True
    info["root_tag"] = local(root.tag)

    locs, lastmods, dcounts = [], [], {t: 0 for t in DEPRECATED_TAGS}
    for el in root.iter():
        t = local(el.tag)
        if t == "loc":
            locs.append((el.text or "").strip())
        elif t == "lastmod":
            lastmods.append((el.text or "").strip())
        elif t in DEPRECATED_TAGS:
            dcounts[t] += 1
    info["loc_count"] = len(locs)
    info["lastmod_count"] = len(lastmods)
    info["deprecated"] = dcounts
    info["url_count"] = len(locs)

    valid_lm = sum(1 for x in lastmods if DATETIME_RE.match(x))
    info["lastmod_valid"] = valid_lm
    info["lastmod_invalid"] = len(lastmods) - valid_lm
    uniq = [x for x in lastmods if x]
    info["lastmod_sample"] = uniq[:5]
    info["lastmod_distinct"] = len(set(uniq))

    if info["root_tag"] == "sitemapindex":
        info["is_index"] = True
        info["child_count"] = sum(1 for el in root.iter() if local(el.tag) == "sitemap")
        for child in locs[:50]:
            info["children"].append({"url": child, "checked": False})
    else:
        info["is_index"] = False
        # sample target URL statuses
        sample = []
        non200 = redirected = missing = 0
        for loc in locs[:SAMPLE]:
            s = status_of(loc)
            sample.append({"loc": loc, "status": s})
            try:
                code = int(s)
            except (TypeError, ValueError):
                code = 0
            if code and 300 <= code < 400:
                redirected += 1
            elif code == 404:
                missing += 1
            elif code == 200:
                pass
            else:
                non200 += 1
        info["sample_http"] = sample
        info["sample_ok"] = sum(1 for x in sample
                                if isinstance(x["status"], int) and x["status"] == 200)
        info["sample_redirected"] = redirected
        info["sample_missing"] = missing
        info["sample_other"] = non200
    return info


def main():
    url = sys.argv[1] if len(sys.argv) > 1 else ""
    if not url:
        print("usage: python3 sitemap_check.py <url> [--json]")
        sys.exit(1)
    want_json = "--json" in sys.argv
    if not url.startswith("http"):
        url = "https://" + url
    base = url
    m = re.match(r"^(https?://[^/]+)/", url)
    if m:
        base = m.group(1) + "/"

    declared = parse_robots(base)
    found = declared or probe_common(base)
    result = {"base": base, "declared": declared, "found": found,
              "sitemaps": []}
    if declared:
        for s in declared:
            result["sitemaps"].append(analyze_sitemap(s))
    elif found:
        for s in found[:5]:
            result["sitemaps"].append(analyze_sitemap(s))
    else:
        result["sitemaps"] = []

    if want_json:
        print(json.dumps(result, indent=2, default=str))
        return

    print("sitemap_check %s" % result["base"])
    print("  declared in robots.txt: %d   discovered: %d" % (
        len(declared), len(found)))
    if not result["sitemaps"]:
        print("  no sitemap found (checked robots.txt + common paths)")
        return
    for sm in result["sitemaps"]:
        print("  sitemap %s status %s" % (sm["url"], sm["status"]))
        if sm.get("error"):
            print("    error: %s" % sm["error"])
            continue
        print("    xml ok: %s  tag: %s  urls: %d  size: %sMB" % (
            sm.get("ok_xml"), sm.get("root_tag"), sm.get("url_count"),
            sm.get("size_mb")))
        if sm.get("lastmod_count"):
            print("    lastmod: %d present, %d valid, %d invalid, distinct: %d" % (
                sm["lastmod_count"], sm["lastmod_valid"], sm["lastmod_invalid"],
                sm["lastmod_distinct"]))
        if any(sm.get("deprecated", {}).values()):
            print("    deprecated tags: %s" % sm["deprecated"])
        if sm.get("is_index"):
            print("    index file with %d child sitemaps (sub-check the top ones)" % sm["child_count"])
        else:
            s = sm["sample_http"]
            print("    sample %d URLs: %d ok, %d redirected, %d missing, %d other" % (
                len(s), sm.get("sample_ok", 0), sm.get("sample_redirected", 0),
                sm.get("sample_missing", 0), sm.get("sample_other", 0)))
            if sm.get("url_count", 0) > MAX_URL_COUNT:
                print("    CRITICAL: %d URLs > 50,000 limit" % sm["url_count"])
            if sm.get("size_mb", 0) > 50:
                print("    CRITICAL: %sMB > 50MB uncompressed limit" % sm["size_mb"])
        if sm["status"] != 200:
            print("    not 200: report as broken sitemap reference")


if __name__ == "__main__":
    main()

mkdir -p ~/.codex/skills/codex-seo-sitemap/scripts
# save the two files above at:
#   ~/.codex/skills/codex-seo-sitemap/SKILL.md
#   ~/.codex/skills/codex-seo-sitemap/scripts/sitemap_check.py
python3 ~/.codex/skills/codex-seo-sitemap/scripts/sitemap_check.py https://example.com

python3 ~/.codex/skills/codex-seo-sitemap/scripts/sitemap_check.py https://yoursite.com

sitemap_check https://auspia.ai
  declared in robots.txt: 2   discovered: 2
  sitemap https://auspia.ai/sitemap.xml status 200
    xml ok: True  tag: urlset  urls: 966  size: 0.16MB
    lastmod: 966 present, 966 valid, 0 invalid, distinct: 1
    deprecated tags: {'priority': 966, 'changefreq': 966}
    sample 20 URLs: 20 ok, 0 redirected, 0 missing, 0 other
  sitemap https://auspia.ai/blog/blog-index.xml status 200
    xml ok: True  tag: sitemapindex  urls: 3  size: 0.0MB
    index file with 3 child sitemaps (sub-check the top ones)

sitemap_check https://felo.ai
  declared in robots.txt: 0   discovered: 0
  no sitemap found (checked robots.txt + common paths)

Read this message. Create the skill below at
~/.codex/skills/codex-seo-sitemap/ by following the two code blocks in it:
1. The markdown block is SKILL.md.
2. The python block is scripts/sitemap_check.py.
Then run python3 ~/.codex/skills/codex-seo-sitemap/scripts/sitemap_check.py
https://example.com as a self-test and report the output (or the error) back
to me, plus what you would check first on my own site.
Do not run any other commands. Do not modify any other files.