How to Set Up Codex for Topic Clustering (Full SKILL.md Included)

seo audit                        x seo audit checklist               0 shared  (separate)

SERP overlap cluster scan serp.tsv
  10 keywords / 45 pairwise comparisons
  ubiquitous domains (7 of 10 SERPs): www.semrush.com (7)
  overlap distribution:
    seo audit                        x technical seo audit               5 shared  (same cluster)
    keyword research                 x keyword research tools            5 shared  (same cluster)
    seo tools                        x best seo tools                    4 shared  (same cluster)
    seo audit checklist              x technical seo audit checklist     3 shared  (interlink)
    technical seo                    x technical seo audit               1 shared  (separate)
    seo tools                        x seo checklist                     1 shared  (separate)
    best seo tools                   x seo checklist                     1 shared  (separate)
    seo audit                        x seo audit checklist               0 shared  (separate)
    seo audit                        x technical seo                     0 shared  (separate)
    seo audit                        x seo tools                         0 shared  (separate)
    seo audit                        x best seo tools                    0 shared  (separate)
    seo audit                        x keyword research                  0 shared  (separate)
    seo audit                        x keyword research tools            0 shared  (separate)
    seo audit                        x seo checklist                     0 shared  (separate)
    seo audit                        x technical seo audit checklist     0 shared  (separate)
    seo audit checklist              x technical seo                     0 shared  (separate)
    seo audit checklist              x technical seo audit               0 shared  (separate)
    seo audit checklist              x seo tools                         0 shared  (separate)
    seo audit checklist              x best seo tools                    0 shared  (separate)
    seo audit checklist              x keyword research                  0 shared  (separate)
    seo audit checklist              x keyword research tools            0 shared  (separate)
    seo audit checklist              x seo checklist                     0 shared  (separate)
    technical seo                    x seo tools                         0 shared  (separate)
    technical seo                    x best seo tools                    0 shared  (separate)
    technical seo                    x keyword research                  0 shared  (separate)
    technical seo                    x keyword research tools            0 shared  (separate)
    technical seo                    x seo checklist                     0 shared  (separate)
    technical seo                    x technical seo audit checklist     0 shared  (separate)
    technical seo audit              x seo tools                         0 shared  (separate)
    technical seo audit              x best seo tools                    0 shared  (separate)
    technical seo audit              x keyword research                  0 shared  (separate)
    technical seo audit              x keyword research tools            0 shared  (separate)
    technical seo audit              x seo checklist                     0 shared  (separate)
    technical seo audit              x technical seo audit checklist     0 shared  (separate)
    seo tools                        x keyword research                  0 shared  (separate)
    seo tools                        x keyword research tools            0 shared  (separate)
    seo tools                        x technical seo audit checklist     0 shared  (separate)
    best seo tools                   x keyword research                  0 shared  (separate)
    best seo tools                   x keyword research tools            0 shared  (separate)
    best seo tools                   x technical seo audit checklist     0 shared  (separate)
    keyword research                 x seo checklist                     0 shared  (separate)
    keyword research                 x technical seo audit checklist     0 shared  (separate)
    keyword research tools           x seo checklist                     0 shared  (separate)
    keyword research tools           x technical seo audit checklist     0 shared  (separate)
    seo checklist                    x technical seo audit checklist     0 shared  (separate)
  verdict counts: same cluster=3, interlink=1, separate=41
  wrote cluster-plan.md
  group 1 (2 keywords, ~1 posts): seo audit, technical seo audit
  group 2 (2 keywords, ~1 posts): seo tools, best seo tools
  group 3 (2 keywords, ~1 posts): keyword research, keyword research tools
  group 4 (1 keywords, ~1 posts): seo audit checklist
  group 5 (1 keywords, ~1 posts): technical seo
  group 6 (1 keywords, ~1 posts): seo checklist
  group 7 (1 keywords, ~1 posts): technical seo audit checklist

mkdir -p ~/.codex/skills/codex-seo-cluster/scripts

---
name: codex-seo-cluster
description: Use when the user asks about topic clustering, content clustering, semantic clustering, keyword grouping, pillar pages, hub and spoke architecture, content architecture, cluster plans, related keyword planning, or cannibalization between pages. Groups keywords by which Google results they actually share (SERP overlap), not by text similarity, and outputs a hub-and-spoke cluster plan with an internal link matrix.
---
# Topic Clustering from SERP Overlap

Cluster keywords by how Google actually ranks them. Two keywords that share
the same top-10 organic results should target the same page; two keywords
with different results need separate pages. Google is the judge, not word
similarity, stemming, or related-search suggestions.

## 1. Expand the seed

30-50 candidate keywords, any mixture of:

- related searches and "people also search for" for the seed
- People Also Ask questions from the seed's SERP
- long-tail modifiers: best, how to, vs, for beginners, tools, examples,
  guide, template, mistakes, checklist
- question stems: who/what/when/where/why/how + seed
- commercial modifiers: pricing, review, alternative, comparison, free, top

Normalize (lowercase, strip articles), deduplicate. Under 30 variants: run
a second pass seeded by the top PAA questions.

## 2. Collect SERPs

For each keyword, gather the organic results. Any of:

1. The orchestration agent's own web search (treat its result list as the
   SERP; run each keyword once; note the counts per row).
2. `serp_google_organic_live` style API exports pasted into a file.
3. A file the user hands you (`serp.tsv`): `keyword<TAB>url` per row, or
   `keyword,url1,url2,...` per row.

Use only organic URLs: drop ads, featured snippets, PAA, knowledge panels,
maps, and links outside the top 10. Normalize each URL: remove the protocol,
the trailing slash, and the query string; lowercase the domain.

## 3. Score the overlap

```bash
python3 ~/.codex/skills/codex-seo-cluster/scripts/serp_cluster.py <serp.tsv>
python3 ~/.codex/skills/codex-seo-cluster/scripts/serp_cluster.py <serp.tsv> --json
```

The script computes the shared-URL count for every keyword pair and applies
the thresholds:

| Shared results | Relationship | Action |
|----------------|-------------|--------|
| 7-10 | Same post | Merge into one target page; use the higher-volume keyword as primary |
| 4-6 | Same cluster | Same spoke cluster; separate posts if the volume gap is large |
| 2-3 | Interlink | Adjacent clusters; add cross-links between those posts |
| 0-1 | Separate | Different clusters, or exclude from this pillar |

**3-4 ties** (the boundary zone): fall back to domain overlap (same
domains, different pages = closer), intent alignment, and volume ratio
(one keyword at 10x+ volume probably earns its own post). When in doubt,
keep keywords in the same cluster with separate posts.

## 4. Classify intent

| Intent | Signals | Include? |
|--------|---------|----------|
| Informational | how, what, why, guide, tutorial, learn | Yes |
| Commercial | best, top, review, comparison, vs, alternative | Yes |
| Transactional | buy, price, discount, coupon, order, sign up | Yes |
| Navigational | brand names, specific products, login | No - drop from clustering |

Mixed intent: classify by the dominant intent; flag ambiguous ones for a
human review.

## 5. Hub-and-spoke architecture

1. **Pillar keyword**: highest volume, broadest intent, most overlap with
   the other keywords.
2. **Clusters per pillar**: 2-5 subtopic areas; 2-4 posts per cluster;
   5-21 posts total.
3. **Template by intent** (avoid duplicating templates inside a cluster
   unless the SERPs justify it):

   | Intent pattern | Template |
   |----------------|----------|
   | Informational broad | ultimate-guide |
   | Informational how | how-to |
   | Informational list | listicle |
   | Informational concept | explainer |
   | Commercial compare | comparison |
   | Commercial evaluate | review |
   | Commercial rank | best-of |
   | Transactional | landing-page |

4. **Word counts**: pillar 2,500-4,000; spokes 1,200-1,800.
5. **Cannibalization**: no two posts share a primary keyword. If overlap is
   7+, merge into one post targeting both. After planning, list all primary
   keywords and scan for near-duplicates.

## 6. Link matrix

| Link type | Direction | Requirement |
|-----------|-----------|-------------|
| Spoke to pillar | spoke -> pillar | Mandatory, every spoke |
| Pillar to spoke | pillar -> spoke | Mandatory, every spoke |
| Same-cluster spoke | spoke <-> spoke | 2-3 per post, contextual anchors |
| Cross-cluster | spoke -> spoke | 0-1 per post, only with a real bridge |

Every post needs at least 3 incoming links; no orphan pages (everyone
reachable from the pillar in <= 2 hops). Anchor text = target keyword or a
close variant, never "click here"; place links in body copy. The script
writes `cluster-plan.md` with the group list, so expand it into the
full plan: per-post template, word count, URL, status, and the link list.

## 7. Output contract

Write `cluster-plan.md` with: seed keyword, date, pillar (title, keyword,
template, word count, URL, status), clusters (name + posts with the same
fields), the link list (from/to/type/anchor), the overlap matrix summary,
and a limitations paragraph naming anything you could not verify (missing
SERP data, volume estimates, weak overlap). State any step that was
performed with assumptions.

## Anti-patterns

1. Never cluster by text similarity alone: "seo audit" and "seo audit
   checklist" can share zero results (tool SERPs vs tutorial SERPs).
2. Never group by stemming: "run" and "running" can target different intents.
3. Never assume related searches belong together; verify with overlap.
4. Never ignore SERP feature differences: a local pack or featured snippet
   on one keyword and not the other argues for different content types.
5. Never treat all domains equally: Wikipedia and Reddit sit in many SERPs;
   flag the most common domains (the script reports them) and judge whether
   to weight or drop them.

## Cost and effort optimization

Full pairwise is N*(N-1)/2 comparisons (40 keywords = 780). Pre-group by
intent and head term, compare across groups only at boundary keywords, and
cache every SERP you fetch - reuse keyword A's results for every pair that
includes A. If an API is involved, run a cost check before each batch; if
the budget check says blocked, fall back to the agent's own web search.

## Errors

| Scenario | Action |
|----------|--------|
| File has fewer than 2 keywords | report DATA PROBLEM; ask for more keywords |
| Keyword with zero URLs | skip it and note the skip in the limitations paragraph |
| SERP data unavailable (search + API both down) | retry once after 30s, then intent-only clustering with an explicit warning |
| Overlap everywhere 0 | that is a real result, not a failure: the keywords are separate pages |
| Ubiquitous domains skew the matrix | quantify them, then rerun without them and compare verdicts |
| Conflicting verdicts after tie-break | prefer cohesion: same cluster, separate posts |

## Security

No credentials are stored or transmitted. The script reads only the local
SERP file. If you fetch URLs, use the agent's own fetch tooling - never raw
shell curl against user-supplied hosts.

#!/usr/bin/env python3
"""SERP-overlap clustering: decide content architecture from what Google
actually ranks. Takes one SERP file (keyword + top-10 organic URLs per
keyword) and returns the overlap matrix with threshold verdicts, cluster
groups (connected components at overlap >= 4), and merge warnings
(overlap >= 7 means one post). Standard library only. Usage:
  python3 serp_cluster.py <serp.tsv> [--json]
Input file: one line per <keyword> TAB <url>, or one line per <keyword>
with comma-separated URLs after it. URLs are normalized (protocol,
trailing slash, query string removed) before comparison.
"""
import csv
import json
import sys
from collections import defaultdict


def url_path(url):
    u = url.strip().split("#")[0].split("?")[0]
    if "://" in u:
        u = u.split("://", 1)[1]
    return u.rstrip("/").lower()


def load(path):
    """Return {keyword: [normalized url paths]}, preserving order."""
    data = defaultdict(list)
    with open(path, newline="", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            tokens = line.split("\t")
            if len(tokens) == 2:
                kw, url = tokens
                data[kw].append(url_path(url))
            else:
                parts = [p.strip() for p in line.split(",")]
                kw = parts[0]
                for p in parts[1:]:
                    if p:
                        data[kw].append(url_path(p))
    return {k: list(dict.fromkeys(v)) for k, v in data.items()}


def main():
    args = [a for a in sys.argv[1:] if not a.startswith("--")]
    if not args:
        print("serp_cluster.py - SERP overlap clustering (no API key needed)")
        print("  python3 serp_cluster.py <serp.tsv> [--json]")
        return
    data = load(args[0])
    kws = list(data)
    if len(kws) < 2:
        print("DATA PROBLEM: need at least 2 keywords with URLs")
        return

    # Ubiquitous domains: domains present in more than half of the SERPs are
    # noise (Wikipedia, Reddit, ...); report them so the user can drop them.
    per_domain = defaultdict(int)
    for urls in data.values():
        for u in urls:
            per_domain[u.split("/", 1)[0]] += 1
    n = len(kws)
    common = sorted(
        [(d, c) for d, c in per_domain.items() if c * 2 > n],
        key=lambda t: -t[1])

    # Pairwise overlap on normalized URL sets.
    pairs = []
    for i in range(len(kws)):
        for j in range(i + 1, len(kws)):
            a, b = data[kws[i]], data[kws[j]]
            shared = sorted(set(a) & set(b), key=lambda x: (x, len(x)))
            pairs.append((shared, i, j))
    pairs.sort(key=lambda t: (-len(t[0]), t[1], t[2]))

    def verdict(score):
        if score >= 7:
            return "same post"
        if score >= 4:
            return "same cluster"
        if score >= 2:
            return "interlink"
        return "separate"

    # Cluster groups: keywords are vertices; an edge is overlap >= 4. The
    # connected components are the cluster suggestion.
    parent = {k: k for k in kws}

    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x

    for s, i, j in pairs:
        if len(s) >= 4:
            ra, rb = find(kws[i]), find(kws[j])
            if ra != rb:
                parent[ra] = rb

    groups = defaultdict(list)
    for k in kws:
        groups[find(k)].append(k)
    cluster_groups = sorted(
        list(groups.values()), key=lambda g: -len(g))

    counts = defaultdict(int)
    for s, i, j in pairs:
        counts[verdict(len(s))] += 1
    merges = [p for p in pairs if len(p[0]) >= 7]

    out = {
        "file": args[0],
        "keywords": len(kws),
        "pairs": len(pairs),
        "ubiquitous_domains": [{"domain": d, "serps": c} for d, c in common],
        "pairs_by_verdict": dict(counts),
        "merge_warnings": [
            {"a": kws[i], "b": kws[j], "shared": len(s),
             "shared_urls": s[:8]}
            for s, i, j in merges],
        "cluster_groups": [
            {"keywords": g,
             "posts_estimated": max(1, len(g) // 2)}
            for g in cluster_groups],
    }
    if "--json" in sys.argv:
        print(json.dumps(out, indent=2))
        return

    print("SERP overlap cluster scan %s" % args[0])
    print("  %d keywords / %d pairwise comparisons"
          % (len(kws), len(pairs)))
    if common:
        print("  ubiquitous domains (%s of %d SERPs): %s"
              % (common[0][1], n, ", ".join(
                  "%s (%d)" % (d, c) for d, c in common)))
    print("  overlap distribution:")
    for s, i, j in pairs:
        print("    %-32s x %-32s %2d shared  (%s)"
              % (kws[i][:32], kws[j][:32], len(s), verdict(len(s))))
    print("  verdict counts: %s"
          % ", ".join("%s=%d" % (v, counts[v])
                      for v in ("same post", "same cluster",
                                "interlink", "separate") if counts[v]))
    out_dir = "cluster-plan.md"
    with open(out_dir, "w", encoding="utf-8") as f:
        f.write("# Cluster plan (generated by serp_cluster.py)\n\n")
        f.write("- keywords: %d, pairs: %d\n\n" % (len(kws), len(pairs)))
        f.write("## Suggested groups (overlap >= 4)\n\n")
        for i, g in enumerate(cluster_groups):
            f.write("%d. %s\n\n" % (i + 1, " / ".join(g)))
        f.write("## Merge warnings (overlap >= 7 -> one post)\n\n")
        if merges:
            for s, i, j in merges:
                f.write("- %s + %s (%d shared)\n"
                        % (kws[i], kws[j], len(s)))
        else:
            f.write("- none\n")
    print("  wrote %s" % out_dir)
    for i, g in enumerate(cluster_groups):
        print("  group %d (%d keywords, ~%d posts): %s"
              % (i + 1, len(g), max(1, len(g) // 2), ", ".join(g)))


if __name__ == "__main__":
    main()

cd ~/Desktop  # or anywhere you keep working files
# one line per keyword-TAB-url; the example file that produced the run above:
python3 ~/.codex/skills/codex-seo-cluster/scripts/serp_cluster.py serp.tsv
python3 ~/.codex/skills/codex-seo-cluster/scripts/serp_cluster.py --json serp.tsv > matrix.json

# serp-10kw.tsv - the file that produced the run above (89 rows, 10 keywords, US desktop capture)
seo audit	https://www.seoptimer.com/
seo audit	https://www.semrush.com/siteaudit/
seo audit	https://app.neilpatel.com/en/site-audit
seo audit	https://seomator.com/free-seo-audit-tool
seo audit	https://www.siteimprove.com/glossary/seo-website-audit/
seo audit	https://www.ama.org/marketing-news/how-to-do-an-seo-audit/
seo audit	https://seositecheckup.com/
seo audit	https://rankmath.com/tools/seo-analyzer/?
seo audit checklist	https://www.notion.com/templates/the-ultimate-seo-audit-checklist-for-2025?srsltid=AfmBOoqFvml_YBIAQ7npELhqt3UBuq-YcFXN3BNaAYHbtFpi7HBnurLM
seo audit checklist	https://backlinko.com/tools/seo-checker
seo audit checklist	https://moz.com/seo-audit-checklist
seo audit checklist	https://shortlist.io/blog/technical-seo-audit-checklist/
seo audit checklist	https://offers.hubspot.com/how-to-run-seo-audit
seo audit checklist	https://www.wordstream.com/blog/ws/2022/01/24/website-audit-checklist
seo audit checklist	https://serpstat.com/blog/seo-audit-guide/
seo audit checklist	https://www.spyfu.com/blog/seo-audit/
seo audit checklist	https://www.quora.com/How-do-I-make-a-perfect-SEO-Audit-Checklist
technical seo	https://technicalseo.com/
technical seo	https://www.semrush.com/blog/technical-seo/
technical seo	https://developers.google.com/search/docs/fundamentals/get-started
technical seo	https://www.mtu.edu/umc/services/websites/seo/technical-seo/
technical seo	https://www.geeksforgeeks.org/blogs/what-is-technical-seo/
technical seo	https://wpseoai.com/blog/what-is-technical-seo-with-10-examples/
technical seo	https://nav43.com/blog/what-is-technical-seo/
technical seo	https://ahrefs.com/seo/technical-seo
technical seo	https://searchatlas.com/blog/technical-seo/
technical seo audit	https://technicalseo.com/
technical seo audit	https://www.semrush.com/siteaudit/
technical seo audit	https://seomator.com/free-seo-audit-tool
technical seo audit	https://app.neilpatel.com/en/site-audit
technical seo audit	https://www.siteimprove.com/platform/seo/technical-seo-auditing-tools/
technical seo audit	https://www.ama.org/marketing-news/how-to-do-an-seo-audit/
technical seo audit	https://seositecheckup.com/
technical seo audit	https://www.seobility.net/en/website-audit/
technical seo audit	https://scandiweb.com/blog/seo-technical-audit-why-and-when-you-need-to-perform-it/
seo tools	https://smallseotools.com/
seo tools	https://technicalseo.com/tools/
seo tools	https://moz.com/free-seo-tools
seo tools	https://zapier.com/blog/best-seo-tools/
seo tools	https://ahrefs.com/free-seo-tools
seo tools	https://www.marketermilk.com/blog/best-seo-tools
seo tools	https://www.youtube.com/watch?v=VOLpViuXh5U
seo tools	https://www.techradar.com/news/best-seo-tool
seo tools	https://morningscore.io/best-seo-tools/
best seo tools	https://zapier.com/blog/best-seo-tools/
best seo tools	https://technologyadvice.com/blog/sales/best-seo-tools/
best seo tools	https://www.reddit.com/r/SEO/comments/1269zj7/whats_the_best_seo_tools_right_now_in_the_market/
best seo tools	https://www.marketermilk.com/blog/best-seo-tools
best seo tools	https://www.g2.com/categories/seo-tools
best seo tools	https://www.visionefx.net/what-are-the-best-seo-tools/
best seo tools	https://morningscore.io/best-seo-tools/
best seo tools	https://userp.io/tools-and-software/top-seo-tools/
best seo tools	https://www.youtube.com/watch?v=B58oCI0XVf0
keyword research	https://www.semrush.com/analytics/keywordmagic/
keyword research	https://business.google.com/us/ad-tools/keyword-planner/
keyword research	https://backlinko.com/tools/keyword
keyword research	https://zapier.com/blog/best-keyword-research-tool/
keyword research	https://ahrefs.com/seo/keyword-research
keyword research	https://cals.las.iastate.edu/post/basics-keyword-research
keyword research	https://www.conductor.com/academy/keyword-research/
keyword research	https://www.wordtracker.com/
keyword research	https://keywordtool.io/
keyword research tools	https://zapier.com/blog/best-keyword-research-tool/
keyword research tools	https://www.semrush.com/analytics/keywordmagic/
keyword research tools	https://business.google.com/us/ad-tools/keyword-planner/
keyword research tools	https://app.neilpatel.com/en/ai-keyword-overview
keyword research tools	https://www.reddit.com/r/Entrepreneur/comments/6vqxrn/heres_a_list_of_14_free_keyword_research_tools/
keyword research tools	https://ahrefs.com/keyword-generator
keyword research tools	https://keywordtool.io/
keyword research tools	https://www.wordtracker.com/
keyword research tools	https://community.hubspot.com/t/best-tools-for-keyword-research-for-seo-newbie/45408
seo checklist	https://www.semrush.com/blog/seo-checklist/
seo checklist	https://www.reddit.com/r/SEO/comments/rpvl7l/here_is_my_complete_seo_checklist_that_i_use_to/
seo checklist	https://university.webflow.com/resources/seo-checklist
seo checklist	https://ahrefs.com/blog/seo-checklist/
seo checklist	https://backlinko.com/seo-checklist
seo checklist	https://www.seoclarity.net/blog/seo-checklist
seo checklist	https://www.marketveep.com/blog/your-monthly-seo-checklist-6-essential-tasks-for-success
seo checklist	https://www.youtube.com/watch?v=vaxXNpxNWl0
seo checklist	https://aioseo.com/seo-checklist/
technical seo audit checklist	https://sitebulb.com/technical-seo/audit-checklist/
technical seo audit checklist	https://www.semrush.com/blog/technical-seo-checklist/
technical seo audit checklist	https://moz.com/seo-audit-checklist
technical seo audit checklist	https://www.notion.com/templates/technical-seo-audit-checklist?srsltid=AfmBOoq5O3QhJFmp2c4c9pp04ScS4QU4R_uOxY6l0AAdY4R33iWCE2Vb
technical seo audit checklist	https://shortlist.io/blog/technical-seo-audit-checklist/
technical seo audit checklist	https://serpstat.com/blog/seo-audit-guide/
technical seo audit checklist	https://www.pixelcrayons.com/blog/digital-marketing/seo-audit-checklist-to-use/
technical seo audit checklist	https://dashthis.com/blog/technical-seo-checklist/
technical seo audit checklist	https://wellows.com/blog/technical-seo-checklist-for-agencies/

SERP overlap cluster scan serp-10kw.tsv
  10 keywords / 45 pairwise comparisons
  ubiquitous domains (7 of 10 SERPs): www.semrush.com (7)
  ... (45 pairs, same as above)
  verdict counts: same cluster=3, interlink=1, separate=41
  wrote cluster-plan.md
  group 1 (2 keywords, ~1 posts): seo audit, technical seo audit
  group 2 (2 keywords, ~1 posts): seo tools, best seo tools
  group 3 (2 keywords, ~1 posts): keyword research, keyword research tools
  group 4 (1 keywords, ~1 posts): seo audit checklist
  group 5 (1 keywords, ~1 posts): technical seo
  group 6 (1 keywords, ~1 posts): seo checklist
  group 7 (1 keywords, ~1 posts): technical seo audit checklist