Crawled vs Discovered, Not Indexed: How to Tell Which Problem You Have and Fix It

Key takeaways

Two status labels that look similar have opposite fixes. One means Google fetched the page and declined to index it. The other means Google has not fetched it yet. This is the workflow that separates them and what to do in each case.

What you finish with

By the end of this workflow you will have three things: a correct classification of every affected URL into one of the two statuses (plus a third bucket for URLs that should not be indexed at all), a tested hypothesis for the dominant cause, and a dated baseline so you can tell in three weeks whether the fix worked.

Item

Detail

Who this is for

Anyone who owns a site where search visibility matters and has access to Search Console

Time

45 to 90 minutes for the first pass; about 20 minutes for the weekly re-check

Prerequisites

Search Console access with at least the Page indexing report, a way to see server logs or crawl-equivalent requests, and edit rights on the affected templates

Done when

You can state, per affected URL, which status it carries, which cause you tested, and what changed (or did not) after the fix

Not required

Requesting indexing for every URL, rebuilding your sitemap, or changing your site architecture wholesale

One framing rule before starting, because it prevents most wasted effort: these two labels are not severity levels. They are different machines. Treating them as the same thing is why people resubmit URLs that were never crawled and wonder why nothing happens.

The two statuses, in Google's own words

Crawled - currently not indexed: "The page was crawled by Google but not indexed. Google doesn't index all pages across the web, so this is a common message for many sites and doesn't necessarily indicate an issue that needs attention."

Read that carefully. Google fetched the page, read it, and decided not to keep it. The decision is about quality, duplication, or value. It is not about access.

Discovered - currently not indexed: "The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report."

Also read carefully. Google knows the URL exists and has deferred the fetch. The named reason is crawl capacity or site load concern, and the diagnostic tell is that the last crawl date is blank.

So: one is a judgment after reading, the other is a queue decision before reading. Your fix depends entirely on which one you are in.

Confirm which status you actually have

This step is where most people skip ahead, and where most fixes go wrong.

Action. Open the Page indexing report. Filter by each status separately. Export both lists. Do not work from the summary counts. Work from the URL lists.

Expected output. Two separate lists, plus a count of how many URLs are in both over time.

Quality check. For ten URLs from the "crawled" list, open URL Inspection and confirm the last crawl date is populated and recent. For ten from the "discovered" list, confirm the last crawl date is empty. If the date is populated on a "discovered" URL, either the filter or your assumption is wrong. Investigate before proceeding.

Recovery path if the lists look mixed. The report reflects a crawl at a point in time; a URL can move between buckets. Re-export in seven days and use the URL that appears in the same bucket twice as your working set. Do not build a fix on a single snapshot.

A third bucket you must create now. Before analyzing anything, remove URLs that should never have been indexable: filtered variants, internal search results, staging copies, syndicated duplicates, tag pages with one post, paginated archives with no unique value. If 60% of your "crawled, not indexed" list is filtered product variants, the correct fix is to stop them being crawlable, not to improve their content. This deletion step often resolves the majority of the report before any content work begins.

Decision flowchart: a URL in the page indexing report splits on whether the last crawl date is populated, leading to discovered-not-crawled with origin speed and link depth, or crawled-not-indexed with duplication and unique answer
One question sorts the two labels: is the last crawl date populated? Everything downstream, including which tests are worth running, depends on the answer.

If they are Discovered, not crawled: the queue problem

The status says Google deferred the fetch. Four causes, in the order worth testing.

Test 1: Is the site actually responsive to crawl requests?

Action. From outside your network, request a sample of affected URLs the way a crawler would (our Googlebot spider simulator does this without a terminal), then measure time to first byte and total response time across ten requests spread over ten minutes. Also check whether parallel requests degrade.

Expected output. A timing table. Anything consistently above a second or two for a simple HTML page deserves attention, and anything that degrades sharply under concurrency is the likely cause of a deferral.

Quality check. Test from a network that is not yours. A fast response from inside your own datacenter proves nothing about what a crawler in another region experiences.

Recovery path. If timing is the problem, the fix is infrastructure: caching, database query review on high-traffic templates, or rate limiting your own origin so bursts do not cascade. Crawl deferral caused by a slow origin does not improve with content changes.

Test 2: Is the URL discoverable only through a deep chain?

Action. For each affected URL, count the clicks from your homepage along the shortest path. Check whether it appears in any sitemap, and whether the sitemap is current.

Expected output. A depth column in your spreadsheet. URLs at depth four or deeper with no internal links pointing at them are functionally orphaned, whatever the sitemap says.

Quality check. Sitemap inclusion is a hint, not a queue ticket. A URL that appears only in a sitemap and nowhere in your link graph is the classic discovery-only case.

Recovery path. Add contextual internal links from pages that already have crawl attention. One relevant link from a frequently crawled page is worth more than a sitemap entry, and it takes minutes.

Test 3: Is the sitemap telling the truth?

Action. Compare sitemap URLs against live status codes. Count non-200 entries, redirects, and URLs that are blocked by robots rules.

Expected output. A clean sitemap with only canonical, indexable, 200-status URLs.

Quality check. A sitemap containing 404s or redirects wastes crawl budget on every fetch. Google's own guidance points at keeping these files accurate rather than large.

Recovery path. Rebuild the sitemap generation to filter on status and canonical, then resubmit once. Do not resubmit daily; the file is read on a schedule.

Test 4: Is crawl demand genuinely low relative to site size?

Action. Compare the number of URLs you publish per week against the number Google crawls per week, using server logs or Search Console crawl stats.

Expected output. A ratio. A site publishing 5,000 URLs a week and receiving 500 crawls a week has a structural mismatch that no per-page fix will solve.

Quality check. If the ratio is extreme, the problem is portfolio-level. Pages that matter compete with pages that do not.

Recovery path. Reduce what you publish, or reduce what is crawlable. Both sound painful and both work. A smaller set of pages that get crawled beats a large set that gets deferred indefinitely.

If they are Crawled, not indexed: the judgment problem

Here Google read the page and declined. Access is fine. The question is whether the page deserves a slot.

Test 5: Is the page a near-duplicate of something already indexed?

Action. Take the affected URL and search a distinctive sentence from it in quotes. Then check whether a canonical tag points elsewhere and whether that canonical is itself indexed.

Expected output. Either a clear duplicate relationship, or no match at all.

Quality check. Duplication is often partial: the template is identical, the boilerplate is identical, and only a data field differs. Copying text into a similarity tool can help, but a manual read of two examples side by side is usually faster and more accurate.

Recovery path. If the page is a variant, decide whether it should exist as an indexable URL. Merge it, canonicalize it, or make it genuinely distinct. Adding 200 words of filler to a duplicate page is the most common and least effective fix in this entire workflow.

Test 6: Does the page answer something no other page answers?

Action. Write one sentence stating the specific question this URL answers that no other URL on your site answers. If you cannot, that is the finding.

Expected output. A sentence, or an honest blank.

Quality check. "It ranks for its own product name" is not an answer to a question. Neither is "it has the specs." The test is whether a reader would arrive with a need and leave satisfied.

Recovery path. For thin pages: consolidate into a stronger page rather than improving each thin one. For pages with real value that are not being indexed: the problem is usually that the value is buried under boilerplate. Move the unique content above the template furniture.

Test 7: Is the page's content available without rendering tricks?

Action. Fetch the URL as a crawler would, without JavaScript execution, and read what comes back. The same simulator shows the raw HTML a crawler receives.

Expected output. The main content, in the HTML, readable.

Quality check. If the HTML contains a shell and the content arrives only after client-side rendering, you are relying on Google's renderer to do work you could do on the server. For pages being evaluated and declined, this is a common and fixable contributor.

Recovery path. Server-render the primary content, or pre-render the affected template. Verify by fetching again with JavaScript disabled.

Test 8: Is the site's overall quality dragging the page down?

Action. Compare indexation rates across sections. If one section has a far worse rate than the rest, the issue is scoped. If the whole site is affected, it is not a page problem.

Expected output. An indexation rate per section, with the worst section identified.

Quality check. Look at whether the worst section shares a template, a content generation method, or a publication cadence. That shared attribute is the candidate cause.

Recovery path. For site-level patterns there is no clever page fix. Reduce the production of the low-value class, improve or remove what exists, and expect weeks rather than days. Google's guidance here is explicit that not every page should be indexed, which means the correct outcome for some URLs is staying out.

Verify the fix

Do not verify by requesting indexing and calling it done. Requesting indexing is a nudge, not a cause.

Action. Record a baseline on the day you ship the change: URL count per status, date, and the specific change made. Re-check at 14 and 28 days.

Expected output. Movement in the status distribution, in the direction your hypothesis predicts. If you fixed a duplicate cluster, the merged URL should be indexed and the variants should be gone from the indexable set.

Quality check. Distinguish "moved bucket" from "disappeared." A URL that vanished because you noindexed it is a win for the report and a loss for nothing. A URL that vanished because it 404'd may be a mistake.

Recovery path. If nothing moved in 28 days, your hypothesis was wrong, not your execution. Return to the test list and pick the next most likely cause. This is normal; the first hypothesis is right roughly half the time.

Maintenance that keeps this from recurring

Three habits, none of them expensive.

Gate publication on indexability. Before a page ships, check that it returns 200, has a self-canonical, appears in the sitemap, and is reachable from at least one internal link. This prevents the discovery problem at the source.

Review the report monthly, not daily. The statuses move slowly and daily checking produces anxiety without information.

Keep a decision record for every class of page you choose not to index. Future you will want to know whether a noindex was deliberate. Six months from now, an unexplained noindex looks like a bug and gets "fixed" by someone who did not know the reason.

What this workflow does not fix

It does not fix a site that is being outcompeted on quality at scale. It does not substitute for original information a reader cannot get elsewhere. And it will not make Google index pages it has decided are not worth indexing, which is the documented and intended behavior of the system.

What it does is stop you from applying the wrong fix to the wrong label, which is the reason most of these reports stay static for months.

FAQ

Is "Discovered - currently not indexed" worse than "Crawled - currently not indexed"? They are different problems, not severity levels. Discovered means the fetch was deferred, often for capacity reasons. Crawled means Google read the page and declined, which points at value or duplication. Fix the one you actually have.

Should I request indexing for the whole list? No. Requesting indexing on a URL that has never been crawled may nudge a fetch, but it does not change why the page was deferred, and it does nothing for a page Google already read and declined. Use it after you have made a change worth re-evaluating.

How long should I wait before concluding a fix failed? Twenty-eight days is a reasonable first checkpoint. Indexing decisions on low-priority URLs can take longer, and a single snapshot is not evidence either way.

Does deleting the pages help? Deleting a genuinely worthless page is a legitimate fix, and it removes it from a report that is measuring your site's crawl allocation. Deleting a page that had value is not. The test is whether the page answers a question someone asks.

Do AI answer surfaces change any of this? They raise the cost of being unindexed, because a page that is not indexed cannot be retrieved or cited. The mechanics above are unchanged, but the consequence of a stale indexability problem is larger than it was two years ago.

Author: Bennett Hayes, Applied GEO Analyst at Auspia. He writes implementation notes and audit workflows for teams that need the fix to survive contact with a real site.

Explore this topic

Keep following the same growth thread