SEO Technical: Indexing
Guides indexing troubleshooting and fix actions. For how to find and diagnose issues in GSC, see google-search-console.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Scope (Technical SEO)
- Fix actions: noindex, canonical, content quality, URL Inspection; verify robots.txt does not block (see robots-txt)
- Noindex: Page-level index control; which pages to exclude and how. Complements robots-txt (path-level crawl control) and google-search-console (Coverage diagnosis)
Initial Assessment
Project context: Read root contextus.md when present and load only the modules relevant to this task. Without Contextus, use available project material or user-provided facts and ask for missing information; do not create a parallel context system.
Identify issue from GSC (see google-search-console for Coverage report, issue types, diagnosis workflow). Then apply fix below.
Crawled - Currently Not Indexed
Static Assets (Next.js / Vercel)
Vercel adds unique dpl= params to static assets per deploy, creating many "Crawled - currently not indexed" URLs.
Static assets in "Crawled - currently not indexed" is normal and expected.
Other Issue Types (from GSC Coverage)
Soft-404
A soft-404 occurs when a page returns HTTP 200 but the content indicates the page doesn't exist (e.g. "Page not found" message, empty state). Google may treat it as 404 and exclude from index.
Noindex Usage
- How:
metadata.robots = { index: false }or<meta name="robots" content="noindex">or X-Robots-Tag - Rationale: Not all site content should be indexed; noindex is a valid choice for many pages
- Caution: Avoid noindex on important content pages
- With robots.txt: robots.txt = path-level crawl control; noindex = page-level index control. Do not block noindex pages in robots.txt—crawlers must access the page to read the directive. Use both: robots for /admin/, /api/; noindex for /login/, /thank-you/, etc. See robots-txt for when to use which.
- nofollow ≠ noindex: nofollow controls link equity only; it does not prevent indexing. To exclude from search, use noindex. See page-metadata for meta robots implementation.
Page Types That Typically Need Noindex
noindex,follow vs noindex,nofollow: Use noindex,follow for most cases—excludes from SERP but allows link equity. Use noindex,nofollow only for login (security), staging, or temporary test pages.
Page Removal Decision Framework
When intentionally removing a page from the web, choose the method based on whether a relevant alternative exists and whether the page should remain accessible:
Before removing: Check the URL's search traffic, backlinks, internal links, and conversion value. If the page has value, consider updating or merging rather than removing.
Common mistakes:
- 404-ing pages that have relevant alternatives (wastes accumulated signals)
- Redirecting all deleted pages to the homepage (breaks user intent)
- Creating redirect chains (A → B → C) instead of direct redirects
- Removing pages without cleaning up internal links pointing to them
- Using
robots.txtto block noindex pages (crawler must access the page to read the noindex directive)
Post-removal cleanup:
- Remove deleted URLs from XML sitemap; update and resubmit
- Update internal links to point directly to the final URL (avoid relying on redirects)
- For 301 redirects, ensure the target URL is in the sitemap
- In GSC, use URL Inspection to verify important pages; use Removals tool for temporary quick-hide (not permanent — use proper HTTP status or noindex)
Google Indexing API
Requirements: Enable Indexing API, create service account, add owner in Search Console, request quota (default 200 URLs/day).
Output Format
- Action items: Prioritized fixes
- References: Page indexing report
Related Skills
- google-search-console: Find and diagnose indexing issues in GSC
- robots-txt: Path-level crawl control; when to use robots.txt vs noindex; do not block /_next/ or noindex pages
- page-metadata: Meta robots implementation; noindex vs nofollow
- xml-sitemap: Submit and maintain sitemap
- indexnow: Faster indexing for Bing
- canonical-tag: Resolve duplicate content
