Skip to main content Scroll Top

Indexing and Sitemaps for Small Business Websites

Indexing and Sitemaps: A Small Business Guide

By David Roche | Reviewed 16 October 2026

A sitemap helps search engines discover the pages you want customers to find. Indexing is a separate decision: Google assesses a page and decides whether to include it in its search index. Submitting a sitemap does not guarantee that every listed page will be crawled, indexed or ranked.

For a small business, the useful question is whether your important public pages can be found and understood. This guide shows you how to check your WordPress sitemap, use Google Search Console and investigate a missing page without trying to index every URL your website creates.

Indexing and Sitemaps

Discovery, crawling and indexing are different

Stage What happens What it does not establish
Discovery Google learns that a URL exists, for example through a link or sitemap. That Google has fetched the page.
Crawling Google requests the URL and processes the response and accessible content. That the page will be included in the index.
Indexing Google analyses content and decides whether and how to store it for Search. A particular ranking, visit or enquiry.

 

A public page generally needs to allow Googlebot access, return a successful HTTP response and contain indexable content. Meeting these technical requirements makes a page eligible for consideration; it does not compel Google to index it. If a customer can open a page, that alone does not show what Google can access.

Google’s technical requirements for Search

Decide which pages should be indexed

Start with a short list of pages that serve a real customer task: your main services, useful product and category pages, location information, contact details and substantial advice. Check those first rather than treating the largest possible indexed total as the goal.

A booking confirmation, private customer area, internal search result or duplicate filter URL may have no useful role in public search. The right treatment depends on its purpose. Keep genuinely private information behind appropriate authentication; an indexing instruction is not access control.

Write down the preferred public URL for each important page. This gives you something meaningful to compare with a sitemap and Search Console, especially when a website has old addresses or several versions of the same content.

What an XML sitemap does

An XML sitemap is a machine-readable list of preferred page URLs. A sitemap index lists other sitemap files, such as separate files for pages, posts or products. Seeing several files beneath one index is normal.

Sitemaps are especially useful for new sites, large collections and pages that are harder to discover. A small, well-linked site may already be discoverable without one. A sitemap remains a practical way to make your preferred URLs explicit and monitor whether Google can read the list.

It supports internal links rather than replacing them. Customers should still be able to reach important pages through sensible navigation and relevant links. A page listed only in an XML file can remain difficult for people to find.

For navigation and internal links, see Meta Tags and Site Structure

Find your WordPress sitemap before adding a plugin

WordPress has included a core sitemap feature since version 5.5. Its standard index address is /wp-sitemap.xml. An SEO plugin may supply its own sitemap and disable the core version. Your actual address therefore depends on your setup.

  1. Check the sitemap settings or documentation for the SEO plugin you already use. Follow its sitemap link if provided.
  2. If you are using WordPress core sitemaps, try your website address followed by /wp-sitemap.xml. Use your own domain.
  3. Open the resulting index and one of its listed sitemap files. Confirm that you see URLs from the right website, not a staging domain or an error page.
  4. Keep a note of the working index URL and which system generates it. Avoid adding another sitemap plugin simply because a different example URL does not work.

A generated sitemap may not exist as a physical file in File Manager. WordPress or a plugin can produce the response when someone requests its address. Test the public URL before concluding that the sitemap is missing.

WordPress documentation on core XML sitemaps

Check what the sitemap includes

Sample a main service page, a recent article and a product or location page if relevant. Prefer final, public URLs that you want indexed. Investigate staging addresses, redirects, deleted pages, duplicates or pages intentionally marked noindex rather than leaving contradictory signals in the list.

Use the canonical version: the address you prefer when similar content exists at several URLs. Keep that preference consistent with internal links and canonical tags. Google can choose a different canonical; listing a URL is a signal, not an instruction it must follow.

Let your website software maintain the sitemap. Where it supplies lastmod dates, they should reflect significant changes, not a daily refresh or a new copyright year. Google ignores sitemap priority and changefreq values, so adjusting them is not an indexing shortcut.

Google’s sitemap creation guidance

Submit and check the sitemap in Search Console

Use a verified Search Console property that covers the correct website address. Take care with https versus http and www versus non-www when working with URL-prefix properties.

  1. Open the Sitemaps report and enter the working sitemap index URL in the submission field. Submit the address your site actually uses.
  2. After processing, open its row and check the status and last-read date. Success means Google fetched and read the file; it does not mean every listed page is indexed.
  3. If it could not be fetched or contains errors, read the details. Check the exact address, login requirements, server response and any security rule affecting access.
  4. Recheck periodically. For ordinary publishing, allow your generator to update the same sitemap address; you do not need to resubmit it after every post. Resubmit after fixing a persistent fetch failure or when substantial changes warrant it.

An empty Sitemaps report does not by itself prove Google has never found a sitemap: the report lists submissions made through the report or API. Submitting your working URL gives you a place to monitor its processing.

Google’s Sitemaps report instructions

Use URL Inspection to check an important page

Enter the complete page address in Search Console’s URL Inspection bar. Use the exact URL, including the correct domain and path. Read the indexing status, last crawl information and any available canonical details.

The initial result describes Google’s stored information, not a fresh test of your website. If you have changed the page since that crawl, select Test live URL. Compare the current result with the earlier information rather than assuming they describe the same moment.

A successful live test indicates that the tested page passes the checks it covers. It does not confirm that the page is already indexed, predict Google’s canonical choice or check every condition for appearing in Search.

Google’s URL Inspection guide

Do not use site: as your indexing count

A search such as site:example.com can be useful for a quick look at visible results. It is not a complete inventory. An indexed URL may be absent from those results, so a missing result does not prove that Google has not indexed it. Use URL Inspection for a specific page.

Google explains the limitations of site: searches

Read the Page indexing report without chasing every exclusion

Open the Page indexing report to look for patterns. Review changes and the reasons shown, then inspect representative important URLs. Use a sitemap filter where useful. The example list is a sample, not necessarily every affected address.

Not indexed is not automatically an error. Decide whether the particular URL should appear before changing anything.

Reported reason What to investigate
Discovered – currently not indexed Google knows the URL but has not crawled it. Check discovery links and reliable access; do not assume a penalty.
Crawled – currently not indexed Google fetched it but did not index it. Review its purpose, usefulness and overlap. Repeated submission is not a remedy.
Excluded by noindex Expected for some pages. For an important public page, check where the instruction comes from.
Alternate or duplicate Inspect the selected canonical. An intended duplicate may need no action.
Not found (404) Expected after a genuine removal with no replacement. Investigate if the page should still exist.
Server error or blocked access Investigate hosting, login and security rules affecting the public page.

 

Google’s Page indexing report explanations

Keep robots.txt, noindex and canonicals separate

Robots.txt controls crawler access. Blocking a URL there is not a reliable way to remove it from Search: Google can sometimes learn about the address elsewhere without crawling its content. It can also prevent Google from seeing a noindex instruction on that page.

Noindex asks Google not to index a page and must be available for Google to read, usually in the page or response header. Remove an accidental noindex only from pages intended for public search; do not change every excluded page at once.

Google’s guidance on noindex

A canonical identifies a preferred version of duplicate or very similar content. It is not a privacy setting or a guaranteed command. If Google selects an unexpected URL, compare the content, redirects, canonical declarations, internal links and sitemap entries before choosing a fix.

Google’s canonical URL guidance

For access and security checks, see Is Your Website Blocking Google or AI Search?

Example of investigating a missing service page

This is a fictional example. Willow Cleaning in York publishes an end-of-tenancy cleaning page. The owner can open it, but cannot find it in a site: search. They record the exact URL and inspect it in Search Console rather than assuming the sitemap has failed.

The stored result reports noindex. A live test still detects that instruction, and the owner finds an accidental exclusion in the page’s SEO settings. They first confirm that this is a public service page, rather than a private booking page.

They record the existing setting, remove the exclusion from that page and clear the relevant cache. They confirm the preferred URL is included in the generated sitemap and linked from the services page. A new live test no longer detects the block.

They request indexing once and record the change for later review. If other pages acquire unexpected behaviour, they can restore the recorded setting and ask their developer to investigate. The correction removes a barrier; it does not establish a deadline for indexing or a ranking improvement.

When should you request indexing?

For an important new page or a page with a substantive correction, you can use Request indexing after checking it. Requests have limits. Repeating the same request does not make Google process it faster, and submission does not guarantee inclusion.

For many URLs, maintain a useful sitemap and good internal links. Google says recrawling can take days to weeks. There is no universal deadline after which a missing page proves something is broken. Check the evidence and the scale of the problem instead of clicking the button every day.

Google’s guidance on requesting a recrawl

Do small sites need crawl budget optimisation?

Usually, a small business gets more value from fixing inaccessible pages, confusing duplicates and weak links. Google’s crawl budget guidance concerns large or rapidly changing sites. Do not buy a crawl budget service merely because a few pages have not been indexed.

Google’s explanation of crawl budget

What does this mean for AI search?

Google’s AI Search features still depend on ordinary Search foundations. Pages need to be indexed and eligible for a snippet to be considered as supporting links in its AI features. Neither a sitemap nor an llms.txt file guarantees selection or a citation.

Other providers have their own crawlers and controls. A Google indexing check does not establish whether another assistant can use your page. Keep access troubleshooting separate from choices about model training.

Google’s guidance for generative AI Search features

A manageable monthly check

  • Confirm that your sitemap still loads and review its Search Console status.
  • Inspect a small selection of important pages, including anything newly published or substantially changed.
  • Investigate unexpected changes affecting valuable pages before spending time on harmless duplicates or retired URLs.
  • Record the URL, observed reason, crawl or test date, action, owner and next review date. Separate observed results from expected outcomes.
  • After a fix, retest the current page. Use Validate fix where the report offers it and the relevant instances have been corrected. Allow reporting time to catch up.

Escalate a sudden loss affecting many important pages, persistent server failures or conflicting site-wide rules to your developer. Send exact URLs, dates, report details and recent changes so they can investigate the cause.

For the broader checklist, read Technical SEO for Small Businesses

Secret Link