An XML sitemap should give search engines a clean, current list of the URLs you want them to discover and consider for indexing. If a sitemap contains redirected, broken, blocked, duplicate, or non-canonical URLs, it sends conflicting signals and can waste crawl attention.
This guide is for website owners, editors, developers, and SEO practitioners who need to create a sitemap, review an existing file, or investigate why submitted URLs are not being indexed. You can use the SatoGifts XML Sitemap Generator to help create a sitemap, then apply the checks below before and after submission.
Central concept: A sitemap is a discovery and preference signal, not an instruction that guarantees crawling, indexing, or visibility. Each listed URL must still be accessible, useful, technically eligible, and consistent with the site’s other indexing signals.
Open the sitemap location directly and confirm that it loads without authentication, repeated redirects, or a server error. Search engine crawlers must be able to request the file. A temporary outage may prevent processing, while a persistent access problem can make the sitemap unusable.
A sitemap should normally contain only URLs that you want indexed. Each URL should provide a successful response, allow indexing, and represent the preferred version of its content.
| URL condition | Include? | Reason |
|---|---|---|
| Successful, indexable canonical page | Yes | It is a clear indexing candidate. |
| Redirected URL | No | List the final destination instead. |
| 404 or 410 URL | No | The page is unavailable or removed. |
| Server error | No, until fixed | The content cannot be reliably accessed. |
| Page marked noindex | No | The sitemap and indexing directive conflict. |
| Duplicate or non-canonical variant | Usually no | List the preferred canonical version. |
| URL blocked from crawling | Usually no | The crawler may be unable to inspect the page. |
Use the Broken Links Finder to identify unavailable internal destinations and the Link Analyzer to review how important pages are linked. A sitemap is not a substitute for a logical internal linking structure.
Test a representative group of URLs and investigate every unexpected status code. A page that looks normal in a browser may still redirect, fail for some requests, or return a misleading response.
Also watch for soft errors, where a URL returns 200 but displays an unavailable-page message, empty template, or other content that does not function as a real page.
The URL in the sitemap should normally match the page’s preferred canonical URL. Check protocol, hostname, path capitalization, trailing slashes, and parameters. Small variations can create separate addresses even when they display similar content.
Use the Meta Tags Analyzer to inspect relevant page directives and canonical information. If a sitemap lists one URL while the page identifies another as canonical, search engines must interpret the conflict and may ignore the sitemap preference.
Confirm that listed pages are not intentionally excluded. Review page-level indexing directives and the site’s robots rules separately. A robots restriction controls crawling; a noindex directive addresses indexing. Blocking a page from crawling can prevent a crawler from seeing page-level instructions.
The Robots.txt Generator can help prepare a robots file, but generated rules still require careful review. Do not block necessary content merely to remove it from a sitemap.
An optional last-modified date should reflect a meaningful content update, not every routine crawl, sitemap generation, or template request. Incorrect dates can reduce the usefulness of the signal.
A single sitemap is generally limited by the sitemap protocol to 50,000 URLs and 50 MB when uncompressed. Search engine requirements can change or add conditions, so confirm current documentation for the services you use.
For a large site, divide URLs into focused child sitemaps and reference them from a sitemap index. Practical divisions may include products, categories, articles, images, languages, or publication periods. Grouping makes errors easier to isolate and helps teams understand which content type is affected.
Do not split files merely to create more sitemaps. Use a structure that supports maintenance, monitoring, and reliable updates.
Check that the document is well formed, uses the expected sitemap structure, and contains complete absolute URLs. Pay attention to escaped characters, malformed entries, accidental whitespace, and URLs from staging or development environments.
Automated validation is useful, but manual sampling remains necessary. Select URLs from the beginning, middle, and end of each file, as well as examples from every major template or content type.
After placing the sitemap on the live site, submit or reference it through the search engine methods available to you. A successful fetch means the file could be retrieved; it does not mean every listed URL was accepted for indexing.
Compare reported discovery or indexing information with direct checks. Inspect selected pages in the relevant search engine webmaster tools, open them in a browser, review their source-level directives, and test their final status. Reporting can be delayed, sampled, or grouped, so avoid drawing conclusions from one number alone.
A shop changes product paths during a site migration. The sitemap still lists the old addresses, each of which redirects to a new product URL. The destination pages are valid, but the sitemap continues to promote obsolete locations.
Resolution: Replace the redirected entries with the final canonical URLs, update internal links, regenerate the sitemap, and verify a sample of destinations. Keep necessary redirects in place for users and previously discovered links, but do not continue listing the old versions.
An article sitemap contains parameterized URLs used for campaign tracking. Each page identifies the clean, parameter-free article address as canonical. The sitemap and canonical element therefore point to different versions.
Resolution: List only the clean canonical addresses. Remove tracking variants, check that internal navigation also uses clean URLs, and confirm that each preferred article returns 200 and permits indexing.
A content management system rebuilds the sitemap nightly and assigns the generation date to every URL, even when most pages have not changed.
Resolution: Connect update dates to meaningful page revisions or omit them. Rebuilding a file does not mean every listed document was modified.
A technically correct sitemap cannot make low-value, duplicate, inaccessible, or weakly connected pages indexable by itself. Search engines decide what to crawl and index using many signals, and their reports may take time to update. Differences between listed, discovered, crawled, and indexed totals are not automatically evidence of an error.
After correcting the sitemap, strengthen internal links, resolve page quality problems, review metadata with the Meta Tags Analyzer, and assess slow templates with the Page Speed Checker. Regenerate the sitemap when URLs materially change, then monitor patterns rather than reacting to short-term fluctuations. If only one section has problems, investigate its templates, directives, response codes, and canonical rules before changing the entire site.
No. It helps search engines discover preferred URLs, but each engine decides whether and when to crawl or index them.
Normally no. Keep the redirect where it is needed, but list the final canonical destination in the sitemap.
It can technically appear, but it creates conflicting signals. Remove it unless the noindex directive is accidental and will be corrected.
Regenerate it when indexable URLs are added, removed, moved, or meaningfully updated. The appropriate frequency depends on how often the site changes.
No. Include the date only when it is accurate and maintained consistently. An omitted date is preferable to a misleading one.
Possible reasons include duplication, canonical selection, low-value content, crawl restrictions, errors, recent publication, weak internal linking, or normal processing delays. Inspect representative URLs rather than assuming one cause.
Not inherently. A small site may need only one. Larger sites benefit from logically divided sitemaps that stay within protocol limits and make troubleshooting easier.
Last reviewed: July 28, 2026