Therefore, you only want to submit your important, public, searchable pages in your sitemap. The whole point of a sitemap-and technical SEO in general-is to make sure you’re giving Google the best intel possible on how and where to crawl (and therefore index and rank) your pages.
In fact, you definitely shouldn’t put everything in there. Unless your internal linking is PERFECT and all your 100’s or 1,000’s of URLs have earned external backlinks, search bots are going to have a hard time finding all of those pages.īut not EVERYTHING needs to be in your sitemap. “If your site’s pages are properly linked, our web crawlers can usually discover most of your site.”īut there are a few cases where a sitemap is a huge help- like if your website is brand new, recently changed a ton of URLs, or you have a big website (1,000+ pages). These giant URL lists tell search engines which pages on your site are most important. If you’ve been in SEO for more than a few days, especially if you focus on technical or On-page SEO, you’ve definitely created a sitemap.Īnd Screaming Frog’s Spider tool is by far the best in the business.Ī sitemap is a blueprint of your website that help search engines find, crawl and index all of your website’s content. Otherwise, if you want to learn the how and why of these sitemap exclusions, keep reading.
This post will show how to exclude them - and tell you which ones to exclude.Īlready know what you’re doing? - Jump the full list. If you run a standard/default Screaming Frog Spider crawl on a WordPress website, you may run into some unnecessary results on your sitemap.