Technical SEO

SEO tools

How to submit your site to be crawled by Google

Written by

Tyler Scionti

Last updated:

March 22, 2020

You've put a lot of blood, sweat, and tears into your site.

Your copy is perfect, you've got a blog full of great posts, and landing pages fine tuned to convert visitors into leads for your business.

What's next?

After taking time to create an eye-catching website, you must ensure that potential visitors find it. As we have written before, that is where SEO comes in. However, in order to become an SEO master, you must learn the intricacies of how search engines work, and why that matters for your success.

Inthis article, we are going to explore what it means to have Google crawl yourwebsite. Beyond just a simple how-to article, we will identify why submittingyour website to Google matters as an integral component of your SEO strategy.

Defining Terms

Before we jump into how to submit your website to be crawled by Google, let's define some terms.

First, what does it mean to have your website crawled? Simply put, Google finds out what pages exist on the web. Without a central repository of every website out there, Google constantly searches for new pages and adds them to its list of known pages. The process whereby Google finds new websites is called crawling and the method they use to crawl pages is by using a 'spider'.

Crawlingmanifests itself in a few different forms. Google can discover a new page ifthey have crawled a website before. Other pages are uncovered when a link froma known page leads to a new page.

Finally,a website owner can submit their website to be crawled by submitting variouspages known as a sitemap. Obviously, this form is the one that contentmarketers will use since it is the most efficient for a new website.

Why Does Site Crawling Matter?

Once you create a new website, it is highly unlikely that new customers will find you immediately. When it comes to appearing in search results, Google needs to know you exist and in order for that to happen they need to crawl your site.

Google automatically crawls most websites once it discovers them; however, content marketers can accelerate the process of discovery by submitting their website to be crawled. This is a bit of a shortcut to Google knowing this page exists, which in turn means you'll appear in search results much faster.

How Do I Optimize My Website for Crawling?

Optimizing your website to prepare for this process is an essential step. To speed up Google’s crawling process, you can improve your website for crawling in a few ways.

For example, you could submit an individual URL to Google if you make changes to one page on your website. This step may be more applicable to businesses that already have a website that has been crawled by Google.

Second,a page on your website could be discovered through a page Google already knowsabout. However, links in advertisements, links in comments or other links thatdon’t follow the GoogleWebmaster Guidelines are exempted from this step.

Finally,make sure you start with your home page. Your home page is the most importantpage as it is the one that Google prioritizes as it crawls your site.

Craftinga solid site navigation system with links to sections and pages on your websitehelps Google as it makes its way through your website.

Submit Your Website to Be Crawled by Google

Nowthat you know what site crawling is, why it matters and how to prepare yoursite for website crawling, how exactly do I submit my website to Google?

In order to begin the site crawling process, navigate to this link which brings you to the Google Search Console.

Upon arriving at the page, you will see two options. One option consists of submitting a sitemap to Google. This process consists of numerous steps which are best explored on the Google website found here. However, we will summarize how to submit a sitemap to Google.

Submittingyour sitemap consists of first deciding which pages on your site will becrawled by Google, decide which sitemap format you want to use and making yoursitemap available to Google by adding it to your robots.txt file or submittingit to your Search Console.

Thelink mentioned above provides a detailed outline of important sitemapguidelines you must follow when submitting your sitemap so make sure to accountfor those guidelines before submitting it to Google.

Onceyour sitemap is created and ready to go, you can submit it to Google in thefollowing ways:

1. Submitit to Google using the Search Console Sitemaps tool

2. Insert the following line anywhere in your robots.txt file, specifying the path to your sitemap: Sitemap: http://example.com/sitemap_location.xml

3. Use the "ping" functionality to ask us to crawl your sitemap. Send an HTTP GET request like this: http://www.google.com/ping?sitemap=<complete_url_of_sitemap>

The second way you can submit your website to be crawled by Google is to use the Fetch as Google tool found at this link.

This tool functions similarly to the sitemap tool described above; however, this tool provides information on the URL Inspection Tool which can crawl individual URLs. To submit a URL to the index, you inspect the URL using the URL Inspection Tool. Then, you select request indexing which will run a live test on the URL to see whether it has any obvious indexing issues. If this tool finds issues with the page, it can provide you with information on how to fix those issues.

Moving Forward

Rememberthat crawling can take anywhere from a few days to a few weeks to complete so theurgency of starting the process now is imperative for content marketers.

Now that you are equipped with the knowledge of how to get your site crawled, make sure you get this step done today! As you continue to create content, and Google is aware of that awesome content you're creating, they'll be able to serve your site up in search results which means more traffic and conversions for you. Like many things with SEO, things might get a bit technical but if you just follow the steps and stick with it, you'll see big returns.

     

Start mastering SEO today

Learn more about Centori