It’s the moment of truth! Your new website is live, and you need people to visit. So you are probably asking the question, ‘how do I make Google crawl my site’?
We have all seen some film where someone creates a website and ‘puts it live’ and within minutes the laptop ‘dings’ because someone is visiting.
Whilst this is a really fun notion, and whilst we have run many a paid campaign and watched with bated breath as the clicks roll in, it still begs the question:
When it comes to organic owned content, how do I make Google crawl my site, how often does Google crawl a site, and is there anything I can do to speed up the crawling process? This blog post will answer that question.
How often does Google crawl a site?
It’s the moment of truth! Your new website is live, and you need people to visit. So you are probably asking the question, ‘how do I make Google crawl my site’?
We have all seen some film where someone creates a website and ‘puts it live’ and within minutes the laptop ‘dings’ because someone is visiting.
Whilst this is a really fun notion, and whilst we have run many a paid campaign and watched with bated breath as the clicks roll in, it still begs the question:
In a webmaster hangout, a web publisher queried how fast Google removed pages from the index if they added a noindex nofollow to it. The publisher saidthey had added noindex but the page remained in Google’s index. Google’s John Mueller explained with an answer that revealed some interesting facts about how website pages are indexed.
John Mueller revealed that URLs are crawled by Google at different rates. That’s not really surprising. What was surprising was that he said that some URLs can be crawled as little as once every six months!
in fact, Here’s exactly what John Mueller said
“I think the hard part here is that we don’t crawl URLs with the same frequency all the time. So some URLs we will crawl daily. Some URLs may be weekly. Other URLs every couple of months, maybe even every once half year or so.
So this is something that we try to find the right balance for so that we don’t overload your server.
And if you made significant changes on your website across the board then probably a lot of those changes are picked up fairly quickly but there will be some leftover ones.
So in particular if you do things like site queries then there’s a chance that you’ll see those URLs that get crawled like once every half year. They’ll still be there after a couple of months.
And that’s kind of… the normal time for us to kind of reprocess/re-crawl things. So it’s not necessarily a sign that something is technically completely broken.
But it does mean that if you think that these URLs should really not be indexed at all, then maybe you can kind of back that up and say well here’s a sitemap file with the last modification date so that Google goes off and tries to double-check these a little bit faster than otherwise.”
Search Engine Journal published an article on this hangout.
So, when it comes to organic owned content, how do I make Google crawl my site, how often does Google crawl a site, and is there anything I can do to speed up the crawling process to get more website traffic? This blog post will answer that question.
Why Do You need To Index Your Site?
If you want your site to show up in the search results, then it needs to be indexed. Although we have some small business clients who aren’t always looking for SEO with their websites (they use them as a calling card), you still need to be indexed to be found at all.
And you don’t want your site to just be indexed once. You want Google (and other search engines) to keep on doing it.
Search engines like Google don’t just update automatically. They use spiders. Not real spiders. That would be too scary. I’m talking about snippets of code that go forth and ‘crawl’ the web.
What exactly is a web crawler?
The friendly spider’s job is to look for new content and update the already indexed version of your website. By New stuff, it can mean a new page, a new blog, or changes to an existing page.
According to the wonderful Wikipedia, A Web crawler sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering).
Search Engine Crawlers will look for URLs on your site and basically put them in the right place, or the place it thinks is right.
Google’s crawler is called the Googlebot. It crawls through websites and adds all the information on a page to a google index so that it can find the information later when someone is searching that topic on a search engine.
How Does Google Crawl Websites?
Crawling is when Google visits your website, with its spider crawler. It’s nothing scary, just a little spider that crawls through all the pages of your website. After crawling takes place, Google Indexes your website.
But what actually is a Google crawl? Simply put, the Googlebot ‘follows a path’ through your website. via a sitemap if you have one, or via its pages and linked pages. This is why you need a really good site structure.
Indexing is the process of adding the pages it crawls to an index. rather like yellow pages, the index is a vast database of web pages. Google and other search engines can then serve them up when you search online for anything.
Now, the search engine is only as good as your website’s code and content. If you want Google to index your site then it needs to conform to web standards. Some pages might not get indexed if they have errors, like not mobile-friendly. or Google may choose not to crawl duplicate content. Gone are the days of having multiple pages with a tweak of only one word. Google will decide which page deserves to be indexed and ignore others.
Where Will Google Index My Content?
It used to be simple. Just stuff the keyword on the page as many times as possible. But come on, this is ridiculous, because it offers no quality to the user! Fortunately, search engines have gotten a lot cleverer than they used to be.
Thankfully, these days stuffing keywords into your content (and meta tags) will not just be ignored by search engines, it will probably get you penalized. you might even have your site blacklisted and booted out of the SERPs altogether. which means you won’t rank for anything. but why?
Because Google’s algorithm is now far more concerned with overall user experience, and something called ‘search intent’. – Why is this person searching for a particular search term? Usually, it’s one of four “I want to go, I want to know, I want to buy or I want to learn” micro-moments specified by Google.
And that’s why indexing is so important. Regular indexing also improves your search results overall. If you want to rank higher in search, try it and see! As the web crawler notes changes to your website, they are updated in Google’s searchable index. The pages will only be added to the index if they conform to webmaster standards and aren’t doing anything black hat.
When the crawler sees changes on your website, it processes those changes and analyses the content and code, and slots it nicely into the index. That’s it really.
When a user types something into Google, the algorithm decides where to rank your page in comparison with all the other pages related to those words. I think it’s a bit like the music charts. The more popular a page is, the higher it will climb.
What are the most important indexing factors?
we update our site quite often and as a result, it does get crawled every couple of days, sometimes daily. As a rule, you want frequent crawling. It’s like Google is interested in what’s going on. Check-in with Google search console, to find out when your site was last crawled. here are some factors that make it better for Google:
Speed. – This means a better experience for your users. the better the experience, the higher your site will rank. The faster a site, the easier to index. I use SEMRush to audit websites, and if a page is slow, sometimes it times out with SEMRush. Google doesn’t have all day, so make sure your pages are fast.
The other way you can help is by submitting your sitemaps. but I will go into that in more detail later. Let’s get to the steps of getting your site indexed.
Step 1: Am I indexed already?
Do a quick check by typing site: followed by your domain into Google. This will tell you if your pages are on Google and how many are there.
Step 2: Set Up Search Console and Google Analytics.
If you aren’t using Google Search Console (formerly Google Webmaster Tools) to monitor and crawl your web page, then you really should be. There are valuable insights here you cannot afford to miss. This is how you find out what people really type in to find your web pages.
Get yourself a Google search console account and hook your website up to it. This post won’t go into how to do that, because it assumes you will have Google Analytics in place, or you probably won’t be asking questions about Google crawl statistics.
Just know, you will have to verify you are the site owner. This is easy if you have Google Analytics set up on your website. All you need to do is add your domain via the search console and you will be presented with a series of ways to verify. The simplest way is via Google Analytics.
The Google Search Console (A free service) will give you specific data on how often Google crawler visits your website.
It looks like this:
At the time of updating this post. Google Search Console stores the last sixteen months of website data and can give you some amazing insights into your website. How you are being found, what key phrases you appear for, and how often users are clicking on those search results.
You can compare results year on year, or with a similar period. for example, you can compare the last three months against the previous three months.
You can also see a list of URLs or pages that were served for each search term. Even more useful, you can view the top pages that drive traffic to your site, and the search terms that triggered an impression for your landing page. This is a really useful tool for rooting out keyword opportunities and seeing just where you could rank when you don’t already.
Google Search Console in this respect is so much more than an error monitor. It’s a brilliant keyword research tool.
if you have a WordPress website, install Google Site Kit. It will take care of all of this for you in a few easy clicks.
Step 3: Create a content marketing strategy & Blog
Most website owners just assume that it’s enough to add content and put some keywords in. it isn’t. You need to build a content strategy around search engine results if you want to be found and indexed.
Make a plan of your content, based on keyword research, keyword difficulty, and search volumes. It’s a great starting point. When I started my digital business, I was also under the impression too that I just needed to add content. as a digital agency, we have content that covers our projects and even company news.
As time went on I learned how little people who don’t know anything about your business are interested in your company news. you need to create content based on topics that users are searching for and within your niche. Don’t start writing posts about unrelated topics…
A recent Hubspot study suggested that 40% of marketers say content marketing is a very important part of their overall marketing strategy. (Source: https://www.hubspot.com/marketing-statistics).
Try and publish informative, useful, and interesting content. No one cares if your company just turned 10.
Use Infographics like this 2020 content marketing infographic
When you create content look to create it not just for yourself but guest post on other reputable sites. conversely don’t link back to your site from any spammy sites. Google will just assume you are spammy.
We recommend Neil Patels content strategy. Here’s his framework.
What are your goals? Specify SMART goals and how you’ll measure your progress (i.e., metrics).
Who is your target audience? Customer profiles or personas are essential to understanding your audience and what they want/need.
What types of content will you produce? You want to make sure you’re delivering the type of content that your target audience wants to see.
Where will it be published? Of course, you’ll be hosting your own content on your website, but you may also want to reach out to other sites or utilize platforms such as YouTube, LinkedIn, and Slideshare.
How often will you publish your content? It’s far better to produce one well-written, high-quality article a week consistently than to publish every day for a week and then publish nothing for a month. Consistency is key.
Once you have your strategy in place, start writing. You have to commit to it and it takes a lot of effort. especially if you aren’t a big company with a team of content writers. I’m sat at home on a Saturday afternoon, updating this blog post. but this is one of my most visited pages and so It’s worth the time it takes to drive traffic to our website.
You should blog even if you have an eCommerce website. if you have new products, write about them. can you write about lifestyle topics related to your products? for example if you have a website selling health supplements, write about healthy lifestyle topics, and link back to your products.
Step 4: Have a linking Strategy
Internal links, which means linking to other pages on your website is one the best SEO quick wins I know. Firstly, you will have lots of links in your website navigation. Your website structure is important, and topics and themes should be grouped together.
Make sure you have friendly URLs. what I mean by that is they shouldn’t be too long and they should contain words, not gobbledegook.
Use a sidebar in your blog to link to your most popular pages and posts. You can even create manual related topic links such as Related Post: Why Your Website Structure Matters.
This type of link building encourages the google bot to come back and recrawl. It also keeps users on your site for longer and provides them with (hopefully) more useful content.
Step 5: Leverage Social media
If you don’t have social media profiles for your website, do it now. one component of 2020 search engine optimization is paying attention to social media signals. Despite being ‘Nofollow’ They do seem to add some weight to your website ranking.
Set up Facebook pages, Twitter profiles, LinkedIn profiles or company pages, Pinterest, YouTube channels, and especially Google My Business pages. If your business has separate brands or projects, it can also be worth setting up social media profiles just for these.
And it goes without saying that if you create these profiles, you need to keep them up to date,
Just How Often Does Google Crawl A Site?
If your perfectionist’s brain is asking, “what is the exact Google web crawler frequency” No one knows. It depends on your domain authority, your backlinks, and any constraints. There are also factors that can waste your crawl budget, such as redirects and pages in your sitemap that no longer exist.
In Google’s Words, their robot spiders “regularly” crawl web content to update its grand index used to create the SERPs (Search Engine Results Page).
These crawls result in changes to the SERPs, which display soon after the index is updated. This can mean you may see changes at any time from a few moments to a few days. Again, no one really knows, as it’s all done by a magic algorithm. Ok, it’s not really magic, but it is quite mystical.
The more visitors you get, typically the more your website will get crawled. Googlebots aren’t going to waste time crawling sites that never get visited.
Can I speed up the process?
Whenever you make a change to a page on your site, for example, if you rewrite it, update the content, or improve on-page SEO, you can resubmit the URL in the search console. Usually, you will get reindexed pretty fast. In my experience, it’s around a minute.
Improvements to a web page can increase its position as soon as it’s re-indexed, so it’s worth bearing this in mind.
Although you have no direct control over when Google crawls your pages, there are a few things you can do to encourage them.
Sitemaps & Error Checking
.An XML sitemap one way to ensure Google Search Console is kept informed of your new content. If you work with a popular content management system such as WordPress, then you can easily generate and submit sitemaps to the search console.
Also, check for mobile usability. You need to know your site is 100% mobile-friendly. The image here shows minor issues over time but all have been resolved when spotted.
Page Speed and Connectivity.
If Google can’t crawl your site due to a server error, then it will stop crawling your site. If this happens, your rankings can plummet. To clarify though, we aren’t talking momentary lapses here, but if your site is down for a prolonged period of time, it’s not going to keep sending bots your way.
Benefits of frequent content.
If you add new content to your site frequently, then Google will learn to crawl it more frequently. It’s not rocket science.
It can be hard to come up with new content, but the simplest way to do this is to add a blog to your site and fill it with information related to your industry.
You can blog about anything you like to be fair, but Google gets a fair view of what your site is about if you keep it relevant. For example, we write mainly about Google Ads and SEO insights, How to work with agencies, and anything we have gotten up to in the digital agency space.
Internal and Backlinking
Internal links and incoming links are still a key way to be found on Google. When Google notices that your content is being published and shared by other platforms, it will reward you.
Be careful though and avoid spammy directory listings and black hat techniques. The almighty Google can penalize you as well as reward you, and black marks are hard to come back from.
The Robots.txt file.
This is a simple text file that should live in your root folder. Occasionally it’s handled by the host, and you can’t edit it, but if you have WordPress and you have a plugin like Yoast or RankMath, you should be able to access it directly there.
This file gives instructions to crawlers about which pages to crawl and which to ignore. Again, the aforementioned WordPress plugins can take care of that for you too. If you are wondering why it’s good to deindex pages that’s a whole other topic, but not every page is worth Google’s attention. we don’t index our categories and tags, because they can end up cannibalizing our deliberate content.
If you are unsure how to access this file, or configure it, then speak to an expert. We are experienced WordPress developers, and can help with this.
Structured Data
Search engines love structured data when crawling and indexing. Add Schema markup to your website to boost your positions in the SERPs when you submit any page for web crawling. You could even show for a featured snippet, which means the coveted position zero.
Do you need help with quality content, or making sure your website is working to benefit your business? Speak to our team today and we can audit
