It’s the moment of truth! Your new website is live, and you need people to visit. So you are probably asking the question, ‘how do I make Google crawl my site’?
We have all seen some film where someone creates a website and ‘puts it live’ and within minutes the laptop ‘dings’ because someone is visiting.
Whilst this is a really fun notion, and whilst we have run many a paid campaign and watched with bated breath as the clicks roll in, it still begs the question:
When it comes to organic owned content, how do I make Google crawl my site, how often does Google crawl a site, and is there anything I can do to speed up the crawling process? This blog post will answer that question.
How often does Google crawl a site?
It’s the moment of truth! Your new website is live, and you need people to visit. So you are probably asking the question, ‘how do I make Google crawl my site’? We have all seen some film where someone creates a website and ‘puts it live’ and within minutes the laptop ‘dings’ because someone is visiting. Whilst this is a really fun notion, and whilst we have run many a paid campaign and watched with bated breath as the clicks roll in, it still begs the question: In a webmaster hangout, a web publisher queried how fast Google removed pages from the index if they added a noindex nofollow to it. The publisher saidthey had added noindex but the page remained in Google’s index. Google’s John Mueller explained with an answer that revealed some interesting facts about how website pages are indexed. John Mueller revealed that URLs are crawled by Google at different rates. That’s not really surprising. What was surprising was that he said that some URLs can be crawled as little as once every six months! in fact, Here’s exactly what John Mueller said “I think the hard part here is that we don’t crawl URLs with the same frequency all the time. So some URLs we will crawl daily. Some URLs may be weekly. Other URLs every couple of months, maybe even every once half year or so. So this is something that we try to find the right balance for so that we don’t overload your server. And if you made significant changes on your website across the board then probably a lot of those changes are picked up fairly quickly but there will be some leftover ones. So in particular if you do things like site queries then there’s a chance that you’ll see those URLs that get crawled like once every half year. They’ll still be there after a couple of months. And that’s kind of… the normal time for us to kind of reprocess/re-crawl things. So it’s not necessarily a sign that something is technically completely broken. But it does mean that if you think that these URLs should really not be indexed at all, then maybe you can kind of back that up and say well here’s a sitemap file with the last modification date so that Google goes off and tries to double-check these a little bit faster than otherwise.” Search Engine Journal published an article on this hangout. So, when it comes to organic owned content, how do I make Google crawl my site, how often does Google crawl a site, and is there anything I can do to speed up the crawling process to get more website traffic? This blog post will answer that question.Why Do You need To Index Your Site?
If you want your site to show up in the search results, then it needs to be indexed. Although we have some small business clients who aren’t always looking for SEO with their websites (they use them as a calling card), you still need to be indexed to be found at all. And you don’t want your site to just be indexed once. You want Google (and other search engines) to keep on doing it. Search engines like Google don’t just update automatically. They use spiders. Not real spiders. That would be too scary. I’m talking about snippets of code that go forth and ‘crawl’ the web.What exactly is a web crawler?
The friendly spider’s job is to look for new content and update the already indexed version of your website. By New stuff, it can mean a new page, a new blog, or changes to an existing page. According to the wonderful Wikipedia, A Web crawler sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering). Search Engine Crawlers will look for URLs on your site and basically put them in the right place, or the place it thinks is right. Google’s crawler is called the Googlebot. It crawls through websites and adds all the information on a page to a google index so that it can find the information later when someone is searching that topic on a search engine.How Does Google Crawl Websites?
Crawling is when Google visits your website, with its spider crawler. It’s nothing scary, just a little spider that crawls through all the pages of your website. After crawling takes place, Google Indexes your website. But what actually is a Google crawl? Simply put, the Googlebot ‘follows a path’ through your website. via a sitemap if you have one, or via its pages and linked pages. This is why you need a really good site structure. Indexing is the process of adding the pages it crawls to an index. rather like yellow pages, the index is a vast database of web pages. Google and other search engines can then serve them up when you search online for anything. Now, the search engine is only as good as your website’s code and content. If you want Google to index your site then it needs to conform to web standards. Some pages might not get indexed if they have errors, like not mobile-friendly. or Google may choose not to crawl duplicate content. Gone are the days of having multiple pages with a tweak of only one word. Google will decide which page deserves to be indexed and ignore others.Where Will Google Index My Content?
It used to be simple. Just stuff the keyword on the page as many times as possible. But come on, this is ridiculous, because it offers no quality to the user! Fortunately, search engines have gotten a lot cleverer than they used to be. Thankfully, these days stuffing keywords into your content (and meta tags) will not just be ignored by search engines, it will probably get you penalized. you might even have your site blacklisted and booted out of the SERPs altogether. which means you won’t rank for anything. but why? Because Google’s algorithm is now far more concerned with overall user experience, and something called ‘search intent’. – Why is this person searching for a particular search term? Usually, it’s one of four “I want to go, I want to know, I want to buy or I want to learn” micro-moments specified by Google. And that’s why indexing is so important. Regular indexing also improves your search results overall. If you want to rank higher in search, try it and see! As the web crawler notes changes to your website, they are updated in Google’s searchable index. The pages will only be added to the index if they conform to webmaster standards and aren’t doing anything black hat. When the crawler sees changes on your website, it processes those changes and analyses the content and code, and slots it nicely into the index. That’s it really. When a user types something into Google, the algorithm decides where to rank your page in comparison with all the other pages related to those words. I think it’s a bit like the music charts. The more popular a page is, the higher it will climb.What are the most important indexing factors?
we update our site quite often and as a result, it does get crawled every couple of days, sometimes daily. As a rule, you want frequent crawling. It’s like Google is interested in what’s going on. Check-in with Google search console, to find out when your site was last crawled. here are some factors that make it better for Google: Speed. – This means a better experience for your users. the better the experience, the higher your site will rank. The faster a site, the easier to index. I use SEMRush to audit websites, and if a page is slow, sometimes it times out with SEMRush. Google doesn’t have all day, so make sure your pages are fast. The other way you can help is by submitting your sitemaps. but I will go into that in more detail later. Let’s get to the steps of getting your site indexed.Step 1: Am I indexed already?
Do a quick check by typing site: followed by your domain into Google. This will tell you if your pages are on Google and how many are there.Step 2: Set Up Search Console and Google Analytics.
If you aren’t using Google Search Console (formerly Google Webmaster Tools) to monitor and crawl your web page, then you really should be. There are valuable insights here you cannot afford to miss. This is how you find out what people really type in to find your web pages. Get yourself a Google search console account and hook your website up to it. This post won’t go into how to do that, because it assumes you will have Google Analytics in place, or you probably won’t be asking questions about Google crawl statistics. Just know, you will have to verify you are the site owner. This is easy if you have Google Analytics set up on your website. All you need to do is add your domain via the search console and you will be presented with a series of ways to verify. The simplest way is via Google Analytics. The Google Search Console (A free service) will give you specific data on how often Google crawler visits your website. It looks like this: At the time of updating this post. Google Search Console stores the last sixteen months of website data and can give you some amazing insights into your website. How you are being found, what key phrases you appear for, and how often users are clicking on those search results. You can compare results year on year, or with a similar period. for example, you can compare the last three months against the previous three months. You can also see a list of URLs or pages that were served for each search term. Even more useful, you can view the top pages that drive traffic to your site, and the search terms that triggered an impression for your landing page. This is a really useful tool for rooting out keyword opportunities and seeing just where you could rank when you don’t already. Google Search Console in this respect is so much more than an error monitor. It’s a brilliant keyword research tool. if you have a WordPress website, install Google Site Kit. It will take care of all of this for you in a few easy clicks.Step 3: Create a content marketing strategy & Blog
Most website owners just assume that it’s enough to add content and put some keywords in. it isn’t. You need to build a content strategy around search engine results if you want to be found and indexed. Make a plan of your content, based on keyword research, keyword difficulty, and search volumes. It’s a great starting point. When I started my digital business, I was also under the impression too that I just needed to add content. as a digital agency, we have content that covers our projects and even company news. As time went on I learned how little people who don’t know anything about your business are interested in your company news. you need to create content based on topics that users are searching for and within your niche. Don’t start writing posts about unrelated topics… A recent Hubspot study suggested that 40% of marketers say content marketing is a very important part of their overall marketing strategy. (Source: https://www.hubspot.com/marketing-statistics). Try and publish informative, useful, and interesting content. No one cares if your company just turned 10. Use Infographics like this 2020 content marketing infographic When you create content look to create it not just for yourself but guest post on other reputable sites. conversely don’t link back to your site from any spammy sites. Google will just assume you are spammy. We recommend Neil Patels content strategy. Here’s his framework.- What are your goals? Specify SMART goals and how you’ll measure your progress (i.e., metrics).
- Who is your target audience? Customer profiles or personas are essential to understanding your audience and what they want/need.
- What types of content will you produce? You want to make sure you’re delivering the type of content that your target audience wants to see.
- Where will it be published? Of course, you’ll be hosting your own content on your website, but you may also want to reach out to other sites or utilize platforms such as YouTube, LinkedIn, and Slideshare.
- How often will you publish your content? It’s far better to produce one well-written, high-quality article a week consistently than to publish every day for a week and then publish nothing for a month. Consistency is key.
- What systems will you adopt for publishing your content? Systems are basically just repeatable routines and steps to get a complex task completed. They’ll help you save time and write your content more quickly, so you can stay on schedule. Anything that helps you publish content in less time without sacrificing quality will improve your bottom line.
- Include the blogging/content tools and technology you’ll use and how they fit into your system.