So, you got your website up. Great!
Are you done? Not even close.
Let’s make sure you’re equipped with all the knowledge you need in order to have Google index your website into their MASSIVE SEARCH INDEX.
It’s funny, I started this article by trying to find a clever photograph of a spider crawling on a web, but then I came to a terrifying realization: there’s not a single picture out there that isn’t utterly horrifying.
There’s a purpose though, I promise.
The method an actual spider uses to build out their web they call their home, is similar to that of the spiders Google uses in order to build out their index. Let’s dig into exactly how to get Google to index your website, shall we?
Here’s an interesting fact:
“Google has the largest database of indexed websites. In fact, they have two times more than Yahoo and Bing. When you search for something on Google, it’s not searching the World Wide Web, it’s searching through its own database of indexed sites.”
That’s when it hits you, Am I on that list? Is the Google bouncer going to let you into the exclusive club? That depends on how you feel about spiders crawling all over your content. I’m being serious, except I’m not talking about real spiders thankfully.
- What is Crawling and Spiders in regards to Indexing?
- How do I get my website Indexed?
- It’s not Working, What Could be Going Wrong?
- Additional Resources
Spiders? Crawling? What is This, Fear Factor?
Let’s start with a basic overview of Google’s index and the scope of it.
Since Google doesn’t have time to look through every single page on the internet every time someone searches for something, it instead uses a specialized program to search through various webpages and add them to its database.
This process, known as indexing, is when your site is evaluated and stored in terms of what it’s saying and how well it says it. Then, when someone searches, Google presents websites from this database that it believes are most relevant to the search. Now you see why SEO is so important.
Currently, Google has 40 billion pages in its database and the number grows constantly. In comparison, Bing only has 14 billion pages in it’s database. Now, a more in-depth examination of the process:
1. Spiders (Not the Eight-Legged Kind)
When I say that Google uses spiders, I don’t mean they send out tiny arachnids into the internet.
Instead it’s a program that moves across the internet in a process known as “crawling.” During this time they act like normal people, except they read a whole lot faster. They search out various documents and gather relevant information from them.
They will crawl the same pages multiple times if there is new or updated content, and add that to the database, so there’s always room for improvement if your first crawl didn’t go so well.
The spiders start with pages they’ve already crawled over and they add in sitemap data provided to them by webmasters.
When they start crawling through previous pages, they detect links within them and add those to the crawl list.
In other words, you want them to get distracted by other high quality links to your content or to other influencer content. In essence, the use of sitemaps and links allows the spiders to discover new content on the web.
The second portion of the spider’s duties is to report back to the
mother-ship Google and report their findings.
This is the part where all of your hard work creating high quality content comes into play. If your website is deemed relevant and useful, the spiders will sing songs of your keywords, structure, alt-tags, entity salience, and all of that good stuff.
In all seriousness, think of indexing like being judged by the lord of the internet. Everything you’ve done will be examined, and no stone shall be left unturned.
Put on your best show, and remember that there are plenty of tools out there to help you.
How to Attract Google’s Crawling Spiders (Your Guide to getting Indexed)
I’m sure many of you woke up this morning and didn’t think to yourself you know what? I would love nothing more than to have a horde of spider crawl all over my website. Unlike you though, that’s exactly the thought that crossed my mind when my beady little eyes opened up this fine day.
I decided that the good people who read On Blast Blog needed to know how to get their blogs discovered by Google. let’s talk about several ways you can improve your website and make it stand out amongst the billions of others out there.
1. It’s Time to Make a Sitemap!
Like the name suggests, a sitemap is an XML document that is located on the server through which your site is hosted.
This document lists out all the pages on your blog or site and tells search engines when you’ve added a new page, or how often you update the current ones.
Depending on how often you update your site (it could be daily, weekly, or monthly) this site map can tell Google when and where to check for new updates to the content.
If you’ve decided to use WordPress as your blogging platform of choice, there is a handy plugin you can use:
- Google XML Sitemaps – This is a plugin for WordPress that will automatically generate a sitemap for your website, update it when needed, and submit it to search engines for you.
- Screaming Frog – If you don’t use WordPress, this is a freemium platform for building a sitemap as well.
SEO by Yoast also creates awesome XML Sitemaps.
2. Present Your Sitemap to Google via Webmaster Tools
Now that you’ve become a cartographer of the web, it’s time to present your map to the big guys and gals over at Google. Head on over Google Webmaster Tools to start the process. If you don’t have a Google account (Why?) you can make one for free. Follow the process to add your website.
Once you’ve done this, go to Crawl > Sitemaps and place the link to your blog’s sitemap in the field.
This will Let Google know that the new map is up. You can also do the same thing with other search engines via their webmaster tools to spread your influence further.
3. Get Started with Google Analytics
You may wonder why this would be needed for indexing.
It’s not directly tied to the process, but this is a program you’ll want to have regardless because it helps you track how well your promoting your blog.
In addition, adding this tool to your set will grant you another avenue of showing Google that there’s a new up and coming blog.
This tool will help you keep track of how successful things are going once you’ve been indexed so it makes sense to grab it early on.
4. Submit Your URL
Just like the sitemap, Google needs to know the URL for your website. While some may be of the opinion that this isn’t needed, it doesn’t hurt and it only takes a moment.
Instead of letting Google guess your name, head back into your Webmaster Tools and select the option to submit a URL.
There is an anonymous URL submission just below the sign-in for the Webmaster Tools that also allows you submit your website’s link to Yahoo. This isn’t the only way to do it, and on its own it doesn’t guarantee anything. That’s why I recommend it as a step in the overall process.
5. Create/Update Your Social Media Profiles
Since spiders find new sites by finding links during their crawls, you need to make your URL more prevalent on the Web.
Mostly everyone has a Facebook, Twitter, Google+, or LinkedIn profiles in some form or another. These profiles that have already been established will make for great fertile ground.
Head onto these profiles and add a link to your new site or blog into the “About Me” page of your profile or somewhere similar.
When these types are crawled, a URL on your profile like that is bound to be spotted and in that way you’ll have spiders all over your site before you know it.
6. Spread the Word
Once you’ve placed your link into the static profile of your social media, the next step is to share it via status updates, tweets, video uploads, or images.
Get creative with this. Obviously Facebook and Twitter will make for great status updates, but you can use Pinterest to upload an image that links back to your site.
You can upload a video to your YouTube channel and place a link in the video’s description.
Here’s an unknown way of spreading the word: use social bookmarking sites like StumbleUpon and Delicious as well. You can submit your site to these and people can, for lack of a better term, stumble upon them.
How to Get Your New Website or Blog Discovered ASAP
7. Create an RSS Feed, submit it to Feedburner
Google has an RSS management tool called Feedburner. This tool allows you to submit your blog through a Google-owned program
By submitting to this, you can inform Google every time you add a new post to your site. When you visit the site, you can place your URL into the field or your current RSS feed link into it.
8. Submit your site to Niche Blog Directories
They’re out there, trust me. You’ve just got to find them.
There are numerous blog directories out there that are specifically tailored to your niche. Brian Dean sheds some light on the topic over at Smart Passive Income.
Submitting your RSS feed or blog URL to these sites will grant you additional incoming links and minimal traffic. It’s worth your time to dig up a few niche directories to submit your website to.
But please, keep it related. Don’t target a directory that has to do with plumbing if you’re in the gardening niche.
Matt, I Need Help! It’s Not Working!
Wow, okay, just take a deep breath.
There are billions upon billions of websites out there that have yet to be indexed by Google.
After all the work search engines have done, less than 10% of the internet has yet to be indexed.
Doing the math, that means that what has yet to be touched, also known as the “Invisible Web” accounts for about 450 billion pages.
Think of that like a haystack, and your website is the needle. Yeah, this isn’t easy by any means.
Luckily, Neil Patel over a QuickSprout has a phenomenal infographic that provides some obstacles that keep your site from being indexed. Let’s talk about them, and how you can obliterate anything stopping you from greatness.
1. You Don’t Have a Robot.txt file, or it’s Not Configured
This concept was developed in 1994 when the Robots Exclusion Standard was created. This handy file instructs the spiders not to touch certain areas of your site.
These could be areas that the public typically doesn’t see like image folders or PDF files, or they could be static pages like 404 error images and so on.
These pages aren’t needed to judge the relevancy of your website, so there’s no reason to index them. You can create the folder manually or, as usual, there’s a WordPress plugin for that.
2. Incorrectly Written Title Tags and Meta Descriptions
When it comes to Google’s results, the title you see and the description below it are both customizable.
One of the top WordPress plugins I recommend actually helps you configure these things as part of your SEO optimization. Without assistance like that, you may not configure them correctly and hinder your chances of being indexed as a result.
The title tag for example should be up to 70 characters long and contain words that are relevant to your topic or main keyword of discussion.
The meta description can be a maximum of 155 characters that includes a call-to-action before it will be cut off on Google’s display.
3. Configuring your URL Parameters in Webmaster Tools
The concept of URL parameters comes up when we’re discussing content that is accessible in a number of different ways.
The best example is when you’re on a website and you search for something.
Those results can also appear as a result of other searches depending on the filters and keywords the user has.
In other words, it can appear like you have duplicate content on your site to Google, which is a no-no.
Using the Webmaster Tools I’ve mentioned several times thus far, you can set URL parameters to inform Google’s spiders how to best crawl your site when it comes to these types of pages.
The search engine themselves even advises that you configure these parameters properly. Otherwise certain pages will be dropped from the index.
4. You Have a Low Page Rank
Matt Cutts from Google has said The number of pages Google crawls is roughly proportionate to your pagerank. How do you combat a low page rank?
Well, for starters check your backlinks and your external links. If your connecting your blog out to low ranking sites, they’re dragging you down with them.
Even worse, if these sites are no longer hosted or they have expired domains, your putting even more pressure on your own blog. In addition, some of the aforementioned things like robot.txt files and poor SEO optimization can drag down your site’s pagerank.
Remember that all of these things connect and intertwine to improve (or lower) your page ranking.
5. Poor Hosting Service/Connectivity
This one may escape the gaze of many people, but not all hosting services are created equal.
Having poor connectivity or DNS issues can most certainly affect Google’s ability to index you, simply for the reason that your site is too slow or poorly maintained to even get in and crawl it.
If you’re purchasing a domain, make sure to research it and see if the name has ever been associated with things like spam, private blog networks, or other black hat schemes to avoid that issue.
You may wondering which hosting service is good to use. I would recommend checking out WebHostingHub. They have fantastic prices and great service.
Plus, On Blast Blog readers get a special discount because everybody knows Matt Banner and more importantly my brother Bruce (A.K.A The Hulk).
Once you’ve set up a blog and purchased your hosting, the next step is to build it up and find ways to have it indexed by Google and other search engines. With these above obstacles out of the way, you should have no problems doing just that.
It’s Time to Shine!
Now when someone asks you how to get Google to index a site, you’ll have a great place to refer them to.
Or you can tell them too, just remember to drop a line about how the Banner man gave you the info.
You can also just refer to me by my real name, but Banner Man also works pretty well if you ask me.
This is just the beginning folks. Getting your name on Google’s V.I.P list is great, but it’s among a massively growing crowd of, oh, about 14 billion. That’s not to discourage you, that’s to give you something to shoot for once you’ve made your way into their database.
Thanks for reading everyone, and don’t forget to tell me about your experiences getting indexed in the comments below!