For many websites, search engines are the largest source of traffic. Because of this, when someone asks, “how to discourage search engines from indexing this site,” you’re probably wondering why they’d want to do that. Let’s explore some of those reasons today.
Common Reasons to Block Content from Search Engines
Many people think that if other websites aren’t linking to your domain, search engines won’t be able to find your site. Even if that’s the case, here are some ways your website can still be found:
- Someone previously owning your domain and still having links pointed to the site.
- Domain search site results could be indexed with your link on them.
- Web pages that contain a list of site names with yours included.
And here are some reasons as to why you might consider how to discourage search engines from indexing this site of yours:
- (Purposefully) duplicate content. The importance of not having duplicate content cannot be overstated enough; it’s a Google penalty that can get your website de-indexed from the search giant. If there’s a need to publish duplicate content (for example, an advertising landing page), consider adding a canonical link to the page in question. If you find yourself in a position where you need to publish an entire website of duplicate content, it’s best to determine how to discourage search engines from indexing this site (the whole thing).
- Creating a development or test website. Before debuting a new WordPress theme, it’s best to create a duplicate of the current website to work on for development purposes. No cowboy coding!. You don’t want your website’s duplicate copy to compete with you for ranking—or ruin your chances of ranking, completely.
- Private content. If you run a membership site, you don’t want paid content to be indexed by search—it’s intended only for paying members!
- Dynamic content. If you have content that frequently changes, you’d want to block search engines from indexing content that might soon be outdated. It takes some time for search engines to index updated content so if yours changes too fast for the search engine to keep up with, it’s better to block search engines from indexing the already outdated content.
How to Block Search Engines from Indexing Your Site
There are several ways to prevent your site from being indexed, but first, let’s clear the air in terms of what that really means.
Many people confuse how websites being “listed” differs from websites being “indexed” on Google. Here’s how it breaks down:
- Indexing: This describes the process of downloading a site’s content to a search engine’s server to add to its index. Check if your site is indexed by typing site:url (where “url” is your website address). You’ll see how many of your pages are indexed. If you find that your website is indexed, this is generally considered to be a good thing. It means search engines can crawl your site to find content. It takes about a week to up to four weeks for search engines to update search results, which is why there’s a chance your website isn’t showing/ranking yet.
- Ranking/Listing/Showing: Your website shows up in the search engine result pages. Because your site is showing up in the SERPs (search engine results pages), it means your site is.
In a nutshell, ranking means your site is indexed but having your site indexed does not automatically mean that it will rank/show on the SERPs.
Note also that a site does not have to be listed to be indexed.
Now, the answer to “How to discourage search engines from indexing this site”?
There are a few different ways to find success if you don’t want Google to index your website:
1. The WordPress Search Engine Visibility Checkbox
You can discourage search engines from indexing your website from within the WordPress dashboard.
Go to the WordPress admin area. Select Settings > Reading:
Go to the Search Engine Visibility option and there you’ll see the WP search engine visibility checkbox. Go ahead and check the box that says Discourage search engines from indexing this site. Be sure to click Save Changes.
Here’s how it works:
WordPress edits the robots.txt file and adds a line to your website header that discourages search engines from indexing your website, as such:
<meta name=’robots’ content=’noindex,follow’ />
Note that although most search engines respect this request, some search engines may choose to ignore it. Even after doing this, there may be some pages and especially images from your website that may still show as being indexed.
2. Editing the Robots.txt File
Implementing the first option to discourage search engines from indexing this site, WordPress takes care of editing the robots.txt file for you.
Another option? Adjust your robots.txt file manually!
Here’s how:
Access your website files through your File Manager or FTP client and look for the robots.txt file. It is usually in the same folder as WordPress, under the public_html file. If there is no robots.txt file, create a new blank one.
Then, enter this syntax, which prevents search engines from crawling your site:
User agent: *
Disallow: /
3. Password Protecting Your WordPress Website
Another way to prevent search engines from indexing your site is to password-protect it, as search engines and web crawlers do not have access to password-protected websites.
To do this, you can either use cPanel or a password protection plugin.
Using cPanel
Most web hosts use cPanel to manage your hosting account.
To password-protect your website through cPanel, first log into your cPanel dashboard and go to Password Protect Directories.
Next, choose the directory you want to protect. For WordPress, it is usually located in the public_html folder or the www directory.
Select the Password Protect This Directory option, choose the name for the directory you want to protect, and then save.
After doing this, create a user account that will be able to access the protected directory. Provide a username and password, then save.
Using a Password Protection Plugin
There are also several WordPress plugins that allow you to password-protect your website.
As any good WordPress developer will tell you, adding plugins to fix small issues can cause more harm than good. Each new plugin can negatively impact page load speed and introduce new security vulnerabilities. When possible, consider one of the previous options to discourage search engines from indexing this site.
If you’re not as adept in your WordPress website’s backend, consider the following plugins:
Here are some of the top options.
- Password Protected. A simple plugin that allows you to protect your WordPress site with a single password. Note that this plugin only protects your WordPress site’s content and not your images and uploaded files. If someone enters an exact URL to an image file, it will still be accessible to them when using this plugin.
- iThemes. iThemes Security is a robust security plugin that does so much more than protect your passwords. We know from firsthand experience, considering we partner with them! Benefit from features like forced secure passwords, brute force protection, file change detection, database backups, and more.
- Hide My Site. This plugin allows you to use a single password to protect your WordPress site and blocks users that attempt multiple logins.
Removing an Indexed Website from Google
If your website has already been indexed and you want it removed from search engines—don’t worry. It’s not too late.
To remove an indexed website from Google, you’ll have to first set up Google Search Console.
Once set up, go to your newly-added website and click on Google Index > Remove URLs.
Enter the webpage URL you want to be removed and press Continue.
Choose the Temporarily hide page from search results and remove from cache and click on Submit Request.
What this does is temporarily remove the URL from search results (about 90 days). If this is the way you plan to discourage search engines from indexing this site, it’s best to also apply any of the more permanent methods detailed above.
Fully Allowing Google to Index Your Site
While there may be reasons why you don’t want Google crawling and indexing your website’s pages and posts, there may come a time when you change your mind.
To make sure Google is allowed to index your websites, there are no major adjustments you have to make after you publish content on your website. Google will automatically crawl these pages and index them.
However, if you’re having trouble getting your pages indexed in Google, you’ll want to go through all the steps we just went through to deindex your website and make sure none of them are active. If they are, you may be discouraging Google from indexing your site even if you don’t want to.
Final Thoughts: How to Discourage Search Engines from Indexing this Site
While most people welcome search engine traffic to their WordPress websites, there are others that may need to hide their entire website. Certainly, there are many different reasons to want to prevent content from being indexed by search engines like Google.
Luckily, there are a number of different tactics you can employ to find an answer to the question, “how to discourage search engines from indexing this site?” From working in your website’s backend files, to just clicking a button—or using a plugin—you honestly don’t have to be a web developer to fix this problem on your own.
Want to give your feedback or join the conversation? Add your comments 🐦 on Twitter.
