It may sound crazy, but stick with me, because some times you really do want to keep search engines and on-site searches from finding certain pages. There is good news and bad news. The good news is that it's relatively easy to do. The bad news is that it requires a little caution and patience.
Before we get started, though, I know that last week I said I'd write about ways to better control your WordPress comments this week. However, that's taking a little more research than I realized it would, so watch for that coming up soon.
Let's start with why you might want to hide a page from search engines or even your own internal search functions.
Sometime you might want to put up special content pages like a free download of an ebook or something. Of course, most of the time when you provide this kind of content, you are doing so as an opt-in bribe. You want something, usually an email address, from the reader and in order to coax them into joining your email list, you give away a small freebie. If a user can search your website, they can find this freebie without giving up their sought after email address.
Also, you may have older content out on your site that you want to leave (for purposes of maintaining old links that might have been shared,) but now you don't want surfers to find this older content, you want them finding newer, updated content. So you might want to turn off the ability for Google to find those old pages. A prime example of this came from a conference organizer I spoke with a couple of months ago. They wanted to keep the information about their previous conferences on their website for historical reasons, but they didn't want those pages showing up in Google's search results because several readers complained when the featured speakers on that page weren't actually at their most recent conference. Now, I know what you're probably thinking: "That's the reader's fault for not understanding what was clearly written." While that might be the case, I thought it was also good that this organization was willing to take the extra steps necessary to make life easier on their readers.
Now that we have a couple of reasons why you might want to do this, let's get to the how.
Hiding Pages & Posts from Search Engines
There are two main ways of prevent search engines from including specific pages on your website. The first is by editing (maybe first creating and then editing) your robots.txt file. The second is by using metatags. For better or worse, WordPress does not directly allow access to either of these. Normally, WordPress handles these things for you and you don't need to manually manipulate these website features. However, for our purposes, you will. The good news is that there are tools to make this much easier on you.
Robots.txt
Robots.txt is simply a plain old text file that gives special instructions to the robots (or spiders, whatever you'd like to call them) that crawl across your website in order to index them for the search engines. To block a specific page using your robots.txt file, all you need to do is include two lines:
User-agent: *
Disallow: /<page-path-you-want-hidden>/
Most likely, the "User-agent: *" line will already exist. This tells the search engines that you want all of them to ignore this page. Otherwise you could specify only certain search engines. The "Disallow" line includes the path of the page you want them to ignore. So, if I wanted this page to be hidden, I'd have "Disallow: "/2015/04/06/how-to-hide-pages-from-search-listings-in-wordpress/". You don't need to include your domain in there.
So, how do you get a robots.txt file? Well, you can go into your cPanel file manager, create the file, and then edit it. Or, much simpler for most users, use the WordPress SEO by Yoast plugin (You are using that, right? It's awesome!), go to the SEO -> Tools menu and then click on File editor. From there you can click the Create button to create your robots.txt file and the plugin will automatically add your "User-agent: *" line and a "Disallow" line to block searching of your wp-admin page. Just add your new Disallow line and save the file.
Metatags
Metatags are snippets of HTML that you can include at the top of your pages to give special instructions to the search engines. Both ways work fine. Working with metatags can be difficult in WordPress but fortunately you have a plugin that will fix this, too. The Add Meta Tags plugin will allow you to directly create site-wide or page/post-specific metatags. I have not used this particular plugin but I understand that it works well. Follow the directions that the plugin provides and then add the following full meta tag to any specific posts or pages that you want the search engines to ignore:
<meta name="robots" content="noindex">
Be sure that you don't put that line in the site-wide metatags. Unless, of course, you want the search engines to ignore every page on your website.
Hiding Pages & Posts from Internal Searches
There are ways to do this without a plugin, but they are tedious and go further under the hood of WordPress than I normally like to take you. The plugin Search Everything not only allows you to hide certain pages from the internal search, it also gives you a dramatic boost in search capabilities by allowing you to include tags, categories, comments, approved comments, excerpts, and other fields in your readers' search results. It even includes a writing helper called Research Everything which allows you to search your own posts while writing new ones and quickly link to your own articles.
I'm not currently using Search Everything on this website, but I'm seriously considering adding it after having used it on another website.
The Quick Takeaway
Hiding pages is pretty simple. I'm not generally a fan of adding too many plugins to a website. WordPress SEO by Yoast is the first plugin that I always add. Contact Form 7 is the second. I'm seriously considering making Search Everything the third. Beyond that, I don't use too many others.
What are your favorite plugins? And have you had the need to hide any pages from searches?