There are many scenarios when someone wants to stop search engines from crawling and indexing his website. For instance: If you want to use your website just as an Android app. In this lesson, you will see how easily you can stop search engines from crawling your WordPress website.
The easiest way is to login to your WordPress website then navigate to admin area dashboard Settings -> Reading and select ‘Discourage search engines from indexing this site ‘.
If the option is ticked, WordPress automatically adds one line to the header of your website:
<meta name='robots' content='noindex,follow' />
Other than this it add some lines to your website’s robots.txt file:
User-agent: * Disallow: /
These lines simply ask robots i.e. web crawlers and spiders not to index the pages of your website. However, it is up to search engines to accept this request or ignore it. Most search engines respect user privacy and stops indexing the website after this request, but not all.
If you are not using WordPress then you can manually add the given lines in the header and robots.txt file of your website.
For more effective result, it would be better to password protect your web directory via cPanel. All SANGKRIT.net hosting options (Linux) provides you cPanel to manage your website.
Now what you have to do is login to your hosting option, access cPanel and scroll down to ‘Security’ section then click ‘Password Protect Directories’ icon. This will open a popup.
From the popup – If your are hosting your website on root then select the first option i.e. ‘Web Root’ else use dropdown box to select document root folder for your domain or subdomain website.
Make sure ‘Password protect this directory’ option is checked then hit the ‘Save’ button. Now next step is to add a username and password, you can assign it there itself.
The created username and password would be required to access your website in future. Once password is set, your complete website will go password protected, and no one including the search engines and bots will be able to access your website.
Good you stated it that it left for search engine to ignore or accept the robot txt. From my experience I still see some my pages on google even after prevent their bot from indexing the pages.
Sometimes the search engine doesnt obey the robot txt. And the pages you try to prevent from showing on SERPs will be there eg archive pages.