A robots.txt is a text file kept on your website’s home directory i.e., Public_HTML. The file is checked by all search engines spiders before they start crawling your website’s content to know which files, file types and directories are not allowed to be archived. So by using a Robot.Txt file you can exclude content you don’t want to appear in search engine results. Even it allows you to block the way for search engine crawlers including the safe ones like Google.

Continue reading Easiest Way Of Generating A Robot.TXT File For Your Website →