The problem of anonymity on the Internet has been repeatedly raised on various forums. Presently there is a plenty of technologies allowing to hide your IP address, location, and data on domain registration. It helps to remember about bots scanning the Internet and collecting data on websites. Bot technology is exploited by Google search engine, Bing, Yahoo, and such web analytics systems as MOZ, Google Analytics, Semalt, etc. A website owner can prohibit indexing pages via robots.txt file or edit .htaccess file. For that end it is necessary to indicate the user agent to set configurations for and the list of webpages. As a rule, archived sections, user profiles, and other supplementary pages come under indexing restrictions. They must not influence ranking.
It is necessary for a search system to index web pages correctly. According to Google requirements, relevant content should be placed on a website. In other words, if you sell cars, articles about evening dresses hardly match your website. A search system will detect keyword manipulations and block such webpage for a long time.
Before disabling bots from visiting a web page, think if this page can be of any help with promotion in search results. Google forms search results on the basis of indexation of web pages. Accordingly, blocked pages will never get positions in search, and Internet users will never know about their existence. Besides, a large number of pages closed for bots raises suspicion of Google.