robots.txt Configuration

The robots.txt configuration is considered a best practice, but unfortunately is not respected by the bots that perform crawling...
In any case, here are the configuration details:

• Create the robots.txt file in the root of the web server:
  That is, it should be available at http://yoursite.com/robots.txt


• robots.txt file content
  # defines that any connection with the user-agent defined (browser signature) will be handled by robots.txt User-agent: *
  # prevents (or attempts to prevent) the crawler/bot from analyzing the /includes/ directory
  # That is, the bot will not analyze the url http://yoursite.com/includes/
  Disallow: /includes/
  # prevents (or attempts to prevent) the crawler/bot from analyzing the /CHANGELOG.txt file
  # That is, the bot will not analyze the url http://yoursite.com/CHANGELOG.txt
  Disallow: /CHANGELOG.txt

There are other rules that can be defined, more details at the link below

References
About /robots.txt