Do you know that search engine optimisation (SEO) can also be done by having a neat robots.txt?
For your information, robots.txt is used to tell search engine robots about what files or folder on your website to be crawled so that certain pages consequently get indexed in the search engines.
To know what pages have been indexed by a search engine, type site:yoursite.com in the search box. A Google search on this website shows there is a total of 677 indexed pages.
The more indexed pages you have, the greater the possibility that visitors would stumble upon your blog as they searched through certain keywords.

For those using Wordpress content management system, you may adapt or adopt my version of robots.txt:
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
# Adsbot-Google
User-agent: Adsbot-Google
Disallow:
Allow: /*
#Googlebot Mobile
User-agent: Googlebot-Mobile
Disallow:
Allow: /*
# Disable Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /
# Disable digg mirror
User-agent: duggmirror
Disallow: /
Sitemap: http://mrdefinite.com/sitemap.xml
Sitemap: http://mrdefinite.com/sitemap.xml.gz
Basically the robots.txt file allows the “googlebot”, which is the search engine bot of Google, to retrieve and index every page from my site except for directories “cgi-bin”, “wp-admin”, “wp-includes”, and so forth.
Google user-agents allowed here are Googlebot-Image, Mediapartners-Google, and Googlebot-Mobile.
Googlebot-image Googlebot-Mobile crawl pages for image index (images.google.com) and mobile index (m.google.com) respectively. Mediapartners-Google, which only oncerns Adsense users, crawls pages to determine Adsense content.
A robots.txt file should be placed at the root of the domain, e.g., mrdefinite.com/robots.txt.













































