Do you know that search engine optimisation (SEO) can also be done by having a neat robots.txt?
For your information, robots.txt is used to tell search engine robots about what files or folder on your website to be crawled so that certain pages consequently get indexed in the search engines.
To know what pages have been indexed by a search engine, type site:yoursite.com in the search box. A Google search on this website shows there is a total of 677 indexed pages.
The more indexed pages you have, the greater the possibility that visitors would stumble upon your blog as they searched through certain keywords.
For those using WordPress content management system, you may adapt or adopt my version of robots.txt:
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*# Adsbot-Google
User-agent: Adsbot-Google
Disallow:
Allow: /*#Googlebot Mobile
User-agent: Googlebot-Mobile
Disallow:
Allow: /*# Disable Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /# Disable digg mirror
User-agent: duggmirror
Disallow: /Sitemap: http://mrdefinite.com/sitemap.xml
Sitemap: http://mrdefinite.com/sitemap.xml.gz
Basically the robots.txt file allows the “googlebot”, which is the search engine bot of Google, to retrieve and index every page from my site except for directories “cgi-bin”, “wp-admin”, “wp-includes”, and so forth.
Google user-agents allowed here are Googlebot-Image, Mediapartners-Google, and Googlebot-Mobile.
Googlebot-image Googlebot-Mobile crawl pages for image index (images.google.com) and mobile index (m.google.com) respectively. Mediapartners-Google, which only oncerns Adsense users, crawls pages to determine Adsense content.
A robots.txt file should be placed at the root of the domain, e.g., mrdefinite.com/robots.txt.
35 comments | Apr 30, 2010 | 44 views |
| |
Pardon me for such self-promo, but if you’re interested in doing link exchange with this PR3 blog — especially those who blogs are MMO/SEO-related, I’d be elated to do it with you!
But if your blog category falls outside MMO/SEO, I’m still more than happy to exchange link with you, but with the conditions that the link to my site is placed your mainpage blogroll and in “dofollow” attribute.
Here are some benefits of exchanging links, especially with blogs with high PR (PageRank):
- As you get more backlinks to your site, especially those from high PR sources, it will increase your blog’s PR.
- In turn, this increases your search engine results pages (SERP’s). In other words it makes your website/blog more relevant in search engines.
- Finally your web traffic will also steadily level up, thereby achieving greater a Alexa rank.
Your link will be placed under “Great Read” on the left sidebar of the MAINPAGE.
Definitely win-win situation at its best!
If you’re interested, kindly send me an email after doing the linking.
When linking to this blog, kindly make sure the details are as follow:
Anchor text: Make Money Online
URL: http://mrdefinite.com
Description: THE FIRST EVER entertainment-based, make money online cum personal blog. Make money online tips and tricks explained.
Kindly provide the reciprocal URL in the contact form once you are done linking to my site. Don’t forget to include the details of your blog or website too!
Apr 25, 2010 | 84 views |