Wordpress robots.txt tips against duplicate content -...
Rated • 2 reviews • seo, blogging, spiders • shoemoney.com
From the page: "Been getting some questions about my robots.txt file and what certain things do.
Thankfully some regular expressions are supported in the robots.txt (but not many).
$ in regex means the end of the file. So if you do .php$ it your robots.txt that means it will match anything that ends in .php
This is really handy when you want to block all .exe .php or other files. For example:"
