 Share This
|
 seanmarler rated 16 months ago- From the page: "Been getting some questions about my robots.txt file and what certain things do.
Thankfully some regular expressions are supported in the robots.txt (but not many).
$ in regex means the end of the file. So if you do .php$ it your robots.txt that means it will match anythin... more
- Tags: robotics, seo
|
|
1 Reviews
-
-
- seanmarler rated 16 months agoseo, blogging, spiders
- From the page: "Been getting some questions about my robots.txt file and what certain things do.
Thankfully some regular expressions are supported in the robots.txt (but not many).
$ in regex means the end of the file. So if you do .php$ it your robots.txt that means it will match anything that ends in .php
This is really handy when you want to block all .exe .php or other files. For example:"
This page is not affiliated with shoemoney.com.
|
