Select Page
Web Masters: does your robots.txt file check out?

Web Masters: does your robots.txt file check out?

Sep 12, 2017

I discovered a new tool today and in discovering it, I found that I needed to make some corrections to some of my sites. The file robots.txt that sits quietly at the root of most web sites tells spiders what they can index and what they cannot. All responsible search...