Month: September 2017

I discovered a new tool today and in discovering it, I found that I needed to make some corrections to some of my sites. The file robots.txt that sits quietly at the root of most web sites tells spiders what they can index and what they cannot. All responsible search engines respect what the robot.txt…