A bit has been flipped on Google Search. Previously, the Googlebot would index any web page it came across, unless a robots.txt file said otherwise. Now, a robots.txt file is required in order for the Googlebot to index a website. This p...