How to prevent loading issues caused by Googlebot being blocked by robot.txt?