How to prevent loading issues caused by Googlebot being blocked by robot.txt?

  • give2kids
    Asked on April 7, 2017 at 1:50 PM
  • David JotForm Support Manager
    Replied on April 7, 2017 at 3:03 PM

    I just checked your form and the header log is showing:

    How to prevent loading issues caused by Googlebot being blocked by robot Screenshot 20

    Let us know if you need more help, we will be glad to assist you.

  • give2kids
    Replied on April 7, 2017 at 4:36 PM

    Hello - I appreciate your reply, but I asked why Google 's system said that robot.txt was blocking my header -- not whether or not it was showing.

    I just checked it (their Mobile-Friendly Test)) again, and it's still showing that:

    https://search.google.com/search-console/mobile-friendly?id=39pm6sLRBtLC0KgFEZz5WQ&view=fetch-info

    Please let me know why. Thank you.

  • David JotForm Support Manager
    Replied on April 7, 2017 at 6:26 PM

    I do not have too much knowledge of Robots.txt files. However after doing some research, I found that that it is a text file a webmasters creates to instruct robots (typically search engine robots) which areas of the website should not be processed or scanned.

    That message shows up for you, since our developers have restricted web robots to access or scan certain files such as submissions, uploads, and images for security and privacy reason. But that does not mean you form is not functional or there is something wrong in it. 

    Let us know if you need more help.

  • give2kids
    Replied on April 7, 2017 at 7:03 PM

    Thank you -- but I know what robot.txt files are.

    I understand that the image is showing. That is not my concern. The key point here is that Google's message is: "This can affect how Google sees and understands your page"

    This could mean that it's affecting our ranking.

    Please forward my question to one of your co-workers who are familiar with robot.txt files.

    Thank you.

  • Mike_G JotForm Support
    Replied on April 8, 2017 at 3:57 AM

    We would like to apologize for any inconvenience this is causing you.

    To my understanding, you have uploaded an image from your computer to the image field in your form.

    How to prevent loading issues caused by Googlebot being blocked by robot Screenshot 30

    And that produced the image link, https://www.jotform.com/uploads/give2kids/form_files/new-mobile-form-header.798.png. Robot.txt is not allowing any access to that link from any search engine bot that is why it's getting loading issues in your test.

    Although this should not affect your page rank, a workaround to prevent this loading issue is to upload your image to your own server or any image hosting site and use the new link to the image when you upload it to your form. This will not produce a link that has a path (directory) disallowed in the robot.txt file.

    How to prevent loading issues caused by Googlebot being blocked by robot Screenshot 41

    I hope this helps. If you have any further questions regarding this, please feel free to contact us again anytime.

    Thank you.

  • give2kids
    Replied on April 8, 2017 at 1:03 PM

    Thank you for your helpful answer, Mike_G. I will do as you suggest.

  • Mike_G JotForm Support
    Replied on April 8, 2017 at 3:31 PM

    You're most welcome and I'm pleased to know that it helped.

    If, in any case, you still encounter any issues, please feel free to get back to us and we will be glad to help you further.

    Thank you.