How to prevent loading issues caused by Googlebot being blocked by robot.txt?

  • Profile Image
    BDAVID
    Answered on April 07, 2017 at 03:03 PM

    I just checked your form and the header log is showing:

    Let us know if you need more help, we will be glad to assist you.

  • Profile Image
    give2kids
    Answered on April 07, 2017 at 04:36 PM

    Hello - I appreciate your reply, but I asked why Google 's system said that robot.txt was blocking my header -- not whether or not it was showing.

    I just checked it (their Mobile-Friendly Test)) again, and it's still showing that:

    https://search.google.com/search-console/mobile-friendly?id=39pm6sLRBtLC0KgFEZz5WQ&view=fetch-info

    Please let me know why. Thank you.

  • Profile Image
    BDAVID
    Answered on April 07, 2017 at 06:26 PM

    I do not have too much knowledge of Robots.txt files. However after doing some research, I found that that it is a text file a webmasters creates to instruct robots (typically search engine robots) which areas of the website should not be processed or scanned.

    That message shows up for you, since our developers have restricted web robots to access or scan certain files such as submissions, uploads, and images for security and privacy reason. But that does not mean you form is not functional or there is something wrong in it. 

    Let us know if you need more help.

  • Profile Image
    give2kids
    Answered on April 07, 2017 at 07:03 PM

    Thank you -- but I know what robot.txt files are.

    I understand that the image is showing. That is not my concern. The key point here is that Google's message is: "This can affect how Google sees and understands your page"

    This could mean that it's affecting our ranking.

    Please forward my question to one of your co-workers who are familiar with robot.txt files.

    Thank you.

  • Profile Image
    Mike_G
    Answered on April 08, 2017 at 03:57 AM

    We would like to apologize for any inconvenience this is causing you.

    To my understanding, you have uploaded an image from your computer to the image field in your form.

    And that produced the image link, https://www.jotform.com/uploads/give2kids/form_files/new-mobile-form-header.798.png. Robot.txt is not allowing any access to that link from any search engine bot that is why it's getting loading issues in your test.

    Although this should not affect your page rank, a workaround to prevent this loading issue is to upload your image to your own server or any image hosting site and use the new link to the image when you upload it to your form. This will not produce a link that has a path (directory) disallowed in the robot.txt file.

    I hope this helps. If you have any further questions regarding this, please feel free to contact us again anytime.

    Thank you.

  • Profile Image
    give2kids
    Answered on April 08, 2017 at 01:03 PM

    Thank you for your helpful answer, Mike_G. I will do as you suggest.

  • Profile Image
    Mike_G
    Answered on April 08, 2017 at 03:31 PM

    You're most welcome and I'm pleased to know that it helped.

    If, in any case, you still encounter any issues, please feel free to get back to us and we will be glad to help you further.

    Thank you.