Decoding form submission data sent to webhook - delimiter issue.

  • genkijacs
    Asked on August 31, 2017 at 7:48 AM

    When using a webhook, when the form is submitted, the form-data contains a field called "pretty" which contains the questions and answers from the jotform.

    The problem is the data is delimited thus:

     

     Question 1 text:Answer 1 text, Question 2 text:Answer 2 text, Question 3 text:Answer 3 text

     

    All well and good until either the question or answer text contains ':' or ', '. Then it is impossible to decode the data since these are not escaped when appearing in the text. So it can end up like:

    Roughly speaking, what is your answer?:I think maybe, I would say perhaps: yes

    and whereas this is one question and answer, using the delimiters prevents it decoding as such!

     

    Decoding the rawRequest data is not really an option since it does not contain the text f the question but instead some sort of field identifier (I guess)

     

    Is it possible to escape the delimiter characters when they appear in a question or answer? Or format the data in some way so that my webhook page can decode the questions and answers from the POST?

     

    Thanks

  • Ashwin JotForm Support
    Replied on August 31, 2017 at 10:10 AM

    Hello genkijacs,

    If you meant to say that you want to change the formatting of the output POST data of form, I am very sure that won't be possible but let first we would like to test the issue you are having.

    With which form you are having this issue with? I did check few of your last edited form but couldn't locate any form field "pretty". Is this question a widget or checkbox question? Please let us know the form and we will take a look. 

    Are you also able to capture the raw post data? Is it possible for you to share it with us?

    We will wait for your response.

    Thank you!

  • Andrew Milford
    Replied on September 1, 2017 at 6:29 AM

    Thanks for your response.

    The area of interest is n the following response is

    form-data; name="pretty" 

    You can see this contains the questions and answers delimited by ':' and ','

    In this block is a question with a ',':

    Roughly speaking, about how many kanji do you know?:0,

    So as you see there is a ',' in the question and it is used as a delimiter with no indication of which is which. So if the answer contains a ',' there is no way to delimit correctly.

     

    Here is the complete data sent to our server via the webhook:

     

         * --------------------------101697030b0f44a4  Content-Disposition: form-data; name="formID"    51751518647966  --------------------------101697030b0f44a4  Content-Disposition: form-data; name="submissionID"    3799155056828380881  --------------------------101697030b0f44a4  Content-Disposition: form-data; name="webhookURL"    http://genki.iq2u.net/Jotform/Beginner1WrittenTest  --------------------------101697030b0f44a4  Content-Disposition: form-data; name="ip"    86.140.32.86  --------------------------101697030b0f44a4  Content-Disposition: form-data; name="formTitle"    2016-04 - Beginner 1 written test - GENKI  --------------------------101697030b0f44a4  Content-Disposition: form-data; name="pretty"    Full Name:Andrew Milford, If you know the Japanese pronunciation of your name, please write it here (in katakana or romaji)::, E-mail:ajmilford@gmail.com, Please indicate your status::I am taking this test to check my own ability level, I will start on::  , I can read hiragana:No, I can read katakana:No, 1.　リーさん＿＿＿中国人（ちゅうごくじん）です。:か, 2.　「すみません、おてあらいは＿＿＿ですか。」:だれ, 3.　＿＿＿に国（くに）へ帰（かえ）りますか。:いつ, 4.　私（わたし）はあまりざっしを＿＿＿。:よみます, 5.　家（いえ）の中（なか）にいもうとが＿＿＿。:あいます, 6.　昨日（きのう）はあまり＿＿＿。:さむいです, 7.　毎日（まいにち）音楽（おんがく）を＿＿＿、寝（ね）ます。:きって, 8.　吉田（よしだ）さんは今（いま）お寺（てら）の前（まえ）で写真（しゃしん）を＿＿＿。:とっています　○, 9.　私（わたし）の学校（がっこう）はにぎやか＿＿＿たのしいです。:だ, 10.　山田（やまだ）先生（せんせい）はとても＿＿＿と思（おも）います。:やさしくないです, 11.　図書館（としょかん）ではたばこを＿＿＿ください。:すわなくて, 12.　＿＿＿薬（くすり）を飲（の）んでいません。:まだ　○, 13.　私（わたし）が昨日（きのう）＿＿＿辞書（じしょ）はどこですか。:買（か）って, 14.　バスとタクシーと＿＿＿はやいですか。:どれが, 15.　明日（あした）学校（がっこう）に＿＿＿つもりです。:行（い）く　○, 16.　いつかイタリア語（ご）が＿＿＿＿。:ならいに行（い）きます, 17.　「このケーキはどうしたんですか。」「今日（きょう）は友（とも）だちのたんじょう日（び）＿＿。」:そうです, 18.　今日（きょう）は雨（あめ）だと言（い）っていたから、かさを＿＿＿ほうがいいよ。:持（も）っていく, 19.　日本語（にほんご）で手紙（てがみ）が＿＿＿。:かけます　○, 20.　これはお姉（ねえ）さんの写真（しゃしん）ですか。＿＿そうな人（ひと）ですね。:やさし　○, 21.　ともだちが私（わたし）にプレゼントを＿＿＿＿。:くれました　○, 22.　頭（あたま）が痛（いた）いんですか。薬（くすり）を＿＿＿＿。:飲（の）んだらどうですか　○, 23.　今度（こんど）の旅行（りょこう）はアフリカに＿＿と思（おも）っています。:行（い）く, 24.　来週（らいしゅう）はテストですから、勉強（べんきょう）して＿＿ください。:あげて, 25.　この本（ほん）は父（ちち）が買（か）って＿＿ました。:もらい, 26.　すみません。ちょっと教（おし）えて＿＿＿＿。:みますか, 27.　新聞（しんぶん）によると、今年（ことし）日本（にほん）は＿＿＿そうです。:あつい　○, 28.　日本（にほん）に＿＿＿、たくさん勉強（べんきょう）します。:行（い）く時（とき）が, 29.　エアコンが＿＿＿いませんね。つけてください。　:つく, Roughly speaking, about how many kanji do you know?:0, If you've passed any Japanese exams, tell us which and which level (e.g. JLPT N2):none, How many years have you studied Japanese?:0, How did you study Japanese?:none, What textbooks are you currently using/have you used in the past? Please list them below and include up to which chapter you studied, if you did not finish the whole book.:none, Any comments or questions?:this is a test by IQ, How many minutes did it take to complete this test?:1  --------------------------101697030b0f44a4  Content-Disposition: form-data; name="username"    genkijacs  --------------------------101697030b0f44a4  Content-Disposition: form-data; name="rawRequest"    {"slug":"submit\/51751518647966\/","q6_fullName6":{"first":"Andrew","last":"Milford"},"q7_ifYou":"","q8_email8":"ajmilford@gmail.com","q57_pleaseIndicate57":"I am taking this test to check my own ability level","q11_iWill11":{"year":"","month":"","day":""},"q13_iCan":"No","q16_iCan16":"No","q19_1":"\u304b","q20_2":"\u3060\u308c","q21_3":"\u3044\u3064","q22_4":"\u3088\u307f\u307e\u3059","q23_5":"\u3042\u3044\u307e\u3059","q24_6":"\u3055\u3080\u3044\u3067\u3059","q25_7":"\u304d\u3063\u3066","q26_8":"\u3068\u3063\u3066\u3044\u307e\u3059\u3000\u25cb","q27_9":"\u3060","q28_10":"\u3084\u3055\u3057\u304f\u306a\u3044\u3067\u3059","q29_11":"\u3059\u308f\u306a\u304f\u3066","q30_12":"\u307e\u3060\u3000\u25cb","q31_13":"\u8cb7\uff08\u304b\uff09\u3063\u3066","q32_14":"\u3069\u308c\u304c","q33_15":"\u884c\uff08\u3044\uff09\u304f\u3000\u25cb","q34_16":"\u306a\u3089\u3044\u306b\u884c\uff08\u3044\uff09\u304d\u307e\u3059","q35_17":"\u305d\u3046\u3067\u3059","q36_18":"\u6301\uff08\u3082\uff09\u3063\u3066\u3044\u304f","q37_19":"\u304b\u3051\u307e\u3059\u3000\u25cb","q38_20":"\u3084\u3055\u3057\u3000\u25cb","q39_21":"\u304f\u308c\u307e\u3057\u305f\u3000\u25cb","q40_22":"\u98f2\uff08\u306e\uff09\u3093\u3060\u3089\u3069\u3046\u3067\u3059\u304b\u3000\u25cb","q41_23":"\u884c\uff08\u3044\uff09\u304f","q42_24":"\u3042\u3052\u3066","q43_25":"\u3082\u3089\u3044","q44_26":"\u307f\u307e\u3059\u304b","q45_27":"\u3042\u3064\u3044\u3000\u25cb","q46_28":"\u884c\uff08\u3044\uff09\u304f\u6642\uff08\u3068\u304d\uff09\u304c","q47_29":"\u3064\u304f","q50_roughlySpeaking":"0","q51_ifYouve":"none","q52_howMany":"0","q53_howDid":{"other":"none"},"q54_whatTextbooks":"none","q55_anyComments":"this is a test by IQ","q56_howMany56":"1","event_id":"1504105637118_51751518647966_dtWLJ30"}  --------------------------101697030b0f44a4  Content-Disposition: form-data; name="type"    WEB  --------------------------101697030b0f44a4--  
      

     

  • liyam
    Replied on September 1, 2017 at 9:08 AM

    It took me some time to figure out what you achievement is being tried. So if I'm not mistaken, you would like the pretty format because it contains the full question, and you want to get that question coming from the form and you do not want to manually create the question and simply match an answer to it.

    Find a pattern

    First matter that we should do here if we need to find a pattern. What are the instances where you get a new question after being separated by a comma? What we have is a colon ':'.

    So what you need to do is pick up the character values after a comma, only if before that there is a matching colon found. If there is an instance of a comma and a comma before it, then it is not a separate question.

    So that's how your algorithm for your script will be in making your elements be your question. Do you have an existing code which you can share that you are using for your webhooks?

    Thanks.

     

  • Andrew Milford
    Replied on September 1, 2017 at 12:22 PM

    Ok, consider a question

    "how are you today, are you doing ok, respond yes, if so:no, if not: or maybe?"

    and the answer

    "I  am sort of ok: but could be better, maybe when the sun comes out: or not"

    So this would be encoded in the webhook as

     

    how are you today, are you doing ok, respond yes, if so:no, if not: or maybe?:I  am sort of ok: but could be better, maybe,  when the sun comes out: or not, 

     

    I know this seems a bit contrived and is bad English and punctuation - but an algorithm needs to cope with whatever is thrown at it!

     

    Scanning along the line I think it would decode as:

    Question "how are you today, are you doing ok, respond yes, if so"

    answer "no"

     

    Question "if not"

    answer " or maybe?:I  am sort of ok: but could be better"

     

    Question "maybe, when the sun comes out"

    answer "or not"

    Thanks

  • Kevin Support Team Lead
    Replied on September 1, 2017 at 1:33 PM

    May you please try using the WebHooks integration through Zapier, it can be found here: https://zapier.com/zapbook/jotform/webhook/ 

    Please let us know how it goes.