What is the setup fee referred to in the following error message. How is it calculated? How do I know what value I should put in the first payment box

You should fill this text box when you want to charge your customer a higher quantity than the regular payment, for example, if you want to charge $50 additional and the regular price is $100 then this text box should  be filled with $150, the $50 will be charged as a fee for the setup of the subscription. 

If you will not charge your customer an additional fee to the regular price then you should not change this value.

You will find the details about this and how to set up recurring payments on this guide: https://www.jotform.com/help/166-How-to-setup-a-recurring-payment-option. 

Hope this helps.