How can I notify and direct customers to my website if the payment is unsuccessful?

If someone completes the form, goes on the Paypal payment page and then changes his mind, or payment is unsuccessful, how can let him know and direct him back to my website?

....how can I let him know and direct him back to my website?

In the payment page, there is no cancel button. If the user decided to go back or close the browser, that particular submission will be added to your incomplete payment list. Here's how you can check that:

http://www.jotform.com/help/564-how-to-view-incomplete-payments-of-your-payment-forms/

I would also suggest checking our incomplete payment FAQ: http://www.jotform.com/help/272-Paypal-Incomplete-Payments-FAQ.

Letting your customer know of the incomplete payment is possible. Here's how:

http://www.jotform.com/help/273-How-to-setup-Incomplete-Payment-Notification-E-mail (Select "All Emails").

However, redirecting your customer back to your website or thank you URL is not possible.

Thanks