Is there a way for me to share a form across different accounts without publishing the form publically?

We do not have private forms, it means that everyone with a form link will be able to access it.

You may consider the following sharing options:

1) Forms can be cloned by their URLs.

How to Clone an Existing Form from a URL

2) Other account can be added as a Sub User Account with form access permissions.

How to Share Forms with a Sub-Account User

3) JotForm Support can transfer the forms between accounts.

If you need any further assistance, please let us know. We will be happy to help.