Author

Profile Image Aytekin Tank is cofounder of Interlogy Internet Technologies. The company that created the popular JotForm web form builder. Developing web applications is his passion.

The companion blog to JotForm, the easiest web form builder.

About DDoS Attack on Friday

September 19, 2011
What Happened

On September 19th, Friday, 3:30 EST. We were hit by a massive DDoS attack. The DDoS attack (probably caused by our recent crackdown on phishers) was so massive and sudden that it took down our data center's firewall even before reaching us. In fact, our servers did not even see an increase in the load. We were down for several hours while we made changes on our system to make us more resilient to these kinds of attacks. We were back online fully before US woke up, but our European users felt the downtime worst.


Unfortunately, the changes we made required our IP addresses to be changed. This caused many users to not be able to access our site because of their DNS cache. In fact, many users kept reporting not being able to access our site in Saturday and even Sunday. We kept suggesting using jotform.net domain name temporarily on twitter.


We also had to move our SSL service from www.jotform.com to secure.jotform.com. The service we currently use for protection against such attacks needs to keep the security certificate on their servers. However, they cannot issue new certificates at this time because of an internal audit performed on their certificate provider. If you are using secure SSL forms, you must change the links on your site from https://www.jotform.com to https://secure.jotform.com.

UPDATE: (9/20/2011) Our certificate for www.jotform.com is now working.  


What We Learned From The Downtime

We have to run JotForm on multiple data centers. We actually knew this before and we were working on it. If the attack came a month later, the down time would have been much smaller. We are now running on multiple data centers and in case we lose one, we are ready to switch instantly.

We need to use a service that protects against DDoS attacks. This is the second time we had experienced such a DDoS attack. Since we fight against the phishers, we have to expect such attacks all the time in the future and be ready for them. That’s why we switched to a service that can protect against DDoS attacks.

We need to increase our fight against phishers. JotForm is so incredibly easy to use that even phishers prefer us for their forms. We think our recent crackdown on phishing has caused this attack. We will increase our efforts on preventing phishing.

We need to get you on Twitter. Our only way for communication was our Twitter feed and Facebook page when we lost connectivity, and we still had a lot of customers that don’t know what the heck happened to their forms.  Please, if you have a Twitter account, follow us on Twitter; or subscribe to our Facebook page.


What We Have Done About it

Since Friday we have been busy. We worked over the weekend to make sure we put everything in place before a new week starts. We are now more confident about our system.

We are now running a security proxy service in front of our firewall. They are specialized on preventing such attacks.

We are now running on multiple data centers. We are ready to instantly go live on another data center in case we lose  connectivity in our first data center.

We have a more robust DNS management system. We are now running on proxy IP addresses which means we can now change IP addresses for our servers and our users will not have similar DNS cache problems.

We have improved our Firewall and Load Balancer. We have a new firewall and load balancer for our app servers. This is actually costing us a lot, but our data center required it since losing shared firewall was not something they wanted to  experience again. On the positive side, having a dedicated hardware firewall and load balancers will probably improve the speed of our service.

We are notifying all users who use secure forms. Since we are not sure when our security proxy service will be able to add security certificate for jotform.com, we are sending an email today to all users who use SSL forms, and asking them to change the URL of their forms to secure.jotform.com until we are certificated again.

We are providing one month service refunds to all users who request it. We have always been trigger happy to provide refunds when one of our users is unhappy about a problem. So, we'd happy to provide full month's refund to any user who sends an email to billing@interlogy.com.

We are aware that you are putting a lot of trust in us by placing JotForm code in your site. That's actually a lot of pressure for us. :) Any problem on our site can create problems for your site and for your customers as well. That's why we worry a lot of about availability and we are continuously improving our systems.

We hope we will continue to deserve your trust.

You have already voted for this item. Your vote was: 1

16 Comments...
   
mikerastiello (September 19, 2011 at 07:49 AM)

Thanks for all your hard work. I noticed my forms were down around 7am CST, and by 10 or 11am they looked like they were coming back up.

The attack couldn't have come at a worse time, as we are using Jotform for our wedding RSVP system, and our guests were starting to get their invitations on Thursday and Friday!

Thanks not only for your tireless work, but being upfront and honest about what was going on and constant status updates on Twitter. It makes those of us who rely on web services trust the providers we use so much more.

   
Davor Tomic (September 19, 2011 at 08:04 AM)

Aytekin, thank you for a thorough report. I knew you were going to learn many valuable lessons from all this and continue to provide an ever greater service than before. Good luck and big thanks to the whole JotForm team!

   
bstolze (September 19, 2011 at 08:34 AM)

I checked on Facebook for updates and nothing was said about the jotform.net option. From my perspective, the site was down pretty much all day Friday in the USA. Just a suggestion...try to synch up communications between twitter and facebook. Thanks for the info.

See Answer for This Question...

   
chutchins (September 19, 2011 at 09:36 AM)

Is there a phone number to contact yall if something like this happens to get updates? Dr. Schlesinger, My Boss, was wanting a phone number just in case something happens like this that we can call to just get updates on when things will be up and running again.

See Answer for This Question...

   
getitleelocal (September 19, 2011 at 10:13 AM)

Awesome work, guys! We're happy to have you back.

   
ajkreiss (September 19, 2011 at 10:18 AM)

Did this attack in any way compromise the information we have collected via Jotform?

See Answer for This Question...

   
schlosser (September 19, 2011 at 07:39 PM)

We just wanted to thank you and all your staff. You were all so fast to correct this issue and provided near instant response to inquiries. You all are doing a great job. We will continue to entrust our forms to you and promote your site. Thank you again for your great service!

   
Amsinger (September 20, 2011 at 08:44 AM)

Due to the IP address change, do I need to update the code of my forms that are on my website so the forms submit properly?

See Answer for This Question...

   
corc_itweb (September 20, 2011 at 10:58 AM)

I am still having trouble with form submissions (connection time-out messages), and the site still seems to be down intermittently. It's not just me, right?

See Answer for This Question...

   
mmsellis (September 20, 2011 at 11:37 AM)

I'm still having trouble submitting forms. One second it works and the next it doesn't. Trying to use these forms for teacher observations and getting extremely frustrated that I continue to have these issues.

See Answer for This Question...

   
mcolon02 (September 20, 2011 at 05:19 PM)

Thanks for all the hard work. I am a premium member and rely heavily on Jotform for my start-up. I am not a techie and am not even really sure what a DDoS attack is. Which is testimony of how easy Jotform is to use... But sounds pretty bad and hope you'll fully recover from this soon. Most of all, I sincerely appreciate the transparency. Jotform totally rocks. Keep up the excellent work!!!

   
lebaus (September 22, 2011 at 04:20 PM)

Thanks for the speedy reply. I'm so glad your system is alright. You guys do an amazing job. I'm so glad I found you!

   
taflaten (September 24, 2011 at 09:24 AM)

We switched from "www.jotform." to "secure.jotform." solution this past week, but our orders were coming in text format, not html. Our only solution was to manually download each to .pdf and forward (ugh!) Also, we have one user in CA that cannot connect. We had her go to another work station where whe was able to submit. What's up with that? Finally, I switched back to the "www.jotform." prefix this a.m. Do you think our problems will be over come Monday a.m.??? Sure hope so - thanks guys - go Colts!

See Answer for This Question...

   
carrollpub (September 27, 2011 at 09:44 AM)

My forms aren't working. The code to embed them isn't right. I'm going to have to use another form builder.

See Answer for This Question...

   
sirmichaelj (October 04, 2011 at 04:15 PM)

You are an awesome service and your dedication to us users is a rare find here in cyberspace. thanks!

   
A sysadmin (October 16, 2011 at 07:43 PM)

Maybe the DDoS is not from phishers, but from phish victims. Your service is perfect for them, especially considering you don't make it easy to report abuse.

Here's one for you, form id #12882142124

Happily collecting user data and sending it off to who knows where.

Back to searching for a "report abuse" link...

See Answer for This Question...

Send Comment