Yesterday wasn’t an ordinary day for us the Jotform team. I woke up with a “Jotform Down!” SMS message. Jotform load balancer was gone. At first I thought this would be a load balancer hardware or software issue. It turned out that we were under attack by thousands of zombie computers in China. To our amazement it was a massive DDoS attack.
We kept fighting to get Jotform back alive and we were brought down back many times throughout the day. It wasn’t working and our service provider was not happy with us. Their network and routers were getting saturated and other customers were getting affected. In fact, they told us that they cannot sustain these attacks any further and we might need to think about other options.
We kept asking ourselves why would anyone do something like this to JotForm. We are the good guys. We provide a free service to hundreds of thousands of people. Our users are very happy with us. We don’t receive any hate emails. In fact, we receive lots of emails every day from people who are very happy with the service Jotform provides. One possible exception is the phishers. They keep creating phishing forms even though we suspend their accounts. So, our conclusion was that the attack was probably initiated by a phisher who was suspended.
Unlike regular denial of service attacks, Distributed DoS attacks are very difficult beasts. They are usually initiated by thousands of compromised computers. So, they look like legitimate traffic coming from different IP addresses on different locations. Since they start their attack in sync, it takes them very short amount of time to saturate computers, load balancers and even routers on their way. A great tool that has helped us a lot to slow down attacks was DDoS Deflate.
Upon further investigation we found out that most of these attacks were coming from compromised DSL modems in China and some other Asian countries. What happens is people leave their DSL modems with factory settings and these settings may have default passwords, or they might contain firmware with security holes. Worms target these DSL models and once compromised they can be used for DDoS attacks.
The attacks are still continuing but we are now able to continue our service without any interruptions to our users. Our final solution was to move part of our infrastructure to Amazon Elastic Compute Cloud. EC2 was able to handle high levels of traffic and sustain us throughout the attacks. We are still getting sudden surges of attacks but our architecture is much more ready to handle these, thanks in part to Amazon’s services.
Jotform users have been very patient with us. We appreciate their support. We tried to keep our users updated on twitter as much as we can. Using our official twitter account we were able to broadcast real-time updates to our users.
We are very sorry about downtime these attacks may have caused for our users. We will keep improving our infrastructure to prevent future incidents like this.