Lessons Learned from (almost) Getting Hacked
I knew it was going to be a long day when I received three emails, to three different addresses of mine, which looked suspiciously like a phishing attack. Unfortunately, these emails all came from one of my employee’s TeamSupport.com email account.
“Uh oh” was my first response.
It got worse when, just a few moments later, we started receiving tickets from customers into our support queue telling us that they had received the same email.
Our immediate thought was that somehow TeamSupport had been hacked and our customer list had been compromised. Worse, was it possible that customer data had been hacked? For the CEO of a cloud-based application company, this is about the worst-case scenario — we spend a lot of time and effort on data security, and our customers trust us to keep their information private and secure: Any breach of that trust is a major issue.
TeamSupport was not compromised at all, and no customer data was breached.Now that we’ve cleared that up, back to the story...
Once we suspected we had a security issue, I quickly gathered up our security and operations teams, and we all worked to determine what happened and what the scope of the issue was.
At this point, we were trying to figure out two things: How did hackers ‘spoof’ our email so that they could send out an email from one of our corporate addresses, and where did they get the list of recipients? We were unclear if this was two separate issues or if they were related.
We use Google Apps for Business to handle email, calendaring and the like, so we reached out to Google to see if they could be of assistance. Google responded very quickly to our plea for help, and they showed us a tool in the administrator console where we could trace the message ID and see who it was sent to.
Since internal users had received several of the messages, we had a number of different message IDs we could look at, and were able to quickly put together a list of some of the email addresses targeted by the phishing attempt. While it wasn’t a comprehensive list of all emails sent, having a partial list was a huge help to us in determining where the breach came from.
By looking at the addresses of people whom the message went out to, we were quickly able to determine that the list had not come from the TeamSupport application. That prompted a sigh of relief as it appeared that the worst-case scenario wasn’t what happened.
We looked into our CRM and marketing automation systems, but we couldn’t get the email addresses to correlate to any of these systems either.
We looked in the employee’s address book, but didn’t find the addresses there; in fact, no contacts were listed in the employee Gmail account at all. We continued to scratch our heads until we looked at several other Gmail accounts and realized that when you send an email from Outlook or another email client through Gmail, Gmail creates a contact for that person. However, there were no contacts in the Gmail account in question — why not?
Someone on our security team had the bright idea to attempt to recover deleted contacts in Gmail, and sure enough, several hundred contacts appeared. The Gmail contacts matched the list we had put together of people who were sent the phishing email, so we had found the list!
This was, of course, great news since it meant that just a single email account had been compromised, and there was no breach of any other system, including, of course, TeamSupport!
So, what happened?
After looking through account activity logs for Gmail, we could see where a user had accessed the Gmail account from Lagos, Nigeria, and I was assured that no one from the company had been there recently! There wasn’t any evidence of a brute force password hack attempt, so the best we could determine was that our employee had succumbed to a phishing attack at some point in the past.
Once the hackers had valid credentials, they were able to access the compromised Gmail account and download, then delete, all of the contacts. The deletion of the contacts was a pretty smart move since it hampered our ability to determine where they had obtained the list of email addresses. There was also no record of any of these emails being sent through Gmail since the hackers used another email client and spoofed our corporate address.
Unfortunately, it's pretty easy for someone to ‘spoof’ an email address and equally hard to prevent. While there are some technologies like DKIM and DMARC which can help, they don’t appear foolproof to prevent spoofing. Here’s a good article which gets into more technical detail about how email spoofing can be done and what you can do to prevent it.
Once we recovered the contact list from Gmail, we had a solid list of people who had received the phishing email. While many companies would have just ignored an issue like this, we felt it was best to be proactive and upfront about it, so we sent an email to this group. The email basically said, “We’re sorry; one of our email addresses was hacked.” We also made it clear that none of the TeamSupport application data was breached.
I wasn’t sure what the response was going to be when we sent that email out, but just a few minutes later, we received the following:
"I'm so glad you handled it this way. When it happened to us it was really hard to do, but we did the same thing. No one else has done it that way. Please pass this appreciation to your team."
We actually received many responses like this, from, “No worries,” to “Thanks for letting us know.” While a situation like this is never positive, these supportive responses certainly helped to put a nice ending on an otherwise stressful day!
Lessons Learned & RecommendationsWe had several takeaways from this incident:
- Enable two factor authentication. Probably the biggest mistake we made was not to enable two factor authentication on our corporate Gmail accounts from the start. Enabling this makes it MUCH harder to have a successful phishing attack and likely would have prevented the issue in the first place.
- Investigate DKIM and DMARC. While neither of these technologies are foolproof, they can help prevent email spoofing.
- Be upfront in addressing the issue with customers. We struggled with how best to respond — it’s always hard to admit to having a problem, especially around security, but sending an email to the addresses the phishing attack targeted was the right thing to do. In fact, in customer service, this is always a good rule of thumb: Be honest with your customers, and they will respect and appreciate you for it. Trust is the basis of any strong and long-lasting relationship.