Transcript
[MORPER] Hey. Good afternoon, and welcome to the latest episode of Virtru's Hash It Out. My name is Mike Morpher along with my colleague, Trevor Foskett, and we're here to discuss the latest cybersecurity exploit, and this one's got a great name. It is called echo spoofing. So, Trevor, can you give us a little background on what echo spoofing is and and really what happened?
[FOSKETT] Yeah. It's an interesting one, and the name, you know, doesn't quite do it justice. It's sort of the sophistication of what went down, but as a long story short, this something that happened to folks who use Proofpoint, email relay services largely for outbound messaging. A group that includes, I think, 87 of the, you know, Fortune 100 companies. So some heavy hitters here were affected. Yeah. In a nutshell, basically, a way to send email through Proofpoint Services while pretending to be someone like a Disney or a Coca Cola. And because of the way the architecture is laid out, have those messages be reliably delivered and inboxed to recipients in a way that looks safe and legitimate.
[MORPER] So for those who, so none of those people would have realized that they are receiving, you know, a bad actor's mail message?
[FOSKETT] No. I mean, if you're someone who uses maybe Gmail, or, you know, Microsoft Outlook, you may be familiar with getting a message that either goes straight to your spam folder automatically or you see one with a banner across the top that says something about this looks odd. And, usually, those warnings are coming because one of these things like an SPF record or a DKIM signing didn't match up with what it was supposed to be, and those are methods that we use to indicate a legitimate message. If it doesn't have those things, it's not legitimate. The tricky thing here is that these illegitimate messages did have those things. So when it lands in your inbox, it looks like, "Hey. Great. I just got a real email from Disney," which, of course, is very dangerous if you think it's coming from someone Right. That it's not.
[MORPER] Well, that's, that's unfortunate. And was it Proofpoint that uncovered this? I'm assuming it was.
[FOSKETT] It wasn't. I think it was a 3rd party firm. I believe they're called Guardio Labs. So they discovered this some some months ago, I believe, and have kind of been working with Proofpoint since then, on delivering a solution here because, obviously, if I'm one of these large companies and I'm finding out that my name is being used out there to send bogus emails. I need that fixed, yesterday.
[MORPER] Yeah. In fact, I was just reading a blog article on this, and according to Gardeo, these bad actors actually started back in January of this year, and it wasn't until March or so that Proofpoint became aware of this, in the wild. So it was definitely out and about for some time, and it appears that it was not resolved until June. So, basically, this has been floating around for about 6 months. So, my instinct is this probably isn't gonna be the last we've heard of this. What are your thoughts?
[FOSKETT] No. Certainly not. And, I mean, that timeline is interesting. There's kind of a lot of a lot of things that play that kinda, you know, to your point of this is this the last we've heard of this? And also just that timeline. Usually, if we hear about something in March, we expect that the patch goes out in March for something with sort of a blast radius, right, like this. Right. But there's a couple pieces at least for me reading through it was, you know, usually if someone's sending bogus emails, you'll get reports. Hey. You know, someone's trying to send these spam emails or these spoofed emails from you because I've noticed somehow. But the attack here is so sophisticated that a lot of people are getting these emails and going, looks great. You know? Nothing to report. Right? And maybe they went and added their credit card number somewhere they weren't supposed to, but, again, unless you're looking at the charges or you're doing some sort of forensic investigation, you're not even gonna know that you've Impound, if you will.
[MORPER] Right.
[FOSKETT] And, also, from the sort of provider perspective as well. I mean, if you're proof point, you're seeing these emails come through, you're saying, like, you know, it looks right. It's coming from the right place. I'm seeing all the right signature information, as far as where this email was sourced from. So based on the rules in place here, it looks good. So, again, not really aware of any wrongdoing here. And it's, you know, relying on all the tools that exist there. We have these things in place. Again, in this scenario, specifically, SPF and DKIM that are there to do these jobs. And so the sophistication here, and they're really kind of wanna compliment these guys too much, but it's such a well orchestrated attack that it was like, yeah. Let's lean on all these tools that are in place to prevent this and use them to our advantage. I mean, if they weren't if they weren't the bad guys, I'd kinda say, nice job.This is kind of an elegant solution. You know? It's pretty interesting.
[MORPER] Yeah. So, you know, if they were using legit SPF and DKIM records, how are they able to pull this off exactly? Have you been able to learn anything about this?
[FOSKETT] Yeah. I mean I mean, it's interesting. So for those who aren't familiar with what these acronyms that we're using are basically ways to prove that an email that you've received is actually from who it says it's from. Back in the day, you could send an email, and I could say, hey. It's me, you know, mike@virtruprivacy.com, and send an email off from my totally separate email server, and you just have to trust that that was it. But we implemented these technologies, SPF, sender policy framework, and DKIM DKIM, which is domain keys identified mail to basically say, here are ways that you can look up and check. So SPF basically says, we're gonna put a public DNS entry that says, these are the services that are allowed to send email on behalf of my domain. Now if you're a Proofpoint customer or Google or Microsoft or Zendesk or Salesforce or any of these software platforms that you wanna use to send email, you need to put an SPF record in there to say, yes. That's okay. If you've set up Google Workspace before, you had to set up SPF to say Google consent on behalf of my domain. So that's one piece. If you're a Proofpoint customer, you need to go in and say, hey. Proofpoint can send email on my behalf. So that's where the first part of this comes from is that that email in this spoofing scenario was sent to Proofpoint. There was a lot of stuff that happened upstream, but, ultimately, that email went to Proofpoint, and Proofpoint delivered it to the recipients. When the recipient server looks up that SPF, it says, yeah. Yeah. Proofpoint allowed. Proofpoint allowed to send it. Says so right here in the public DNS. So that's one. The second piece with the DKIM signing is that is, again, usually signed with a private key that is only available to the authorized sending parties. That's not public. You can't look that up as part of DNS. Now you can look up a check to see if the signature that you've received is correct, similar to how things like, you know, message signing with SMIME or PGP works. DKIM is sort of the same for SMTP traffic. You can look up and say, yes. This came from a legitimate source. Right. Again, if Proofpoint is gonna send DKIM sign mail for you, you gotta put your private key in their system. So now we have a second component where Proofpoint is sending off this email, and it looks good as far as the recipient looks when they go to look up that signature. So the sort of intricate thing about this is that how did those emails get into the Proofpoint ecosystem? To begin with, you would think that systems like this should have a way to say, who's allowed to even send the emails to Proofpoint on my behalf so they can do those things, SPF and DKIM, to get it in the inbox. And they do have a mechanism for that. When you set it up, they say, hey. Where's email coming from? Is it coming from Google? Is it coming from your on prem exchange server, or are you a Microsoft 365 shop? Well Right. The interesting thing about kinda public cloud services is that if I'm using Microsoft 365 and you're using Microsoft 365, both of our emails, despite being 2 separate guys, 2 separate companies are coming from Microsoft 365.
[MORPER] That's right
[FOSKETT] And so there's a lot of upstream processing that happened basically by blindly relaying through a couple different Microsoft 365. It's blindly meaning it's not changing any of the headers. So I could do the sort of old school approach. Say, hey. It's from Mike atvirtrueprivacy.com. Blindly relay it through, and then when it hits Proofpoint, it says, well, it looks like it's coming from 365. And Virtru privacy said we could send from 365, so it might be legit. Right. And then go ahead, process it, and pass it on.
[MORPER] Right. So it kinda sounds like the configuration on the m 365 side was a bit more permissive perhaps than it needed to be or specifically Proofpoint's configuration of accepting the traffic from m 365 was probably a little bit more permissive than it needed to be. It wasn't explicitly looking for a specific tenant, a specific header. Am I going down the right path?
[FOSKETT] That's right. That's what it seems to be. Yeah. I wasn't looking for something specific enough. And as I was reading through this, there wasn't a singular thing that kinda jumped out to me that was like, Microsoft really dropped the ball here. Proofpoint really dropped the ball. Could each system have been doing something a little bit better? Yeah. But, again, sort of going back to the elegance of this, it sort of used some of the standard ways that we do these things, in order to take advantage of them in a way that that really nobody saw coming. Yeah. You know, so, again, there's more that can be done, but neither was, like, a huge red flag. It's sort of a couple beige flags in a row that got laid in a pretty in a pretty bad way.
[MORPER] Right.
[FOSKETT] They weren't doing anything like, hey. You know, you need to include a specific header on your message when it comes through, or we're gonna look it up and trace all the way back to the server that originated this because of the blind forwarding. You can really do that. And there are use cases for a blind forward. Right? You may need to use that for something else. Sure. And there are scenarios where we don't wanna be overly restrictive because we need to allow, you know, multiple workflows into a proof point scenario. So, again, reasons why things were sort of as open as they may have been, but because they needed to be open to account for some of these things, a result that we didn't want. So we need to look at how we can do that better while still allowing the workflows that we need.
[MORPER] Yeah. For sure. You know, for sure. You know, and and look, to be fair, you know, proof points in generally the same type of business Virtu is in. So there is a degree of empathy. You know, we don't like to see, you know, bad actors do bad things to any of us. That said, did Proofpoint report any key learnings from this episode yet that you've been able to come across?
[FOSKETT] They did. I mean, I think that there's again, there's a lot of things that can be done on the Proofpoint side of the platform. There's also things that can be done on the customer side. So what are your configurations that you wanna look at? And so, you know, if you are doing custom relays of mail, let's take a second look at those. Whether there's proof point involved or not, let's take a look at what our permission sets are, how we're filtering out that traffic, and, you know, scope it down to whatever the kinda email analog of provable, principle of least privileges. Right? Like, if you can get away with not having an open relay because you have a very specific workflow, then do that. Have it be hyper focused to that whether that's based on IP addresses or firewall rules or some other logic that you have, do that. Right? Anytime we're leaving something sort of slightly open because it's easier and more convenient, you're gonna maybe see something like this happening.
[MORPER] You know, this let me just hop in here for a sec. This also kinda reminds me, this is not edge case stuff. There are all kinds of systems of record that relay SMTP traffic out to the world, whether it's an invoice that's needing to go out, you know, a CRM system, what have you.
[FOSKETT] Yeah.
[MORPER] It makes me think, you know, if I just joined an organization in IT and all of that infrastructure's already been in place, probably a good practice. You know, maybe go ahead and inspect the way those things were configured. Just because you inherited something doesn't mean it was set up correctly. Probably a nice opportunity to audit that configuration. Would you agree with a best practice like that?
[FOSKETT] Absolutely. You know, I I always you know, to take it out of this context a little bit and go more broadly, you know, anytime you have a new new set of eyes on something, whether it's a new hire or someone in a new role, you know, the I think the status quo or the the the feeling is to let the status quo sort of exist because people don't wanna rock the boat. But anytime someone's got new eyes, I'm like, no. Ask me a question. Hey. Why are we doing it this way? You know, that seems bad. And a lot of time, the answer is, well, it's just because it's how we do it. But, yeah, you're, you know, you're right. There's probably room for improvement here if you wanna take a second look.
[MORPER] You know, going going back to any lessons learned, the 2 of us were chatting about this right before we hopped on, and I think one of the things that, you had read in there that resonated with me was this notion of, you know, looking in the mirror, and I think this might have been proof points, one of the points they were trying to make that, it's important to make sure that from a user interface, from a user experience, all the controls should be really clear, very user friendly, very accessible so that there is no ambiguity as to, you know, what does this tick box do. Right?
[FOSKETT] Right. Right. Yeah. Or or, you know, something that looks like it's the default config because all the other stuff seems, you know, maybe hidden or whatnot is, you know, you wanna put that. And this is part of their remediation actually that they did put, you know, one of these settings right there up front now as you configure. So that's a great move by Proofpoint to say, look. Here's something you need to be aware of. Let's surface it so you can put this right front and center. You're definitely gonna wanna configure this as you set up rather than something that you know? I've dealt with software before where you kinda call up your service rep and they say, oh, yeah. I know. We can change that for you. And say, I didn't even know that was a setting, let alone something I could enable. And so having it right front and center goes a long way in just making sure people are configured right from the start.
[MORPER] Yep. That's right. You know, I think they also pointed out that organizations that host virtual infrastructure, they also have an opportunity to be looking in the mirror as well. And one of their recommendations back to those types of organizations is to be very thoughtful about, you know, free accounts, limited accounts, anything that in this case is emitting SMTP traffic, to maybe give consideration to how much of that can they really send on these immature accounts. Obviously, in this case, you know, this threat actor took advantage of 100s of tenants, to be able to take advantage of their nefarious, nefarious works.
[FOSKETT] Yeah. That's right. And, you know, I don't know a ton about what went on sort of on the Microsoft side here, but it does sound like maybe Microsoft said, look. We don't see an issue here even though it was known that there were some, you know, whether it was volume or or or what exactly was reported to them. But I think that's very important. I know that Google does this. They'll limit you unless you have, you know, you sort of are in good standing and have asked to have more volume. They'll say, look, can't send more than 10,000 emails a day or something like that because they don't want you using their platform to spam people. And so I think those controls are really important. It's how we kinda detect anomalous behavior on new accounts or or small accounts that are generating more traffic than any, you know, the 15 licenses they have could possibly ever generate. Things like that to just better understand what's, like, a legitimate use case and what's something that's not, you know, above board, I think, is a great place to look as well. Again, sort of just different pieces of this chain all sort of not doing anything horribly wrong, but things that could have been done better along the way.
[MORPER] Yep. And as I was reading one of Proofpoint's, blog posts on this, one of the things that I learned is, that white hat that surfaced up the, this, exploit that they found in the wild, it was Guardio Labs. They reached out obviously to Proofpoint. Proofpoint acknowledged that they had telemetry to give them indications something was up. But what I thought was, just as good action by both parties is day forward collaboration. It sounds like they were in lockstep together with one another. They also reached out to Microsoft. It sounds like there was collaboration among at least 3 parties. And it's just a you know, it's another one of those good ways to see something, say something, and work together to run this down because, obviously, everyone benefits or certainly a large population of organizations benefit from this collaboration.
[FOSKETT] Yeah. That's absolutely right. I mean, you love to see that versus, you know, Guardio instead going, you know, day 1, 0 day publishing it. They, you know, gave Proofpoint an opportunity to work with it. Right. Not put that information out into the wild where it could be further exploited until now there's a fix in place. Now we're getting the news, so you can't really take advantage of this anymore. Great for them to kinda go and and treat it all in certain in the spirit of good faith when we are assuming you're not doing this to screw over your own customers, but I thought you'd like to know about it and give you a chance to respond, which they did. So
[MORPER] Yep. That sound that sounds right to me. Any, any kind of parting thoughts on this? Like I said earlier, we're in a similar business. Are there any best practices beyond things that we've already been talking about that maybe as you reflect on this that any of our existing customers that might be listening to this conversation, should be giving consideration to as well?
[FOSKETT] Yeah. You know, anytime there's something like this in the wild, I think, as a customer of a similar solution, my thought would be, does my solution have something like this? And if so, what are they and what are they doing about it? And so for the vast majority of our customers, you don't need to worry about this type of attack because most of you, out there, if you're listening, I'm assuming you're a Virtru customer, are using our client side integrations. Meaning, you've plugged Virtru directly into Google Gmail or into Microsoft Outlook. And so we haven't had to change your mail flow at all. You're not relaying. You don't need to add SPF or any other records in order to enable that because it's just being sent by the system that already sent it. So really good news for you if you're in this client side only or end to end mode as we might call it. Then we do have a couple products that are involved in email relaying. If you're doing something like sending secure emails from your scanner, or one of your SaaS applications that you've stood up, it's gonna go through our gateway product, which is sort of the server side analog. And that is a mail relay in more of the traditional sense. There's 2 flavors of that. 1 is hosted by us. We call it the SaaS version. It's sort of similar to the cloud model of Proofpoint that was involved in this. But we've already implemented basically the fix that Proofpoint put in, and we had this implemented from day 1, when we first deployed the gateway many, many years ago. And that's an authentication header. So if you're using that Virtru hosted gateway, every message you send needs to be authenticated with a unique header for your organization. Without that, we'll kick it out because it's not not a legitimate message. And so in that scenario, this attack would fall down because it doesn't wouldn't have had that authentication. Only you have that ability to add. Now if you're using, what we call our customer hosted or on prem gateway, similar deal. There are lots of different ways that you can grant access to those things. Again, you can do an authentication header. But a great time, as we talked about earlier, to review your configuration, make sure you're using all the tools available. There are a couple different authentication methods. You can look at the environment where that container service is running. You can look at your firewall rules. You can look at the upstream systems that are allowed to send to it. Again, a great time to review and make sure you're taking advantage of all all the controls you have there, but the controls are there. And if you have questions about, you know, how do we how do we evaluate and make sure that we actually have these in place, reach out to whoever your your virtual contact is, and, and we'll get you, set up and looking at the right spot for how you're configured and make sure it looks good.
[MORPER] Awesome. Trevor, I don't have anything else to ask you. Anything that we didn't discuss that you wanted to bring up?
[FOSKETT] I don't think so. I mean, you know, just really interesting, this one I thought given the sophistication and sort of the proximity to the area where we play, I thought it was a great conversation topic and enjoyed talking about it with you, Michael.
[MORPER] Yeah. Likewise. Well, it was great to go dumpster diving through all the information that's come out over the last 24 hours on this one with you, and obviously talking about this with you. So, we'll see how this one continues to play out. Like I said, I don't think this is gonna be the last time we have heard of echo spoofing. By the way, we should get the shirts. I think shirts. What do you think?
[FOSKETT] Yeah. I'll get the Etsy store going and see if we can get those out there.
[MORPER] Maybe decals for laptops too. What do you say? We can quit our day jobs. There you go. Alright. Cool, Trevor. Alright. It was great chatting with you. Have a good one. And for everyone else, Mike Morpher with Virtru, Trevor Foskett with Virtru, and thanks for listening.
