Alexis
Hi everyone. Welcome to… Well, this isn’t Talk Tech with Data Dave. This is Whiskey in the Cloud with Patrick Walsh. Yes, it’s another Talk Tech Takeover episode. Dave is taking a summer break for some much-needed rest and relaxation and like, frankly to do his real job. So, I’m bringing in some very special guests to talk with me about all kinds of things that are all about the cloud, the data, the AI, and the all things D3Clarity stuff.
So, today I have with me Patrick Walsh. You guys have seen him before on Talk Tech with Data Dave, but today we’re talking about Whiskey in the Sky or maybe more specifically, the Cloud. We’re doing a nightcap podcast.
Cheers.
Patrick Walsh
Nightcap. Cheers. Clinky, clinky. Drinky, drinky.
Alexis
Yeah.
I kind of took for granted that Patrick would be drinking whiskey because I know Patrick, but what are you actually drinking this evening, sir?
Patrick Walsh
I am drinking a very delightful Buffalo Trace out of Frankfurt, Kentucky. Yep, that’s what I’m sipping on tonight.
Alexis
That is a delightful whiskey.
I’m a calorie conscious person, and so I tend to avoid alcohol. So, my nightcap is Sierra Nevada Hop Splash. It’s a nice hop water. It’s the citrus flavor, and so I like to put a little fresh grapefruit in it. In its entirety, it’s like 20 calories. I love it.
Patrick Walsh
Is that alcohol? You said hop water. So is it?
Alexis
No, it’s just totally, totally alcohol free. Just carbonated water that is made with.
It says, I don’t know, it’s made with hops. Not a sponsor, but I would recommend it to anyone who likes hoppy things and is interested in a delicious non-alcoholic beverage.
Patrick Walsh
Not a sponsor… YET.
Alexis
Not a sponsor. Could be. Sierra Nevada… Give me a call. When I did drink alcohol, I drink a lot of Sierra Nevada too. So, I would be up for it.
Let’s talk cloud because that’s what we’re here for.
Patrick Walsh
Let’s do it.
Alexis
Okay, so in true Talk Tech fashion, I do have a cloud question for you, but it’s really, really specific to you. Because you are our security guy. We’ve talked about it before on the pod, but you are our security guy for all things cloud.
We’re really into AWS right now, and lately, you have been doing a lot of AWS security scans for a lot of clients. I know this because I run the bills, and I keep seeing, like, “Security scan, security scan, security scan.” And I’m like, “Oh, Patrick Walsh is doing the things. He’s doing everything.”
So, I was hoping that you could share with our listeners the three things that you’re seeing over and over again. Like, what are the three most common issues that you’re seeing in those security scans? My thought being that those could be easy wins for our listeners to just go check their system and make sure that they’ve hit those three markers.
And then later, we can talk a little bit about what those security scans look like, and then maybe even what we can do to help some of our listeners if they’re interested in some additional help.
So, we’ll get to that fun part later. Let’s just talk about the main question. What would you say are the three big ones?
Patrick Walsh
I’ll do the three most common ones. When I get on the phone with a client, I literally ask them a few simple questions. And then I realized that they are using the default configuration that AWS provides, which is not secure in itself.
Alexis
Which is really interesting to me. Right? That, like, by default, it’s not really that secure.
Patrick Walsh
Well, they’ve taken steps recently. But what happens is people have an idea, and they go, “Oh, well, I need to launch my idea and make a proof of concept.” And all of a sudden the proof of concept becomes production. And then we get into the place where we’re at.
But the top three.
I’m going to go with no MFA enabled on the root account for said AWS accounts.
Alexis
Okay, so before we go on, what does root account mean?
Patrick Walsh
So, when you create an AWS account, you put in your email address, and that email address becomes the root user, which means it’s the super admin user. It’s the owner account, if that makes it easier to understand. It’s the owner of the account.
And a lot of people still use that to log in, which is mind-blowing for me. So anyway, I would say that would be the number one. Or if they are not using the root account to log in every day to do work, then they could be using an administrator account that does not have an MFA token assigned to it.
I would say that’s the biggest one I see all the time. A lot of times it’s the opposite. They’ll have a bunch of users, and all the users will have MFA, but the root account still doesn’t have an MFA set up on it. So, you just go, “Guys, kind of missed. You kind of missed the most important one.”
And I can get into some horror stories later, but that would be number one.
Alexis
Those are easy wins.
Patrick Walsh
Right?
Alexis
Set up your MFA for your root account. And if you. If you have a root account, don’t use that as your everyday user account. Like, set up a separate user account for you to use and keep that as your super admin.
Patrick Walsh
Yeah.
Alexis
Okay. Easy wins. Love it.
Patrick Walsh
You really actually, once you create the account, there’s very few things that you need to log into the root account for. Unless you’re changing contact information on your account, you don’t need that. You don’t need to use it. You should store it in a secure location and put an MFA on it and tuck it away so the world can’t find it.
Alexis
Okay. I love that.
Patrick Walsh
The number two is: people will launch EC2 server;. Or really EC2.
Alexis
Elastic container?
Patrick Walsh
No, it’s. Yeah, elastic container. It’s basically a virtual machine.
They’ll launch virtual machines. And this is not just in AWS. This is in Azure as well. We’ve seen this layout across both of those clouds. They’ll just launch a server and put it on the Internet publicly because they’re serving a website, and the website needs to be reached from the Internet. Right?
But then they also leave open the RDP, which is a Remote Desktop Protocol for Windows. Or they’ll leave open SSH or database ports. They’ll leave those open for the world. So, that would be the number two. When we find this, we go, “Guys, you are just a password guess away from being taken over.”
Not really a funny story, but it is a story nonetheless. We had a client two weeks ago. He calls us up, he goes, “Hey, my application.” He had a mobile app application running on a virtual machine. He goes, “It just stopped working. I don’t know why.”
So I go, “Okay, well, let me go take a look”. So, I go in and I look, and luckily, he had some security tools on that were monitoring the environment. And I looked at his security group, and I could already tell exactly what was going on.
He had SSH-Port 22 open to the world, and he got brute force attacked, and passwords were guessed, and usernames were guessed, and his server got turned into a Bitcoin mining server.
Alexis
Oh, no.
Patrick Walsh
Yeah. So a word of caution to our listeners. If you have a virtual machine out on the Internet. You should pull it off the Internet and put it behind into like a firewall or load balance or something. Because yeah. And this client had security groups wide open. Would have never believed how common it is that people just like, oh, just leave it wide open. And they do. And people get taken advantage of all the time. So.
Alexis
Well, yeah, because like, I think, “Oh, my application needs to be out there for the world to be able to access it. Like, I need them, I want them to be able to use it. I want them to be able to handle my mobile app and be able to deal with it.” But it’s the little things like, “Oh, no, I don’t want them to be able to get in there and turn it into a bitcoin mining machine.”
Yeah, just gracious. That’s terrible. I shouldn’t be laughing. I’m sorry. This is what we get for doing a podcast at 8 o’clock at night.
Patrick Walsh
Right?
Alexis
It’s my bedtime, man. Like, well, well, I’m ready to go to sleep. Go ahead.
Patrick Walsh
While. While imbibing.
Yeah. So generally, the solution that we recommend is, number one, if you’re using default VPC or virtual network or whatever, depending on your platform, just don’t use that and build your own that follows best practices, which is a layered approach and it’s subnetted out and you put a NAT gateway in there to protect your resources and then, well, the NAT gateway is used for the egress. I don’t want to have people think I’m an idiot. Anyway, that’s a lot of stuff…
Alexis
Yeah, I believe you know what you’re talking about.
Patrick Walsh
Yeah.
Anyway, use a load balancer or a CDN in front of your server. Whether it’s serving up content or whether it’s a data server behind, use some sort of interface there and then there’s application firewalls. If it’s just simple serving web content, you can put a WAF in front of it. Even for API servers or API gateways or anything like that. I’ve seen the same problem there. They’re wide open and people take advantage of these APIs and go data mining.
So, protect your virtual machines, don’t leave them open to the public, and only allow the traffic that should be going through there.
And then the third major thing I see is: No one ever logs anything.
Like there’s no network logging. I can’t tell what traffic’s hitting. There’s no logs against the server. There’s no logs against the application. No one turns logging on.
Alexis
Logging, logging like in layman’s terms is just something that keeps track of what’s going on, right?
Patrick Walsh
Yeah.
So, for example, let’s talk VPC flow logs. Flow logs allow me to go look at the network card on the virtual machine. And I can see all the traffic coming in and out, and you can see the actual IPs hitting the server. And at that point you can go and see, “Are these legit IPs, are these legit calls?” You can look up the port numbers that they’re calling on. It’s very handy to see, especially if you get a client that says, “Hey, my application server is running slow,” or whatever, right? And you go dig through the application server, and you find out, well, I can’t find anything crazy with the application server.
We step it back a layer and you dig into the network and you find out, “Oh, there’s 5,000 IPs coming in from Singapore or China or Russia or whatever. They’re just beating the server up.” Having the logs, it leaves the clues behind. So, if you’re calling for help or you need help or whatever, people can have a…. we’ll call it breadcrumb trail that they can follow and determine what’s going on.
So those are my three bigs.
Alexis
My personal story with logs is like the IP story that you just told, just a little bit backwards.
I play our M365 administrator because I mostly just help another guy who does most of the hard work for us, but once in a while, we get hit with these, like, suspicious activity alerts, where I’m like, “Oh, somebody just tried to log into Patrick’s account. Where did that come from?” And thank goodness for my other guy, who handles all that stuff for us. He has the logs enabled and we can go in, and I can see the IP address of where it hit from.
And then I just send Patrick a quick message. “Hey, Patrick, is this your IP address? Did you just log in from here?”
And he’s like, “Oh, yeah, I’m traveling today, Alexis. No big deal.”
And oh, thank goodness, because one time I was at a conference and I got a “Hey, Patrick’s account just tried to hit from Russia. Patrick’s not in Russia, Patrick’s in Texas.” Luckily, everything worked. Everything stopped it from happening. But, like, the log showed us that we had a suspicious activity and that it was not coming from Patrick. We were able to flag that really, really quickly. So highly recommend. If that’s something easy to turn on, people go turn that crap on. That’s a good idea. It’s very, very helpful.
And I mean, I hear it like, there’s a reason why I still manually edit this podcast, and it’s because I don’t want to spend the money to purchase a platform that’ll do the editing for me, and I don’t want to pay somebody to do the editing for me. So, I manually do it. And, like, I hear it and I get it.
But there’s a difference between me spending a couple of hours editing a podcast and us putting people’s personal information out there for the world to see, or me leaving my ports open for somebody to turn into a bitcoin mining site. Like, there’s a little bit of a difference there between the two, so I get it. But, like, there’s a place where the line gets drawn, and it’s worth it. Turn on those logs, guys. Come on.
Patrick Walsh
Turn the logging on, guys. Turn on the logging. Yeah, let’s close those unused ports. Let’s put some MFA in place. Yeah.
Alexis
Yes. Yeah, Two Factor Authentication. Yes.
Patrick Walsh
Yeah. 2FA, MFA, all of it.
Alexis
If you listened to the podcast, go look at your stuff and go see if any of those applied to you, and then hopefully take steps to fix those. We would appreciate that. Just because, like, we’re trying to educate the public. That is the big point of the podcast.
Here we are doing a summer of shameless plugs. I kind of started that on accident a couple of weeks ago, and so now we’re just kind of sticking to it. So, I started the podcast by saying, “Patrick, you’re doing a lot of these security scans,” so tell me what those security scans look like. What does that mean? If the Listener is like, “I, would be interested in one of those security scans?” Like, what would they get out of it?
Patrick Walsh
Oh, well, they would get amazing information! Let me tell you.
I’ve had CTOs and CIOs go, “Well, what am I getting for this?” I’m like, “Well, okay, first of all, you get a full scan of your entire AWS environment.”
So if you’re using organizations or you’re using a single AWS account or whatever the situation is, we scan every single account. And it covers from your identity, your IAMs, whether you’re running CloudTrail logs, security groups, all your S3 buckets. Like, it covers everything, and it also provides compliance scans. So, if you’re trying to be HIPAA compliant, if you’re trying to be NIST800-53 compliant. Your compliance is GovCloud. The tools that we use provide compliance reports for all of those things. The standard scan we get is a CIS 3.0 standard. So, it covers everything.
Every time I do a scan, I get probably…I think the worst one was over like 24,000 problems and the best one was like 3,000, somewhere in their… problems that need to be fixed. Some of them are critical. And so, there’s a scale. There’s the critical, the high, the medium, the low, the informational. It picks up all of that stuff. We do that scan, and it basically just dumps raw data.
So. I take that raw data and I create basically a C-level report that says, “We found this many criticals, this many highs, mediums, lows, informationals.” You get it. Kind of bullet pointed there. And then we go through the, about the top 10 of each critical, high, mediums. The lows, and the informational – kind of leave off because by the time I get there, it’s overwhelming. You’re on page, you know…
Alexis
3,000. Yeah, I can see why you might.
Patrick Walsh
Well, you guys, you also have to remember. So, say you have 20 serverless functions that have secrets in the environment variables instead of using secrets manager on AWS, for example. So you’ll see 20 line items, one for each Lambda function that’s violating the Standard. So those 20, you know, line items, I’ll summarize and go, “You have 20 lambda functions that are not using Secrets Manager.” So I’ll consolidate it down, and I usually put the most, obviously, the critical, the highs, and the mediums in there, so they get that.
Additionally, while I’m in there, I put together an architecture diagram that shows the entire organization. All the VPCs, what resources are in the VPCs, what resources are not in the VPCs. It’s funny, the clients that we do this for, they’re like, “This is amazing. I didn’t even know we had all this stuff out there. This is great.”
And the reason why they don’t know that stuff’s out there is. And they’re like, “Why is my bill $20,000 a month? How come?” And you know, they talk to the developers, the developers like, we need all this stuff. But when you draw it out, like, it’s a huge diagram that we put together, and it shows all the network connectivity, and it shows all the ports, and it shows public IPs, private IPs, and all the IDs of every resource in there. They get that C-level report, plus they get the raw data scan that the tools come up with. And we said, here you go.
Alexis
We package it all up, put it in a pretty, easy-to-read bow, and hand it over.
Patrick Walsh
Yep. And I’ve done this for several clients, and several clients have said it’s not just the security scan or whatever, that they’re like, this is awesome. But they take the diagrams. They’re like, “This is the best thing ever. Because now I can see they have a bird’s eye view of their entire environment!” And they can go, “Oh, we’re working in this section and we’re working in this section. Or why do we have this and why do we have that?” And they have something they can reference and talk cost-reduction and talk, “Do we really need stuff like this? And can we implement, like turning things off in our development environment and just do some cost saving stuff as well?”
The other thing that usually comes out of that is we do well-architected reviews with everybody. Right? Which is – if you’re saying we’re doing shameless plugs.
Alexis
That’s another one to to slide in there.
Patrick Walsh
We’re gonna throw the WAFR [Well Architected Framework Review] in there.
Alexis
Love the WAFR! Shout out to Alyson.
Patrick Walsh
Yep. Which is basically – You have the six pillars that AWS says you know is well architected and we review against that as Security is a pillar. But then you bring the architecture diagram in and you go, “Look, guys, we can modify this, that the other. And we can improve your operations; we can improve your performance; we can improve reliability, all of the things that are part of the well-architected framework.”
So, you say it’s a security scan. It’s not just a security scan. It’s a security scan. It’s recommendations. And here’s an entire architecture diagram for your environment. And then the pages of like, “Here’s how we would re-architect. Here’s how… Or not necessarily re-architect. Here’s how I would fix things. Here’s how we can improve your security posture on the Internet, and here’s how we can set you up for a successful infrastructure going forward.
Alexis
We are going to offer a special promo with this podcast. And so if anyone wants to refer to the Whiskey in the Cloud with Patrick Walsh Nightcap podcast and request a security scan, we’ll offer you 50% off what your security scan rate would normally be. This has been fun. I feel like I’ve learned something. I feel like, hopefully, our listeners have learned something.
I’ve definitely learned that my bedtime is set for a specific time for a reason. So, I’m going to go probably, to sleep very soon. But I did promptly finish my cocktail on time, so I am good to go.
Patrick Walsh
Pretty close. Pretty close.
Alexis
Well, cheers, Patrick. I hope you have a good night and listeners out there. If you have a question, you can always reach out to us at talktech@d3clarity.com Happy Summer. Happy Summer Talk Tech Takeover. And until next time.
Patrick Walsh
Yeah, thanks guys.
