1

Hello,

First let me say that I’m only posting this to share my experience from the last couple of weeks, I am not here to be questioned. I’ve learned a lot these last couple weeks and and my story might help someone else, so I feel its worth sharing.

Our company was hit with ransom ware in the late hours one night during the week of 12/9. Starting sometime around 11:00 p.m. Anything and everything that was powered on was hit and completely encrypted. File servers, sql servers, ERP servers, application, workstations, ect. This was spread across 7 subnets in our network. Hundreds of machines were affected.

Not only that, but the intruder also gained access to our SAN storage appliances. They removed my snapshot backups from these appliances. My heart sank. SAN storage syncs to another SAN at a remote site… snapshots also removed there.

Keep going… I checked the veeam server… encrypted. I checked the NAS where the veeam backups were stored. Volume deleted, and device is suddenly completely full. They wrote a script to fill this device. At this point I’m white as a ghost…

My NAS syncs to another NAS at a remote site for our company. This was also hit… NO DATA.

I was screwed. Everything was backed up in these two, three, four places…

I had some backup tapes with our mission critical information, file storage, HR data, ERP, servers that couldn’t easily be rebuilt and reinstalled. This was the route I was going to have to take, although slow and tedious at least I had something. I hadn’t used tapes in years, and never thought I would need them again with the way I was doing things.

I have never heard of such a sophisticated target, it almost seemed like it was an inside job. The way they were able to locate my storage appliances on different subnets, and take the time to gain access is incredible. They knew everything and didn’t miss anything. Usernames were different, but passwords were shared in some cases. My fault, lesson learned. Lesson #1. Have a different password for everything. Lots other lessons.

I have yet to find out how they gained access. There was an RDP port open on the firewall, probably that, maybe, its turned off now. All servers that were encrypted or hit were deleted and recreated due to space on our SAN, can’t investigate them, they were encrypted anyway. More to check, and think about for sure.

Please take it easy on me, my last couple of weeks have been hell. I am curious if anyone else has experienced anything else like this in the past?

Thank you.

69 Spice ups
2

Wow that was difficult to just read. Gave me the chills. From what you describe it seems that your network was compromised for quite some time, or someone with intimate knowledge of your network was colluding. That or the hackers were able to gain access to your IT documentation. Hopefully your company has cyber insurance.

23 Spice ups
3

I was thinking RDP while reading this and then you confirmed it during my read. Had this happen to a ‘test RDP’ server that I forgot I left on the internet. It had only been stood up for about 4-6 months. It was on its own network - but I did have restored copies of some VM files servers that I had put on the same test Hyper-V host during some backup test restores. I was able to poke around after I discovered it. Looks like they had just brute forced a password, changed it and then added their own accounts before kicking off the encryption. I just nuked the server after I was done poking around on it. But I’m a strong believer in no RDP from the outside without a good VPN and a good password policy.

10 Spice ups
4

I agree, this is not what we normally think of as “ransomware.” This was a full-blown, sophisticated, targeted attack. Somebody got in a while back, and has been setting this up.

22 Spice ups
5

Wow I wish I could say I feel your pain but I have not been though something that bad. I been through a few ransomwares for my clients. Some were pretty bad, but I never had them get into Veeam let alone get into any production or backup storage. But I know there is one ransomware out there that exploits RDP, I had one client get hit that way. But not to the extend you went thought.

My hats off to you, I hope all is well and backup and working. Sucks this happened right before the holidays.

2 Spice ups
6

Wow. That is horrific. Thank you for posting this. It sounds like they were in the network for awhile poking around. Did you have any sort of honeypots set up? I’ve often considered it, but find commercial versions too expensive for my budget and assume a real intruder would smell my homegrown attempts a mile away. After reading this I may just throw something out there to see. Your setup sounds great… I hope you get a break over the holidays. You’ve definitely earned it.

1 Spice up
7

I’m pretty sure that I threw up a little as I read that. I do agree that it sounds like someone spent time with this, as opposed to finding a hole and popping in. I’ve worked very hard to simplify what we do and backup onsite and also use OTS software like Carbonite. When I arrived, there were no backups except one portable drive. Gulp.

5 Spice ups
8

Experience is knowledge you get after you need it.

33 Spice ups
9

It will be important to know if it was a hack or ransomware ?

Hackers seldom have the time to fill HDD & storage or delete files…whereas ransomware just goes through the network…it even makes things easier if you like to use mapped drives…connections remain open to the other sources…then from one source to another source via mapped drives.

Then mapped drives sometimes store user passwords…especially if you use a batch file to map drives or save the password…anyway malware does not need to see password

2 Spice ups
10

Then instead of using tapes or if you do not use tapes…encrypt your backup sets.

It is likely that the backup snapshots etc was not deleted but the log or config was “overwritten” so they became missing…this is dangerous as this is one main source or the malware imploding again…we had a case where we managed to clear almost every case, but someone went to recover the backup solution config and the malware imploded…but it was contained as the AV for most systems were updated and/or patched.

…

This brings to another point…A lot of people said I was stupid as some of the AV or security products were either costly, heavy or “not the best”…but I practice using several or different brands of “paid” products…like one brand for firewall, one brand for certain VMs & another for other VMs, a different AV for backup solutions, built-in AV products for Synology NAS (there are only 1 free & 1 paid available) and if possible another 1 or 2 brands for end points.

Each product may cover 95-97% of all malware…but at least some may cover while some may be left out or slower to be updated. So all I can hope in worse cases is that with the servers go down or the backup solution goes down…but not both…so they should not be using the same products (much like airline pilots on the same flight to have different meals sets at different times so that both will not be down with food poisoning).

8 Spice ups
11

Scary stuff. I’ve seen ransom ware hits before but nothing like the magnitude of what you described. Make it seem like they were possible on your network for a while logging everything possible. Thanks for the insight.

12

Your story mimics the events that happened when I once encountered Samsam. My takeaway from the ordeal was if I’m even in a position to design infrastructure, I’d see how I can incorporate some of the concepts of “Lanless” design – where applications, services, etc. don’t trust or allow connections or allow data modifications just because the request comes from something on the LAN.

5 Spice ups
13

Ouch, that sounds scary! I agree with what some of the others have said: it sounds like they were inside your network for a while, so consider if any of your sensitive data may have been lifted.

This may not have been much use if they were inside your network scraping it for configuration and access and will only protect flat file shares, but you could deploy FSRM rules against honeypot files and folders. Create honeypot folders and files at the top of each sensitive directory and monitor them with FSRM. If FSRM captures any change to the honeypot you run a script to forcibly disable network access for the user that attempted to make the change. We do this and also monitor for accidental deletion of the honeypot too.

4 Spice ups
14

+1 for honey pots!

Oh man I feel for you, what are your superiors saying? Are they supportive of you or essentially throwing you to the wolves?

3 Spice ups
15

Thanks for posting this. Takes a lot of guts. I would highly suggest subscribing to the Veeam support forum weekly digest/newsletter. This exact attack happens pretty regularly. Gostev has been preaching in that newsletter to have air gapped backups as part of your backup design. He’s given a few examples of good strategies over the last few weeks.

1 Spice up
16

Completely agree. We had RDP exposed to the Internet and I worked out we were suffering an attempt to access every 15 secs, 24/7. Needless to say I have shut that particular hole in our firewall and now users must access via VPN. In this day and age it’s the only way.

I also would say that 2FA is another essential weapon in your armoury.

3 Spice ups
17

Oh my…I don’t even have words. I have come across a crypto/ransomware before but never to this level. I agree with many already on this thread that think this was an inside job or at the very least they have been there for a while; quietly monitoring in the background. I hope and pray that I never see anything that bad in my career. You will be in my prayers…

Please share more if you are able to; I would love to try to make this a learning opportunity for myself and others here. Good luck and you and don’t forget to take care of yourself.

CK

1 Spice up
18

People used to laugh and say that tapes were old hat. Using Veeam I backup to disc, then disc to tape. What a life saver. I had a ransomware attack and the tape saved my life… I smile wryly to myself whenever anyone tells me that tapes are so 20th century…

12 Spice ups
19

I feel like the worst part when reading this is that your backups were ruined too. I have often felt that if something happened to the servers they could just be restored from backup but if you don’t have that, I’m not sure what I would do. We’ll find out I suppose when we do some DR testing here in the next couple of months.

1 Spice up
20

Wow, sorry to hear that you had to live through my worst nightmare. We use tape backups still, though I have been trying to move away from them, this however, makes me rethink that strategy.

1 Spice up