1

As Texas’ Gulf Coast comes to terms with the destruction wrought by Hurricane Harvey, the technology infrastructure of Houston and surrounding communities is being tested like never before.

IT pros epitomize resilience and focus in the face of adversity. We support you, and we want to know how you’re supporting those around you, both professionally and personally.

While they deal with the emotional cost of the disaster, the regional tech community is also focusing on disaster recovery challenges that pit the US’ fourth-largest city against 100-mph winds and flood waters that are still rising.

On Friday, Spiceworks’ Susan Adams spoke with IT administrator April Lundy ahead of the storm about the steps her office was taking to prepare.

Now, the Spiceworks newsroom wants to hear from SpiceHeads in the region about how they’re rising to the occasion: helping their companies and communities while caring for friends and family in the area.

In the comments below, we would be grateful to hear about your experiences preparing and responding to Harvey. (Please tell us if we can reach out to you with follow-up questions — we understand the workload a disaster imposes on the people seeking to get the region back to business.)

We value the opportunity to share the lessons of our Community with other IT professionals via accounts like these from other natural disasters:

Spiceworks’ editorial team appreciates your dispatches from the front lines, and we want to broadcast what you’re accomplishing this week.

@susan @aprilwlundy

124 Spice ups
2

First of all, my thoughts and prayers are with everyone impacted by Harvey. I urge anyone who has the means to assist in rescue, recovery, and rebuilding in your community, at the very least it helps you cope and strengthens your bonds with the people you live with. I spent a quite some time helping to sort donations and deliver meals to those that needed them in my area after Sandy, there were many that didn’t have it anywhere as lucky as I did.

Also thank you @matthew-r-spiceworks for the nod. As always there is no one size fits all answer to DR or BCP and when you’re a Jack of all trades Admin It can be tough to know where to start. For those of you working through your DR and/or BCP procedures now document as much as you can, determine what may have helped you this time and that can guide you for “next time”. If there is no IT involvement in your company’s DR or BCP say something, start the conversation, but don’t just blame all of IT’s failings on lack of involvement, instead document how you may have been able to implement a solution (maybe you already did and it was shot down) and try and get your concerns heard.

Finally: As you rebuild consider your other processes and see how they intertwine (they do, they all do, it’s a web for sure) and see what other improvements may need to be made.

If there is anyone out there in need of advice reach out, we are a community. (My inbox is open for anything personal however please do not PM me for technical questions, search/post on the forums)

Stay Safe

-P

26 Spice ups
3

It’s really heart breaking to see what mother nature can do, some of those areas have had FEET of rain just in the last 24 hours and now the storm is gaining strength again as it moves back into the Gulf only to move north again. I cannot imagine what the people in Houston are going through and even once the rain stops the water will have no where to go for some time.

I’ve already donated privately and will again through my company which generously matches donations to organizations dollar for dollar up to $500 each time. If anything these events at least show that humans still have compassion for one another and communities truly band together. I suppose that’s the one silver lining in all of this.

From a technical standpoint I can imagine larger companies have plans in place for such an event in terms of co-lo, cloud, hybrid deployments, etc. Smaller/medium businesses with less capital to spend on I.T. might be harder hit if they don’t have more robust plans in place. When you consider some smaller companies might be doing off-site backups via tape to 3rd parties like iron mountain it makes you wonder what type of plan they have in place. Looking at Iron Mountains website they have a facility directly in Houston in the city. so it makes you wonder how they’re protecting all those tapes.

4 Spice ups
4

HI All,

We are in Houston on the northwest side. I’ve measured over 32 inches of rain (roughly 80 centimeters) since Friday.

We have weathered the storm relatively well. Our (local and for our cloud services) cloud facilities in town are not in peril, and our replications to Dallas and Las Vegas are happening.

We went through, what was a more significant event for us, during Hurricane Ike. The storm only lasted hours, but we were without power for 11 days. Our generators lasted (with fuel refills) and we were able to maintain services through IKE.

We are not to the clean up stage yet, as we cannot travel around the city. We do have notifications of customers without power, but it might be next week before we can get to these locations safely.

Our brothers and sisters on the gulf coast are a resilient bunch. We will dig out and resume services as normal.

I can speak for all of us on the coast, Thank you for your thoughts and well wishes.

John

17 Spice ups
5

" IT pros epitomize resilience and focus in the face of adversity."

:confused:

3 Spice ups
6

If you live in a likely flood prone area… Time to McGyver your solution. Everyone should start buying these , building these , and charging them with whatever you can find as a power source. A good flat bottom boat with some emergency gear in it might also be a good idea, for rescuing dogs.

3 Spice ups
7

Ya don’t think so? I always look to you guys when things go wrong, big and small, technical or not. (I know it’s a bit of stereotyping, but people who go into IT always impress me as fixers.)

6 Spice ups
8

I would agree with that for many IT hardware people, but not all, and then there are the computer science types that are focused only on the software couldn’t fix a paperclip. So, your mileage may vary.

5 Spice ups
9

http://www.khou.com/news/local/resident-uses-aquadam-to-protect-home-from-floodwaters/238455798

looks like someone did it once before although I imagine $8-10k is out of range for most people. in this guys case it says the water got to 27inches but didn’t get through so I question if it could even hold up to the amount of rain people got this time around.

4 Spice ups
10

Mother nature is fierce and unpredictable but humans will persevere through disasters like this. Kudos to SpiceWorks for linking the DR links and for those not in the affected area, time to pay attention because the information that this article could potentially generate might be invaluable to you one day. To Texas, have you tried turning it off and back on again? Kidding aside, may your DR be thorough and your backups successful.

2 Spice ups
11

It might be it sounds like a winning block in Buzzword Bingo.

3 Spice ups
12

I think it’s a matter of perspective.

Normal day at the office where Steve in Accounting just infected the network with cryptolocker for the third time in a month: sure.

Natural disasters… While I know some of us are in the guard or may just generally be “helpers” I can think of plenty of people who are more resilient in the face of adversity right now (like the people boating door to door to help with evacs).

Networks get rebuilt, systems will be back up and running, but always after the safety of the people who rely on them.

3 Spice ups
13

I don’t think the original statement is saying IT Pros are Superheros but rather, when the literal shit hits the fan most of us can handle pressure while we watch a person think their life is over just because Steve infected the network again. If the article is comparing this industry’s ability to setup search and rescue missions with the end goal of saving lives during a major natural disaster then I read it wrong but I don’t think that’s what the writer intended. Maybe step back and remember some people want to inspire when they write and hope that it helps someone make a difference in their day.

3 Spice ups
14

one of my favorite disaster recovery plans for SMALL businesses running 1 physical server with 1 virtual. The virtual machine is the domain controller, terminal server, and file server all in one. Once i’ve installed the above I install a DATTO 1 TB for 175/mo and run a local and cloud backup of the server. Copy all data from the client machines and move them over to the terminal server and then let it replicate to the cloud. True business continuity for a small business. And a excellent money maker as this solution generally costs the clients about 5k to set up properly.

3 Spice ups
15

The aquadam home page shows them holding back over 21 ft of water with the system, but I’m sure that it with a bilge pump would still work :slight_smile: I was actually wondering since Houston is a big datacenter town, how many different server providers are down in that area?

1 Spice up
16

I am too far away to do a thing of any meaning and all I can do is to share my knowledge so if anyone has troubles post them and I’ll see what I can do. I remember last year’s Airport bombings here and my inbox was crammed with messages asking if I were ok.

2 Spice ups
17

Thanks, yes! I’m not trying to mythologize IT pros, and I’m not saying no one’s mileage varies.

But over the years, I’ve found that different professions attract different personalities and cultivate different qualities in people … And by and large, my friends and colleagues in IT are among the folks who keep their heads and prioritize in a crisis. :slight_smile:

Personal aside: I’ve experienced all three big NY blackouts (although I don’t remember the 1965 one); the Loma Prieta earthquake in SF; and the September 11 attacks in Manhattan. Each experience reminded me how immersive and how fragile our urban grids are as reality seemed to tip sideways — and how important it is to have people who can handle a shift in circumstances, whether they’re emergency personnel or military troops or delivery drivers or shopkeepers or IT workers. (Or editors!)

People handle stress in different ways — but based on personal experience, I’d say the tech pros of my acquaintance tend toward the get-it-done end of the spectrum.

3 Spice ups
18

I don’t have a lot to say at the moment simply because too much is going on right now but we are mobilizing disaster services as we speak to provide every level of support needed. My company came into being providing services during Hurricane Katrina so Harvey is our sweet spot. This means we may be providing shelter, food, base management, housing, gov support and various other tasks. During the Texas flooding in 2008 we served over 360,000 meals. The thing I am most impressed by is this is usually pulled together in a matter of days. I can’t take credit for that. My higher ups are fantastic at what they do.

However, the role of IT in these situations is to provide support as much as possible and fill any gaps that may arise. So today we are coordinating with wireless carriers to make sure we have enough coverage once we hit blackout areas. However, I also took the tour of an RV we’ll be riding in to make sure I know how to run the thing (Empty sewage tanks etc.) once we get there. I may be hooking up temporary offices one day and scooping food for relief efforts the next.

No, IT are not superheroes but I’ve found those that make it in this business are great in the clutch situation. When the pressure is on you’ve got to be able to keep a level head and get that mission critical service back online. No matter who is yelling at you. :wink:

11 Spice ups
19

Remember Fukushima. Don’t mount your emergency generators where they can be flooded, ever. Dittos for all your electronics. High and Dry is where electrical equipment belongs, or it’ll be wet and dead.

1 Spice up
20

Wasn’t agreeing or disagreeing. One commenter went with Buzzword Bingo, now that I agree with.

1 Spice up