I’m a manager for a small manufacturing company that runs 3 shifts a day 5-6 days week. 200ish employees about 175 devices across 5 different buildings in 3 different cities. My team is a helpdesk guy for general troubleshooting (sometimes struggles with the basics, but is great with people) and a developer/system admin (primarily supports our ERP).
This weekend we had a host crash while I was out on vacation through Tuesday. Tuesday morning my helpdesk guy calls and we walked through a few things. Not sounding good I called our dev to check the servers. He can’t access a host but is WFH, so it’s offline and we’re not sure why. Helpdesk guy isn’t comfortable in the server room and I don’t to make things worse. My family was already on the way home and we were 2 hours out. The impact was printing in a few buildings and a mail server for pushing ERP and copier emails to O365. This delayed reports and a few business flows, but nothing on the production floor. I told them to wait until I get there vs trying anything else.
Within 30 minutes of getting to the office I have host online and services restored. From start of 1st shift to resolution it was about 3 hours. I’m not thrilled about it, and I’m already making adjustments and improving documentation to help in similar situations. But it wasn’t terrible. Of course our firewall blipped later in the day causing about 3 minutes of internet downtime too. Not our best day in IT, but even still neither of these issues were earth shattering.
Yet just about every office I walked into Wednesday brought it up like it’s was the end of days. Pretending business stopped and nothing could get done… it absolutely wasn’t that. Some even told me I can’t take vacation because of it (not knowing I came in). Normally, I let this roll off my back, but this time it is sticking.
I’m sure it’s due to anxiety about having another, longer, vacation next week already. My family is going to Disney where I know service/wifi is not good, and my wife and I have too much invested (time and money) to let work be a distraction. I never like taking time off because of the unknowns, but I know the importance of time for family and just disconnecting. I’m compounding a recent near miss with the fear of being away. I’ve been at this for almost 15 yrs and dealt with some stuff over years but this is hitting different.
I don’t need a change or to run. I just felt like I needed to get it out. Thanks for reading and know that I’m not looking for solutions or advice here just letting it out. Maybe some others can relate and know it’s OK to feel like shit or be worried about things sometimes even when you know better.
39 Spice ups
itguytk
(ITGUYTK)
2
I sometimes feel like that, I have a smaller invironment but, here its just Me. When it all craps out I have to deal with it.
9 Spice ups
This is reason I value our relationship with my outsource backup so much. When I am gone, I know that problems are dealt with and in case of a major emergency, I can lean on them for extra resources.
Yeah, users can be brutal when things go a little sideways but most of them should be happy they are not held to the same standard we are.
11 Spice ups
We have the same anxieties/issues here, we are a bit bigger, but are all so highly specialized, and money constraints keep us from having multiple people for the same roll…We do our best to cover when others have vacation, but it’s never easy, especially knowing that your co-worker is on leave, and we REALLY don’t want to bother them…
2 Spice ups
It has always amazed me when people react to small outages like it is a personal assault on all they hold dear. I rarely have seen any of the aggrieved come in at zero dark 30 to reboot a firewall (or petty functionary equivalent) and never seen a delayed vacation or canceled pet spa appointment from them. It has always seemed to me that the loudest complaints are from people who were in the process a making a hash of their tasks and are making sure everyone knows that an IT problem is their get out of jail free card.
As with you, I always try to let it slide but don’t always succeed.
Deep cleansing breaths.
8 Spice ups
pcase2
(pcase2)
6
I’ve felt your pain before man. Had a customer that refused to upgrade a windows 2008 server that was basically running there business we had warned them for years!!! and one day it has a drive die and we have to start emergency recovery procedures. It took a couple days but what was really annoying was that the customer would call us every hour trying to figure out where we were at in the process and how much longer it would take… You have had years of us telling you to get a new server! You do not get to complain that it is taking so long on 14+ year old hardware!!!
9 Spice ups
If it helps, your situation sounds almost exactly like mine. I’m the manager with a help-desk guy & a dev. Around 215 computers & 175 people spread between 3 buildings.
First, having a second pair of hands is a huge relief (I’ve only had my help-desk guy for about 6 months)
Second, people do not know how much work goes into keeping IT going, this is just the proof.
Third, if that is so business critical, do they pay you enough not to have a personal life? Or should they hire someone another backup tech?
4 Spice ups
NULL
(NULL)
8
I feel your pain, for sure.
I, too, rely on an outsource backup who is familiar with our network. I prebuy hours from them, and willingly spend some of those hours just making sure the main support tech for us up to speed.
4 Spice ups
Once they pull the “You shouldn’t take vacation” card that is where I would get upset too. Am I not entitled to use that time? Emergencies happen but if they cannot survive without you being there, maybe they need to look into adding at least one more staff. My team and org size is remarkably similar to yours and we have had a few incidents. Even so, I am still happy for my boss when he gets some time off as he works harder than anyone I know.
3 Spice ups
crltnfsk
(CrltnFsk)
10
Before I had backup, it never really felt like a week vacation was a ‘real’ vacation. I was only ‘allowed’ a one week stretch at a time. Monday-Tuesday was me worrying that at any moment the problem call would come in, Wednesday was a real day off, but only if Monday-Tuesday was quiet… Thursday-Friday was spent thinking about how Monday was going to go, and how much additional work would be piled on. I can now take 2 weeks without much worry.
3 Spice ups
James404d
(James404d)
11
Got to agree with @rhummel here. I’m a one man shop for a single location manufacturing operation with about the same number of people and devices so I feel your pain.
2 Spice ups
patti8216
(Patti8216)
12
Enjoy vacation. Don’t understand how people think printing is such a high priority. Print to PDF and email. I have 2 high speed MFP devices on each floor of the building. One goes offline and requires a service call and people do the chicken little thing.
I have an MSP I contract for extended support, when I go on vacation 3 engineers are available to assist. We do pre-pay some hours to make this easier on their end to accommodate.
3 Spice ups
Issues are always magnified by 10 fold in end-users eyes.
3 Spice ups
shnool
(SHNOOL)
14
I have been you, and in many ways am still you.
I was the only guy at a manufacturer for a few years, then it was just me and another guy (fellow member here).
Having just one other guy is a huge hurdle to overcome I get it, but now it is a worthwhile time to bring that to management.
With that, keep in mind you aren’t getting another you. As a good manager you must find the right mix of outsource/insource and personal grit to deal with it all.
There are a lot of IT types like you out there. Many of them are reading your post.
I too must agree with @rhummel , this is evidence that the level of service you provide has been noted, and is noticed in your absence. Now is a good time to put on your big boy shorts though, and look like that manager title, and say stuff like “business continuity” and “Service levels” to management. I’d throw in the idea that a manufacturing plant that runs 24x6 needs more availability than a 8x5 IT guy. That “on call” hours are a great idea, but impossible with just 1 head.
Let the business set level expectations. My directive to you is, find that one thing that can happen, when you are X miles away on vacation, and ask the business partners how much will the business lose if it takes you 8 hours to get them back up and running. This is all part of Business continuity planning, and Disaster Recovery Planning, and should also be part of your own personal growth into the idea of Incident response, Recovery, and “Access/Availabiity,” right?
Go get 'em, and you some well earned vacation time.
3 Spice ups
davecork
(davecork)
15
You’ve got my empathy. No need to offer any further words of wisdom, because others have said what I’d thought about saying.
2 Spice ups
I have definitely been in your shoes before.
1 Spice up
Yep, been there and felt that way. But it would never seem to fail…when I would take a day off something would happen and I would get called. I hated that.
1 Spice up
james485
(James485)
18
Yep I’ve been there but Robert5205 is right.