So how's this for a week?

alex3031 · 2011-09-02T10:18:08.000Z

I don’t mean to diminish anyone hit harder than we were, and I specifically was, here in CT, but this has been a week.

Monday we had no power so we closed because we couldn’t do anything.

This morning I walk in to have my supervisor tell me that our ERP server has taken a turn for the worse. It had shut itself off last night, and this morning upon turning it back on it rebooted itself. We ran some diagnostics and they came back seemingly OK. Then we call IBM and make use of our extended service/warranty contract, after about 3 hours of back and forth with them and attempting to run their diagnostic test suite (trying because the server kept rebooting at shorter and shorter intervals) We get a tech to come out with what we think, but are not positive is the hardware component at fault. I only requested a tech instead of just the part in hopes that if this isn’t the culprit they may have other components they can swap out to find the issue (which it sounds like they don’t) or they may see something we missed and know exactly what needs to be replaced immediately.

Anyways long and short of it most of the company is sitting on their thumbs because there is little to nothing they can do as we don’t have some sort of backup system, or manual backup plan in place to address issues like this. And I’m sitting on my thumbs waiting for the tech and part to show up.

Anyways how is everyone else’s day going?

rmuniz9336 · 2011-09-02T10:24:56.000Z

Obviously a lot better than yours! Good luck, Buddy!

claytonmurph2569 · 2011-09-02T10:26:34.000Z

dude, that sucks. however, if it makes you feel any better, I just got Poblano 1 post before this one! hope your day gets better and you don’t get any calls this weekend.

jasondavis6900 · 2011-09-02T10:27:00.000Z

I went through something similar about a month and a half ago, and to top it off: it was my third day on the job, and the previous administrator’s backup plans were based off of hopes and prayers.

Talk about a trial by fire.

skillet · 2011-09-02T10:31:49.000Z

And I thought my firewall issues were bad. Here’s hoping you get everything back up and running soon.

alex3031 · 2011-09-02T10:33:13.000Z

Well my experience isn’t nearly as bad as LiterateGeek, so I find some solace in that.

CAG congratulation on Poblano and as for getting called in on the weekends, the only nice thing about have a supervisor like I do who did my job before I came on is that he is salary and I am not, therefore I only get called in if he needs my help or can’t figure it out.

and rmuniz, thanks for the well wishing we need it, lol

alex3031 · 2011-09-02T10:34:17.000Z

Thanks skillet, really hoping that tech doesn’t take until 3:20 to get here…

brian9460 · 2011-09-02T10:35:42.000Z

Yes, that truly does suck!

It could always suck more, such as:

like IBM not having the parts you need,

or the contract somehow expired and now you have to wait for financial approval process.

or you find the backups of the server are bad for the past 6 weeks.

or they decide it’s all your fault and they decide to upgrade you to customer…

(Our internal phrase for exited employee…)

alex3031 · 2011-09-02T10:37:14.000Z

upgrade to customer, huh? I like that lol

morrelstamant · 2011-09-02T10:41:36.000Z

Tell your users to make Scumbag Nic meme posters and post them in my contest thread

http://community.spiceworks.com/topic/154512-scumbag-nic-the-meme-off?page=1

It’s totally sanctioned by Nic

He even started the base meme poster at Meme Generator

philipdestefanis4212 · 2011-09-02T10:46:38.000Z

Good luck with it Alex…you will get things back up.

scott696d · 2011-09-02T10:55:39.000Z

As someone who worked at IBM, I hope the techs they send to actual customers are better than the ones we dealt with internally. I would be surprised if s/he has the necessary spare parts in their vehicle, which probably means them overnighting the parts to you after they go through the whole troubleshooting process. Anymore, most companies would rather not spend the money on parts and hope their supply chain can handle the strain. Back when I was a traveling service tech I kept a complete spare parts inventory in my car. That way no matter where I went and whatever I encountered I would be able to fix it right then and there. I miss those days sometimes…

Good luck!

it-matt · 2011-09-02T11:05:24.000Z

oh boy i feel bad for you Alex but I know the feeling.

This is how things work around here.

We see a server or UPS or something going bad, so we tell the company execs that it needs to be replaced or it will go down soon and we will be screwed. Two months goes by. That device fails and they get pissed at us. but we just come back with all the quotes and recommendations we made that they wouldn’t accept. so then we are down with that machine for a couple days till something new comes in or we can fix that current device. Its awesome. So we go through this often but at least they can never blame it on us. luckily recently we got them to buy alot of backup systems so at least when one goes down we can still work till the other is fixed.

bob-13 · 2011-09-02T11:19:07.000Z

was going to say I hope you week ends better than it started, but my keyboard stopped working

…

Hope things take a turn for the better for you, and look on the bright side… if you were in Canada any “emergency part” you need would need to come from the US and get held up in customs for days…

brianfoley · 2011-09-02T11:19:09.000Z

Sh*t happens! I hope they get someone to help you get the issue resolved soon!

jasondavis6900 · 2011-09-02T11:43:13.000Z

Alex3031 wrote:

Well my experience isn’t nearly as bad as LiterateGeek, so I find some solace in that.

CAG congratulation on Poblano and as for getting called in on the weekends, the only nice thing about have a supervisor like I do who did my job before I came on is that he is salary and I am not, therefore I only get called in if he needs my help or can’t figure it out.

and rmuniz, thanks for the well wishing we need it, lol

Oh it gets better. My first day on the job we had electrical storms, I was getting “The grand tour” and I watched as the servers clicked off and on five times in a row as the building power flickered. Apparently the battery packs of an old UPS had gone bad and the previous admin had purchased a replacement… and ordered the wrong thing and hid it behind the racks so no one would know he ordered the wrong thing.

While I was watching this happen I was like “Priority one is replacing that UPS, what’s going on here is very hard on servers, and it’s not doing any favors to the various raid controllers/hard drives it’s power cycling… we’re adding years of wear here”. Two days later the raid array on the ERP server fails. And it failed about an hour and a half after I said “Hey you know those backups he said he was taking, let me tell you a story about them backups”.

I’ve since learned that this building is cursed and just mentioning a failure causes it to come to fruition.

johnwhite · 2011-09-02T12:20:42.000Z

Man, good luck. Suddenly the cost of downtime is made clear, right?

Spend some waiting time architecting a virtualization plan. :-/

gabriel1594 · 2011-09-02T12:29:56.000Z

It sounds like a power supply failure on your server!

I had 2 computers last week that would restart on their own, sometimes in windows, then it became quicker and quicker, up to where sometimes it restarts at the bios boot screen…

This week has also been a pain for me.

Monday the power went out. I shut down the servers before the UPS ran out. Had to wait (in the dark) if the power came back during my shift to restart everything… it didn’t.

Tuesday is my only day off, I got called in to power everything up and fix everything that went bad, so many UPS units and power supplies were bad, our backups were no longer working, etc.

Wednesday was more of the same repairs. Then for the first time, a user calls me in panic, All her files were deleted after she emptied the recycle bin… As I said, the backup was not working right on Tuesday, so I had to get her older files from the week before…

Thursday there was an earth quake, but nothing was damaged. yay

Today, I have a corrupt hdd in a PC I’m setting up for a new user… easy fix/swap

rosannakrenn5567 · 2011-09-02T12:41:41.000Z

I really hate weeks like that. Mine was considerably better, thanks for asking.

alex3031 · 2011-09-02T13:04:10.000Z

Just an update tech was contractually obligated to be here by 3:20, no show so far! I have called and spoken to a national duty manager that messaged the tech and was able to verify the tech received the message, but we still have not heard from him/her. I then called back and she has now messaged the techs manager, who has not called me yet either, getting ready to call back and about to stop being Mr. nice guy, because that’s who I am but even I am starting to get peeved. I don’t give a rats ass so much that he wasn’t here on time but communicate with your damn customers. If the part isn’t available locally call me and tell me we can’t do this tomorrow. If you are stuck in traffic and going to be late call me, but don’t remain silent!!!

Anyways thank you all for the kinds words and encouragement, I just hope this is resolved soon.