1

After an internal promotion (and our previous 2 IT managers leaving without documenting stuff) I’ve inherited my company’s IT systems (and problems ofc).

The fact is, being a Web Developer primarily, I have very little experience with enterprise level systems, but a good working knowledge of Windows. I now find myself in a position where I’m using outdated documentation to try and piece together the IT infrastructure here, in both Hardware & Software.

So now I’ve got Spiceworks installed and it’s giving me a clearer picture of what’s going on, a helpful first step in the world of IT management but I could use some general advice on how best to keep this ship afloat!

I’ve included a rough outline of the setup of the business afaik, but my main question is what steps would you as an IT professional take in a situation where the documentation is either outdated or non-existent? What software/hardware would be of benefit to me performing a full audit of the business IT?

I’ve made this an open discussion as I’ll bet there is no single correct way to do this. What i’d like to achieve is simply getting a handle on the systems, see where I can make improvements/reduce costs, and of course, ensure that we have a good backup policy in place in case the worst should happen, although I’m still new I’ve already had to replace a disk on the storage server, thankfully it’s a RAID5 setup so nothing was lost!

A quick rundown of the set up:

Total endpoint users: ~15-20 (Windows 7 & 8.1)

Server(s): 4 Racks - 1 Domain Controller, 1 SQL server, 1 Storage Server, 1 Automated Process Server

1 External Web Server (Windows)

VoIP Phone System (Unknown amount of devices)

1 Network running 3 VLANS: Internal Wired Network, Internal WiFi & Guest WiFi

Watchguard Firewall

Thanks for reading!

4 Spice ups
2

I came in to my current position with crap documentation. Document everything.

Doc’s I have created: Network Diagram, IP List, Switch and Router Configurations, Firewall Configuration, ISP Information, Workstation Inventory, License Inventory, Setup of Applications (specifically if you have apps that have weird setups), Backups (When do they run, Where do they go, What is on them), Restore Plan (Know how to restore files, folders, or the whole server before you NEED to).

Then, when you think you are done documenting, dig deeper.

1 Spice up
3

Hi James,

Thanks for the advice, it’s always good to get some pointers on things. Seems as though I’m roughly on the right track as I’ve documented most of the workstations, servers and software. There’s one piece of software that’s giving me headaches but thankfully we have support for it. As for the network the cabinet is a complete mess with unlabeled cables going everywhere (as is tradition), so I’m having to manually pull each one and run NMap scans to see what goes offline/run around the office asking people if they’ve lost connection each time.

My colleague is attempting to get a handle on the backup systems, but we’re both in a little over our heads with it. DB’s are being backed up but they are full backups each time, not incremental/differential so there’s a huge amount of disk space being chewed up by it. Backup drives that are taken offsite are just old HDD’s with exposed PCBs so there’s plenty of room for improvement there. I’ve noticed the storage server also has full backups but the drive its being backed up to may be reaching the limit in terms of capacity.

On the plus side of things I’ve installed IPBan on our external web server with a completely open RDP port as it was getting brute forced to hell, thankfully it seems no one’s managed to get in but a password reset on all the accounts will be needed to make doubly sure.

One particular issue I can see being a problem is our VPN policy - users are logging in remotely on machines without out endpoint protection and exposing the network to other threats, so some re-Neducation may be required there, unless there’s a way I can check if they have protection before allowing the connection. Some more research to be done there. Thankfully the original IT guy from 10 years ago seems to have known what he was doing as the systems are pretty stable, just idiots coming in afterwards (myself included potentially) cutting corners and the like. For instance the last IT manager made an “improvement” to our website’s login SQL which resulted in allowing access with just a username/email with no password. He convinced me it was a JS validation issue and had solved it by, I kid you not, using a noscript tag… Once he left and I was able to grant myself access to the SQL Server it took me 20 mins to find and fix it, again thankfully no one caught on and did anything silly.

I guess I’m just worried that some disaster will strike and I won’t know how to solve it, but I guess that’s why we have google. Fingers crossed for a stable few months while we get this backup system nailed down. Any advice on that? Namely 4 windows servers (1 of which is running SQL) and about 20 workstations with a centralised storage. I like the idea of just imaging discs so re-installing software isn’t needed but not sure who/what is the best service/software for it.

Anyways, back to the grindstone, wish me luck!

Thanks again!

Dan

4

Regarding Backups, your company needs to determine the RPO and RTO first. How much data can you afford to lose and how much time can you be down. Once you determine that, you can look into adjusting your backup strategy. One thing I like to do before committing to a solution is getting a trial of the software and test hardware that I can play with. I want to know I can recover anything within the time frame of my RTO. It’s not cheap, but I would look at Storagecraft ShadowProtect for your servers.

5

What’s the current strategy?

6

Hi Dan - if you would like someone to warranty check your kit list to confirm the support status on any of your hardware, please feel free to get in touch with us, and we can advise you on the best options for your environment.

Support Warehouse are an HPE partner who inform end users when their existing support entitlement expires. Customers then have the option to renew via Support Warehouse, or to go back to their preferred reseller.

@Dan3811

7

My colleague is currently testing some software for backing up on a machine, the initial test failed spectacularly but the second test went off without a hitch. I’m not sure on the RTO/RPO as its something I’ve not looked into yet, all I know is that backups are being taken at least once every 24 hours of the servers/storage/dbs but no imaging of the actual drives. So any recovery of a server failure would probably require reinstallation of the OS as well as any software before being able to load the data back in. Ideally I’d just like to image the disks (as well as backing up data on a more regular interval) so any recovery can be performed relatively quickly by first loading the image then loading the latest dataset. Unfortunately DB’s and replication aren’t my strong suit so this is something else I need to figure out in order to keep the data fully up to date. Sorting the backups will be my next priority after I’ve finished sorting out this topology, just spent the last 2 days ripping out old redundant cabling and mapping the current connections within the building.

8

Hire a professional consultant / IT company to determine what is what, and get the business on their feet. If you aren’t experienced in this, all good will and intentions only means you will cause more damage.

Maybe @danielle can help you out.

9

I would definitely suggest a audit of every system not just security (though that would be step one). Verifying all backups is also really important make sure what you currently have works and then go back and take a look at what the business really needs. Chances are if things were that big of a mess on the surface you might have some very deep major issues somewhere as well.

Bringing in a consultant to take a look might be very wise especially if this is not your area of expertise. That way they can walk you through what you have, what you need and the road map to get there.

1 Spice up
10

Note the discussion title please - If we’d have wanted to do that, then I would have asked about it. The whole point of this discussion was to avoid bringing in expensive 3rd parties without being able to confirm what they were telling us. I’m sure you all know that IT is an industry with a large amount of BS being thrown around, I’d end up with a quotation for an IT equivalent of headlight fluid if I did that without getting an good grasp on what we have already. Just because I’m being open with my skill level both here and with the management doesn’t mean I’m clueless. I’ve presented far less info here than I actually have in order to keep things less complex for discussion, as well as security.

So far we’ve managed to rewire the network to separate the VoIP traffic from the rest, and perform an audit of the local workstations; Some need AV updates, some need windows updates etc, nothing major to report back there. Backups look ok, even if they are a bit OTT with creating a full backup each time, however the network has minimal load overnight so that doesn’t affect operations, ironically they are vhd images with separate db backups, just like I was planning on.

The servers are interesting, no virtualization but separate boxes for different things, one runs the SQL, another is the Domain Controller/DNS/AD Manager, one is the storage share, and a final one runs automated processes and the intranet. There were also 2 servers that were offline, booting them back up they’re still in good shape but are no longer being used for anything, so we’ve decided on utilising the extra hardware for creating a development environment/testing server backups.

All in all, I think we’ve got a good handle on it now, thanks for the tips (In particular james404d for actually giving me useful thoughts). Moving forward will mostly be software related rather than changing hardware, hence the desire for a development environment. As mentioned by everybody, hammering out the backup processes is a top priority but it seems the procedures in place are adequate if not ideal, other than that, one of our VLANs can’t connect to the internet since the rewire last night so I’m assuming something’s up with the routing from the firewall.

Further suggestions are welcome, as long as they aren’t “Make yourself redundant by outsourcing everything for double the cost”. I probably will be getting an IT services provider involved at some stage in the future, but their role will be minimal and will likely only be used as an offsite backup storage vendor.

Thanks,

Dan

11

Hi Dan,

In terms of organizing the phone system stuff, I can tell you that it’s been very useful when I’ve seen IT departments have the below items documented. It’s saved them a ton of time and mistakes (i.e. disconnecting lines that they actually needed)., when looking to upgrade/streamline their communications.

  • Know your phone system make & model and have 3 vendors you can call to repair it (in case your first choice can’t get there for a couple days). It might also be useful to make sure you always have a few back-up handsets, in a designated place, ready to go when your next one dies.
  • Make a spreadsheet with your phone system extensions and assignments. This makes it easy to assign them to new employees.
  • Make a spreadsheet of all of your active phone numbers, phone lines and their purpose. At least start chipping away at this one, by obtaining a list of all active phone numbers from your telecom provider(s), then try to figure out where they’re all going. You don’t know how much money we’ve saved companies just by pulling up their phone bill, getting a Customer Service Record (CSR), from their provider and having them find phone lines that are no longer needed and never disconnected.

Feel free to PM me if you’d like more tips on how to do this stuff. Happy to help.