We all had a good kick out of this (after it was resolved that is).
Recently took over a facility’s IT needs over a year ago. Updating a bunch of old hardware, software, infrastructure, etc. One nice day all of sudden an external clinic calls saying they had a small power outage and the wireless was down. Verified with staff AP was on, but kept power cycling. All other APs in the organization are just fine. On the controller verified the AP was not connecting. Myself and another staff member were going to run out to troubleshoot and thought we would prep another AP we had in the office as a spare. Turned off hot spare, pulled off mount and decided to test it just to verify it connect back to the controller and capture healthy logs that we can use to troubleshoot the other “bad” AP. Re-plugged in spare AP and now it does not connect to the controller?
Consoled into the spare and saw some logs about a certificate that was out of date. Okay, we thought, this setup just probably had a cert. we missed updating when we re-up’d our certs. Searched ALL over the controller and all the certs were up to date. Pounding on the research and vendor site. Low and behold we stumbled on a small paragraph deep…deep in a dark cave of the vendor release notes for the version that had a unique “feature”. A cert used for AP to controller communication, not visible via command or UI, and amazingly hard coded with a 10 year expiration which was up 2 days ago. Any AP that reboots will not be able to create a new session connections with the controller. Disabled NTP, manually set clock back a couple of year, and moved the controller refresh project higher on our project list…a lot higher 
Any other fantastic hard coding stories? XD
27 Spice ups
Neally
(Neally)
2
Dang respect for being thorough on going through the documentation.
10 Spice ups
I bet they never dreamed those AP’s would last 10 years!
5 Spice ups
A couple of week before, our crew were wondering how long some of these systems were in place. We found an answer to at least one 
1 Spice up
Rivitir
(Rivitir)
5
Wow… that is sad… and dumb. Awesome job figuring it out though.
Mike400
(Mike400)
6
You’re probably the first person to ever read the documentation on that AP.
4 Spice ups
documentation. whats that?
4 Spice ups
Omg, can’t you just see it? After about 20 minutes of jacking with it, in the trash it went! Pretty sure that’s what I would have done.
4 Spice ups
robhall
(robhall)
9
Reminds me of the hard-coded back door discovered in in Juniper’s ScreenOS a few weeks ago:
I was introduced to an old green screen program working on an AS/400. The developers had hardcoded an administrative account; username = admin, password = admin.
There was a “user” account called “Administrator”; but no-one used that. The staff were supposed to be using individual accounts that had to be set-up for them; but they mostly were using the hardcoded admin account. We couldn’t even change the password on it.
3 Spice ups
dennisbarr
(Dennis3779)
11
Great bit of detective work. Sometimes it’s all in the details, printed in tiny text, buried under fifteen other footnotes, etc., etc.
Hard-coding anything is pretty well frowned on anymore. It’s great to know that at least there’s been SOME progress in the way we program…
techelp
(Techelp)
12
Good job on finding that. Sometimes I wonder what goes through the minds of vendors when they write this. Any reason for a 10 year expiration?
corey901
(Corey901)
13
There is a reason, built in obsolescence. While it is a rare piece of long term thinking by that company, it almost guarantees a new purchase when it expires. It is still kind of shady to build something like that.
1 Spice up
stopthenoise
(Captain Frostbyte)
14
After a max of an hour, I would have binned it and replaced it. Cudo’s on the nice detective work (and finding the manual for a 10 year old device!)
Ya, that was an interesting read 
I wish we could do that sometimes. This situation unfortunately the problem cert is on the controllers which would be a nightmare to replace same day.
We recently had to replace a java plugin on our website because the there was a cert on it (from the software vendor) that expired.