1

Hello,

We use Ubiquiti Cloud Keys (mostly UCK G2 Plus but some UCK G2) and Ubiquiti Access Points (U6 LR and Nano HD), and we’ve had weird disconnect issues recently. We’ve rarely had any issues over the past two years, but we’ve had 3 remote locations have at least 1 AP go offline and either stay offline or stay on for about 1 minute, turn off, repeat over and over again. I’ve looked into this, and it could be a hardware issue, but I’m starting to lean more towards a software/update issue because the issues started happening at the same time for these remote locations.

Is it possible a switch, cloud key, or access point update broke something and is causing our APs to lose power and or network connectivity? Please see the screenshots below for reference:
image

image
image394×508 16.7 KB

image
image1822×755 120 KB

Side note: I know the screenshots are regarding AP-520-009 and in the first screenshot AP-520-004 is offline and AP-520-009 is online. For reference, AP-520-009 has been going off and on over and over again, and AP-520-004 has stayed offline.

6 Spice ups
2

Have you or anyone else tried the obvious, checking the switch port it’s connected to, to make sure the switch isn’t dropping the connection, or that someone hasn’t used the same IP somewhere else, causing the two devices to reconnect as they fight for the same IP?

Try changing the IP first and see how it behaves.

4 Spice ups
3

Sorry for not clarifying in my original post. I’ve tried turning PoE off and back on, turning the switch port itself off and back on, verified that no other device is using the same IP, etc. I didn’t do it on this switch specifically, but I tried a different switch port on a different location having issues and the issues persisted.

I’ll give this a try once the AP decides to come back online after doing it’s routine of offline and back online again haha.

4

so offline in this case doesn’t mean the device is non functional. It means the device is not reporting (communicating) to the unifi controller.

Step 1 would be to verify if the device is actually suffering an issue taking it offline or if it appears to be working normally, but not communicating to the controller.

If its working normally and not communicating you can SSH into it (assuming you have SSH enabled) and use the set-inform command to tell it to call home to the controller and see if it comes online.

set-inform http://[controller_ip_address]:8080/inform

if that works, then I would ask a few more questions about your network setup. Specifically about what your gateway devices are and what you use for DNS. it could be a simple as you need to setup the Unifi DNS A record so devices know how to call home to the Unifi controller.

2 Spice ups
5

I set the AP statically and it looked like it fixed the issue initially (the AP stayed online for 2-3 minutes instead of 30 seconds), but it eventually lost power and restarted again. I’m not sure if that means anything but it’s better than it was.

This is random but do you think channel optimization could cause issues? I looked it up and if it’s set to auto, I heard it might cause similar issues but wasn’t sure if this was true.

6

For AP-520-004, it looks like it’s entirely offline because I can’t ping it and it’s not showing up under DHCP. Granted, this could mean it’s either fully off or just not getting a network connection in general. For AP-520-009, I’m able to ping it even when it shows offline on the cloud key.

We have SSH disabled, so I’m assuming this won’t work since the AP won’t have console connectivity to recognize that SSH is on sadly.

Our gateway device for each location is our firewall, and we use Google’s DNS (8.8.8.8 and 8.8.4.4) for cloud keys and APs. We use our internal DNS servers for most things except for APs and cloud keys though. I thought about this being a DNS issue too, but it seems weird that we’ve used Google’s DNS for cloud keys and APs for years and never had any issues. I’d also assume if it was Google’s DNS that all of our APs would go offline and not just a few.

(jeremyduncan) 7

When I’ve had something like this happen before it was usually a power issue. I put the AP on a POE injector and it solved the problem. I’m not sure why the POE switch stopped supplying enough power, but that was my issue and I’ve seen both the constant power cycling and the device staying offline. If adding the POE injector didn’t work I just swapped out the AP.
Hope that helps.

3 Spice ups
8

Check the uptime of the AP, if it hasn’t changed then it’s probably a connectivity issue not a power issue.

How many APs do you have connected to this Cloud Key? My guess is the Cloud Key is overloaded on CPU/RAM and then causing false alarm disconnection notices.

The Cloud Key marketing materials say it supports up to 40 APs but with features like client historical data, traffic identification, and IDS/IPS enabled it might be much less.

1 Spice up
9

We have about 5 APs having issues, and 2 or 3 of them are showing the uptime go from 0 seconds to 30 seconds, and back to 0 again. These ones seem like a power issue, but there are 2 or 3 APs that don’t show an uptime at all, so I’m not sure if the APs are turned off or just losing the network connection.

Ironically enough, the cloud keys with the troubled APs only have at most 15 APs adopted, but one of our other locations that isn’t having issues has 29 APs adopted.

10

I might have to give this a try since it does seem like some APs are losing PoE entirely. This makes me think a firmware update of some kind did cause issues because I read online that some updates can mess up the PoE functionality. Since the locations having issues are remote, I’ll add this to our list to try the next time we are there. Thank you for the help!

11

What switches are the trouble APs connected to and how many other POE powered devices are connected to the switches? Is it possible that the switches available POE power is at its limits? Are any other POE powered devices showing any issues?

1 Spice up
12

Quick question - do you have automatic updates enabled, and has anything updated recently?

I ask because I’ve been a Ubiquiti fan for over a decade, but their firmware updates can still be a little …unpolished. I’ve been burned a few times by that, so one of the first things I do with a new Ubiquiti install is to turn that off. I still receive notifications when there is a new release, but I always read the change logs and only apply the update if it: A) addresses an important security flaw, B) fixes an issue I already have, or C) provides a new feature I can’t live without.

If you’ve had a recent update, you might try rolling back to the last “stable” build, if only to rule that out.

1 Spice up
13

my question about the firewall devices and DNS is as follows. If you are using a Unifi firewall it automatically knows where the Cloud key \ server is and directs Unifi devices to it. But if you are using a non Unifi Firewall you need to set this up via DNS to ensure all your Unifi devices can properly locate the Unifi controller and report in (so they don’t go offline).
to Do this you need to create an A Record in DNS call “Unifi” that points to the cloud key \ Unifi controller IP. When ever a Unifi device boots up it queries for this DNS record to determine where to report to.

Its a simple thing, but I see it get overlooked so often. Having this record ensures Unifi devices can always determine where to call home to.

There are a few other ways to accomplish this also, but I find DNS the easiest

1 Spice up
14

Our two switch brands are Cisco and TP-Link, and the models vary per location. Of our three locations currently having issues, the models are TP-Link TL-SG3428MP, TP-Link TL-SG3452P, TP-Link TL-SG3452P, and Cisco CBS350-48P-4G. Each of these switches only has 7, 12, 13, and 14 PoE devices plugged into them respectively, so I don’t think this is too many devices where it would cause issues. We only have phones and APs using PoE, and our phones seem to be fine, so I think it’s just the APs. I don’t think they are drawing too much power for what the switch can handle based on my research. Thank you for the ideas though!

1 Spice up
15

We do have automatic updates enabled for our cloud keys and APs, but I’ve honestly never noticed any issues with this over the past 2 years. However, I checked the update history and Ubiquiti did push an update out about a month ago, so maybe this caused issues. I also restarted our APs and cloud keys recently (we do this every few months), so maybe doing this along with the update caused issues.

I might do this even if this isn’t the issue because that’s a good point. Updates can be buggy and break stuff.

What’s the easiest way to roll back an update to test this? I didn’t realize this was possible in the cloud keys.

16

We are using a non Unifi Firewall, but I was wondering if it’s necessary for our APs/Cloud Keys to have access to our internal DNS servers. I can definitely set something up in DNS if you think it would help, but I don’t think internal DNS is needed since all of our cloud keys and APs use Google’s DNS, and only a few are having issues. I might be misunderstanding how it all works though, so feel free to correct me. :smile:

17

this is bad advise. its critical to allow auto updates in todays security world. Not allowing auto updates is asking for a security breach. Its simply not possible to keep a network up todate and secure without auto updates. Sorry but people saying turn off auto updates is a huge pet peeve of mine. It will cause more issues than allowing them will. Log4J being a fairly recent example that comes to mind.

Yes its possible for an update to bring in a bug, but a bug can be dealt with and won’t open you up to ransomware and lawsuits.

This is in the Unifi manual and is best practice for unifi setups. I would 100% move your Unifi devices to internal DNS and create the Unifi record. If you don’t want to use internal DNS the other option is DHCP Option 43 assuming you don’t apply static IPs to your Unifi devices, but you should be using one or the other to ensure they report properly to the controller

2 Spice ups
18

Don’t disregard the physical layer. I had a Unifi switch try to use the AP that was plugged into it as a mesh uplink because there was a microbend in the CAT5E uplink that broke wire 5. Snipped the end off and reterminated and have had no issues since.

2 Spice ups
19

From all the info as I understand it is still not known if the APs are fully loosing power and turning off, or if they are just loosing network connection, or restarting etc.
It is diffcult as they are remote but when one shows offline ask somone to go and check it to see if it is powered on.
Does wifi from these APs stop working when they are offline? again hard to know but maybe a user based near one will know.

Do the switches log any info - For example are there any errors such as spanning tree that blocked the network port?
Are all APs cabled? if so then make sure that Mesh is turned off. I have seen a firmware update cause mesh issues, If the AP meshes to another it can cause a spanning tree issues which blocks the network port on the switch.
There may also be other switch port logs helping identify if the device is going down.

You could try rolling back to the previous firmware on one AP to see if it solves the issue.

2 Spice ups
20

You can check:
https://community.ui.com/releases
for release info and downloads.

Just look up the devices in you network. Read the release notes and commentary by other users to see if they are having similar issues with a particular version.

I will also add that @molan is correct about disabling auto-updates as a general practice. I only do that with UniFi, and only because my environment is small enough to manage security updates manually, which would be impossible in a large org. If you do that, make sure to enable update notifications, read the release notes, and promptly apply important security updates.

2 Spice ups