1

So, over the last 8 months, I’ve really beefed up my Ubiquiti network. The biggest move was moving our IP cameras off unmanaged / 192 network to the main network and they have been so much more stable.

We also moved to a UDM Pro. We’ve been on the pro as our primary gateway / firewall for about 1 1/2 months since we had issues with our cloud gateway.

One problem we have though are random events where over 1/2 of our switches will just go bonkers and go offline and/or readopt…the examples below are from this evening around 5:30 or so…

image
image592×1282 111 KB

image
image592×1282 110 KB

image
image592×1282 101 KB

We don’t have paid support from Ubiquiti but they give advice that is all over the map.

Digressing, our DHCP / DNS is on our Windows Server.

That said, Ubiquiti support said to set the DNS for our WAN’s (we have two) to 1.1.1.1 and 8.8.8.,8.

I did that tonight but is this going to stop these random events? We can’t afford these to happen during business hours anymore.

9 Spice ups
2

When they go offline, does the network go down as well? Switches not checking in vs. not passing traffic. Any commonalities about when it goes down? Could some backups be running? Could a device get powered on about that time that may have a dup ip? Is there anything in the logs of the switch, any ports bouncing or getting blocked by STP, etc.?

6 Spice ups
3

in your windows DNS Server do you have a DNS A Record Unifi that points to the UDM-Pro so devices know its your Unifi Controller IP? Is the UDM-Pro the only Unifi controller running on your network?

Do you have Meshing enabled on your APs? Turn it off if you aren’t using it.

I doubt changing your DNS to 1.1.1.1 and 8.8.8.8 will help. It will cause issues if you have an Active Directory and change this for your AD Clients as they must only have your AD DNS servers in their records to work correctly.

Without knowing the intimates of your network this would be a hard one to figure out, But Shanetech74 is correct I would start by determining if this is just the management component of the switches and APs failing to reach the Unifi controller or if these devices are actually going down and not serving network traffic as step 1.

4 Spice ups
4

Firstly…it is not that you do not have paid support or their advise is all over…the question is WHO IMPLEMENTED THE NETWORK ?
You should be paying support to the vendor that implemented the network and its appliances ? Who else would know what have been done, what features have been activated or not used etc ?

Then are the appliances in and/or

  • bridge mode
  • power saving mode
  • failover or HA mode etc

Did you practise the core switch and distribution switch ? Or you just plug switch ports to any switch and AP ports to any switch and the UDM controller to any switch ? Then did you use NIC redundancy for the switches and UDM controller ?

Then as the support does not know your network, did you or your network vendor provide a network diagram ? Even as logic would tell you to use your DCs as DNS servers and your firewall appliance as gateway ?

Then you need to know what is happening around 5pm to 6pm…like users running heavy EOD reports, servers starting backups, SQL or DB servers doing data dump (another term is DB backup) or users starting windows update (some companies ask users to run windows update then “update & shutdown” before they leave).

3 Spice ups
5

I prefer switches etc on fixed static ip-addresses, manual not DHCP. (anything fixed to the building basically)

DNS needs to goto your windows AD server then on to external provider, again 1.1.1.1 ot 8.8.8.8, as Ive founf wierd issues with ISPs DNS servers over the years.

Any switch port activity you can correlate to the events… eg Sarah in production moving a cable as its there was for the next task.

4 Spice ups
(maxsec) 6

Also why so many switches?

2 Spice ups
7

can you post a screenshot of the topology - wihtout client devices.
as an example here is my home:

image
image504×483 13.1 KB

Use the options to de-select clients:
image
image239×333 7.7 KB
3 Spice ups
8

Start with some basic checks
Background info - what network/IP range do the Unifi devices use? Is it the same as your clients/PCs? Is DHCP reliable in this network? Does it use the native vlan 1?

The ideal configuration is that your native/default network (as defined on the UDM) is for the Unifi devices ONLY (or possibly other network devices also). the UDM should be the dhcp server - lease time at least 1 day. This is the defualt untagged vlan so any device connecting can discover the controller on the UDM.

Check a device to see what inform URL it is using - SSH to a switch (the username and password can be found in the network app, settings, system, advanced). type “info” - is the inform URL using a dns entry or ip? by default it will be the IP of the udm.

Devices should not loose connection to the UDM controller unless there is a genuine network outage. Even when they loose connection they should not need re-adopting, just reconnect when they can.

If you do not require wireless Mesh (a Wifi AP that does not have a wired connection to the network) disable this function: network app, settings, system, advanced: Wireless Connectivity- untick Wireless Meshing and apply. Mesh can cause loops.

If you can seen no reason why the device stops looses its connection:
get a ping monitor app and ping all switches/APs to log connectivity - will be useful to work out which stop. Do this from a PC connected to main switch.

Does your network have a ‘core/central’ switch - one connected to the UDM that all others connect to?
If any switches are daisy chained (one connected to another) then faults will affect any other switches downstream of it.

2 Spice ups
9

We lose internet and LAN when this happens. We are not getting STP alerts and nothing indicates port bouncing.

1 Spice up
10

But who implemented this network ?

Coz now everyone is going to “guess” or make recommendations only on what you are saying ?

For example if you are daisy-chaining and have busy servers or performing backup etc and thus causing some network congestion leading to APs and switches “time-out” or using DHCP and getting IP conflicts etc (between users and appliances) ?

Then are there any error logs on the controller and/or the appliances when they come back online ?

1 Spice up
11

We use 10.0.0.x network on a subnet of 255.255.254.0. All handled by Windows Server DHCP. The DHCP server is currently on a Hyper-V VM handled by what we call the Server Room Switch which is upstairs and goes down to a switch called SwitchRoom 02. It does use the native VLAN1.

So about the lease…we recently moved from 1 day least to 10 day lease thinking it was causing these problems because we had them a lot more often. Now, they are coming back…

I will try the SSH option when I get to office this morning.

I have 4 mesh devices and in the grand scheme of things, I don’t need them. We did mesh across our campus from a shop to a new shop. I placed an outdoor Mesh device at our fueling tank to hop over to our new shop as it was being constructed. We really don’t need that anymore except to have Wi-Fi at that location if needed. The 2 of these can perhaps be turned off and one replaced with an AP. So I can turn those devices off when I get to the office here in a bit.

Let me get a picture / verify the UDM direct connections here in a bit. I do believe “SwitchRoom01” is connected to UDM and then the rest are daisy chained.

2 Spice ups
12

I implemented Ubiquiti years ago but my new IT team member set up the UDM. And he utilized Ubiquiti Support and forums. But I will say, since going to the UDM, we’ve had these issues.

I may kind of go back to the basics today before people get to the office to figure this out. I don’t need this hiccup during normal business hours.

We do kick off hybrid backups (cloud/on prem) around 5PM for a select few machines but most of those kick off at 10PM.

1 Spice up
13

Quick Update - I am about to head into office. I have disabled the Mesh Devices from home.

Also, someone asked why so many switches. We have about 15 users who need multiple LAN ports in their office (LAN / Printer / IP Phone) so about 15 of these users have desktop switches. We have two main switches in the same area as our ISP modems, two switches in the server room (one for backend redundancy for Scale), Two Switches in trim shop. Two in our Steel Shop. One in our retail store and one in our new shop. Most of these shop switches are for IP cameras and heavy machines that need LAN.

So we have two ISP’s, load balanced and they come from these modems…

image
image1709×1282 202 KB

Our Dream Machines has connection to both ISP’s and then a fiber connection to our Metal Store (retail).

The only connection to the UDM goes to SwitchRoom02 switch. From there, we are daisy chained to SwitchRoom01 and then the cable from the Server Room upstairs goes to SwitchRoom01.

image
image1709×1282 298 KB

Side note…we have two new USW Max Pro’s cabled and ready to go live but they aren’t live yet but wanted to point that out since someone may ask in the picture above.

image
image1709×1282 264 KB
2 Spice ups
14

The thing I know coz I assisted my IT manager & his staff with the new UDM just in Oct and there are so many new features and constraints (coz we had to controllers in 6 sites) but we updated the firmware and almost all hell broke lose as some APs were using old firmware and now can only use 1 controller for all 6 sites.

Then I also mentioned about the switches and you need to know which of the 2 methods you are connecting the switches and also where the placement of the servers and APs etc.
Like one common method is using core switch & distribution switches

  1. 1 or 2 core switches “stacked”
  • these should be your most powerful switches
  • 2 LAN cables in LCAP (or teamed) going to the distribution switches, linked to “ONT” or Internet, linked to firewall, linked to router
  1. Distribution Switches : Server
  • These connect to servers
  • Some may have teamed (server with 2 NICs, one cable goes to 1 switch thus 2 switches are needed)
  • Sometimes servers have the most load (eg backups, reporting, data warehouse etc)
  1. Distribution Switches : Wireless APs
  • This is so that any Network or heavy data congestion between users LAN and servers do not affect the Wireless AP
  1. Distribution Switches : Users
1 Spice up
15

I recommend connecting ALL of the ubiquity gear including your cameras to the default LAN 192.168.1.0 which should manage the DHCP for that network. Inter switch ports get set to the default network, switch to APs and Cameras get set to Default.

Wire all your APs and turn off meshing.

Consider having multiple VLANs for different traffic like VOIP and printers. I let the UDMP manage DHCP for VOIP, but set static IPs on the printers.

Personally I like having DHCP handled for the various networks handled by the UDMP except for the servers network and printers - because they should be static anyway.

This will keep the network gear on it’s own managed network and the connections stable and keep the switches from dropping and re-adopting.

16

How is your DHCP scope vs total IPs issued, noting that every one of those switches has to have its own IP and have you confirmed overlaps (or lack thereof) in your DHCP versus static assignment scopes?

17

I’m going to repeat/reiterate turn off meshing on all aps. If you have a flaky connection to one of the APs it can cause flapping macs.

1 Spice up
18

I’m going to second everyone who says turn off meshing; that caused me headaches in my network initially.

I will also second the advice to set a static DNS entry for your UDM, and if you have unifi network gear on any subnet than 192.168.1.0/24, you should consider setting up DHCP option 43. Refer to their documentation for how to set that.

1 Spice up
19

Thanks…turning off our 4 meshed devices was the first thing I did today.

20

Are you in a school ??
Only 15 users ? They have their own switch ?

Daisy chain is a very bad thing as if one switch gets busy, it may clog up the while network.