1

Hi. For the past 5 days our users have reported that our office ethernet network is slow. I’m struggling to find out what the cause is so was hoping for some pointers from everyone here.

We are an office of around 250 people, though the number of people in the office at any one time varies day to day. The majority of the users now run a Macbook Air M1 as we have been rolling them out over the past month or so. There are now a small amount of Windows machines around the place, though no issues have been reported for those.

Our internet line is 1Gb/s and we are not hitting that limit (nowhere near hitting the limit most days). All of our complaints come from Macbook users so I thought it might have been the docking station they’re using. Today I asked one of the users to connect to the ethernet using an external ethernet adapter instead of the dock to see if the issues go away - they don’t.

So, what is the issue? When users are connected to the LAN via the docking station they intermittently get disconnected. The connection will then come back after 15-20 seconds. People also complain about internet pages loading slowly or not at all. There isn’t a pattern as such and can vary between users.

I have done the following:

  • Ran a tracert from a user’s machine
  • Ran a ping to an internal and external location
  • Verified we’re within our bandwidth limits
  • Checked with our ISP that our connection is ok
  • Checked on the core and access switches for packet drops/other errors - there are none
  • Checked on the core and access switches for rx/tx levels on relevant interfaces - all good
  • Had a call with Cisco support to get a second opinion - they agree with what I’ve found
  • Ran a speed test - this is a bit more concerning as it only returned 184Mbps for download speed

I’m running a Windows machine and am sat in the office which is now empty. The only thing I notice is that on YouTube the videos seem sluggish but nothing major.

What else can I check? I’d say the issue is either with a Mac or the docking station but I want to make be definite that the network isn’t the issue. Apart from all the above, what else can I do?

63 Spice ups
2

Might want to get an inventory or scan of the network to see if there is an oddball device on the network slowing things down.

11 Spice ups
3

Any recent policy changes for your firewall?

How is the health of your DNS server(s)? (always start here actually lol)

Is it possible to try connecting a macbook directly to your ISP’s modem to test bandwidth?

12 Spice ups
4

Not sure about the random disconnects, but have you looked at DNS?

A flaky DNS will cause all kinds of issues.

Also, security software of any kind? Any updates installed recently? I know Apple was pushing updates out for some 0 day’s.

4 Spice ups
5

Try checking your firewall as the other suggested to see if there is any abnormal activity in your traffic graphs hogging the bandwidth. Also are all of your DHCP leases used up or close to it? Is your DNS pointing to external source or is it facilitated through your DC?

If your pipe is 1GB symmetric, that’s definitely an issue. It could be the modem, data link, etc. but for that much of a variance, something is off with your ISP connection. Could be worth having one of their techs check it out.

6 Spice ups
6

I disagree this could be a false flag.

Speed tests are inherently unreliable for several reasons. Especially if the test was run during the day when the line is in use as you won’t get the full throughput of the line. I have also found many speed tests are extremely unreliable with faster WAN links like 1GB. remember to get an accurate test they have to have a connection at least as fast as yours and their connection must have adequate bandwidth at the time of the test too.

What is your firewall \ Gateway Device? many devices can’t support 1GB WAN throughput, especially if you are running any security services on that device. have you verified yours can? doesn’t explain the random disconnects, but its worth checking.

3 Spice ups
7

A few questions

  1. Are all 250 on the same VLAN ?

  2. How many switches do you have and how are they connected ?

  3. If more than one switch (which 250 users would suggest or you have a giant chassis or lots of wifi users) have you actually defined a root bridge or did you just plug everything together hoping it would work seamlessly ?

  4. Have you checked the logs of the switches for evidence of SPT TCNs ?

  5. Are these actually switch users or are you just using the term Ethernet to generically encapsulate Wifi users as well ?

3 Spice ups
8

To confirm that all devices are wired not wireless? Check macbooks are not also using wireless.

When you say " disconnected" does the actual link drop - check switch port logs. If they do look at the macbook logs for info - this should not happen? check obvious things like dhcp lease time is not short. Can you leave one macbook on all day pinging the default gateway - any drops.

5 Spice ups
9

Check your DNS, have you done a wireless survey to see if there is significant congestion, if you are running fiber look at the light levels on the interfaces to see if they are low or even running hot. It could be an issue with how you are authenticating clients on your network. If that is flaky results could be unexpected.

10

If I’m reading this correctly, it’s only your Mac users that are complaining about slow connections? And the only change you’ve made is swapping Windows machines out for Macs? If that’s the case, I’d be looking at the network setting on the Macs.

Gigabit settings for Mac

Find a guinea pig to test any changes on.

2 Spice ups
11

Have you checked for network loops? There could be a dumb switch somewhere that you don’t have noted in your inventory that could be leading to a loop somewhere and slowing things down. And, as others have suggested, check DNS.

10 Spice ups
12

tl;dr Any chance some device on the network is competing with your DHCP or DNS servers?

I had an incident in my last job where we had similar problems. Timeouts connecting to file shares, web pages not loading, lots of intermittent connection issues.

Now this job was at a non-profit and we had a hardware refurbishing team that refurbed donated computers and gave them out to the community. And of course they got lots of other equipment donated including SOHO equipment - like Netgear routers.

So they had decided they needed more ports in their office and plugged in one of the Netgear routers to use it as a switch. Problem was, it was still set for DHCP and it also happened to be configured with the same subnet as our office, including our gateway.

So it turned out this router was giving out IP addresses and clients were intermittently being given duplicated IPs or connecting to the routers ‘gateway’. It was random as hell and It took me way too long to finally find this thing on the network, but there it was under their desk. They thought it was funny as hell. I didn’t.

15 Spice ups
13

Have you tried running a speed test directly connected to your ISP? If you get those same speeds from the modem/gateway device, it is likely an ISP issue. The big name ISPs Teir 1 support typically don’t actually check anything useful. If the speeds look as expected from the modem, then the issue is on your network. Start testing working back from the ISP to the firewall directly, to the router if you have one separate from the firewall, then the core directly, then the access switches until you see the speeds drop.

1 Spice up
14

I’m interested in this. I’m the only Mac user in the office (I also have an M1 MacBook Air), and I’ve had random network slowdowns as well. I assume if I were doing a lot of network-intensive tasks I might see it more often, but I’ve only noticed it a few times and it always went away after a restart.

You can check out this thread with a few tips given to a user with a similar issue. The best piece of advice seems to be to try safe mode (hold the Touch ID button until you get the startup options) and see if that gives you different results. Then check for any installed software that might be causing issues.

If you’re on Monterey, a user on this thread hints that some applications (Cisco AnyConnect in their case) can cause the OS to run out of available sockets, preventing further DNS queries etc. So if you have any network-related third-party software installed I’d start there.

4 Spice ups
15

Do you have IPS or IDS enabled on your firewall or any other type of security services running?

2 Spice ups
16

I’m thinking it is something with the Macs. Network settings need to be reviewed and check for possible updates regarding OS or firmware.

17

So they had decided they needed more ports in their office and plugged in one of the Netgear routers to use it as a switch. Problem was, it was still set for DHCP and it also happened to be configured with the same subnet as our office, including our gateway.

So it turned out this router was giving out IP addresses and clients were intermittently being given duplicated IPs or connecting to the routers ‘gateway’. It was random as hell and It took me way too long to finally find this thing on the network, but there it was under their desk. They thought it was funny as hell. I didn’t.

Agree, check something else is giving out DHCP addresses.

1 Spice up
18

Get Wireshark and sniff the network at one of the bad connection points. Realistically you would sniff a known good time and a known likely bad time and compare the results. You’re likely going to see lots of retransmissions and FIN-ACKs on the bad segment. But you will get some idea of who’s complaining about what…that can point you in the right direction.

Also, when trying to troubleshoot an unknown network problem, work the problem from the ground up and top down. The top down is the sniffing. The ground up is at the physical level. For example, are the docking stations connected using physical wired connections? If so, start not allowing one of the docking stations (at a time) from being active…and see if the problem stays away when one of the docking stations is not plugged in.

I’ve had one bad wired connection cause this sort of problem in the past. The connection appeared to work until network traffic volume got up (and the network retransmits overwhelmed the shared LAN)…and chopping off the head of that wire and re-connecting a new RJ-45 head on it fixed the problem. Bad wiring can be hidden by low network traffic volumes, but becomes too much as re-transmits really go crazy under busy network volumes.

But if it were me, sniff the network to see what’s happening at the logical level.

7 Spice ups
19

I’ve used Wireshark to find similar issues.
It’s been a while, but something was doing a lot of talking and Wireshark helped me find the device and then I checked the configuration and found the problem.

2 Spice ups
20

A lot of replies are asking about your internet connection. BUT if people are losing access to the LAN then the internet connection is irelevant.

do you have managed switches? do people have a problem at the same time of day or is it arbitrary?

you haven’t changed a cable in the switches which has caused a loop have you?

as others said, look at the MACS as i have a few issues with those here and the windows machines are fine!

Also you are not using firefox are you? i am finding that firefox can be very unreliable when browsing the web and can fail to be responsive, just something new i have noticed with it.

1 Spice up