POE cameras going offline everyday at the same time

spiceuser-lxej · 2025-02-07T13:01:11.952Z

Hello, I’m new in the community!

I have a very strange (to me) problem, and the explanation will be a bit long. A few months ago, I installed a POE switch to power 3 outdoor cameras from different brands; I use Frigate (an NVR software) to access the recordings. For several months, everything worked normally, but for the past few days, the cameras regularly go offline at exactly 12:00 PM or 2:00 PM (both from Frigate and from their native apps), while the Wi-Fi cameras continue to work.

This is my configuration.

network960×720 24.9 KB

The cameras have static IPs and belong to the same subnet as the other devices (192.168.0.*). Both switches are unmanaged. The poe switch has a budget power of 140W with 30W max for each port, and according to datasheets each camera requires less than 10W.

The issue only affects the POE cameras connected to switch B, while all the devices connected to switch A or via Wi-Fi work normally and are accessible from the outside.

I disabled Frigate thinking the issue might be there, but that didn’t solve it.

I set up a script that runs a traceroute from my PC (behind switch A) to one of the cameras (on switch B), and until just before 12:00 PM, I get responses in a few ms, but right after that, there’s no reply.

Now, the strange part: if I turn off and on switch B or the router, nothing changes, but if I restart switch A, the cameras come back online immediately! So I don’t understand where the issue is. I thought it could be a hardware failure, but the fact that it happens exactly at 12:00 PM suggests a software cause.

What can I do to debug the situation?

notmauricemoss · 2025-02-07T15:54:47.669Z

It’s strange that rebooting Switch-A will resolve the issue. Check the IPs…Is there a conflict somewhere? Maybe an ARP issue?

DHCP or lease expiry?

Some sort of security policy on the router (or switch as well maybe) filtering the MACs after a certain time?

Have you checked the PoE status on those ports during the outages?

Are there some schedules set in Frigate that you’re unaware of?

What else is happening on your network at those times? Do you have anything scheduled?

Edit: I just realized that you mentioned they were dumb switches, so you wouldn’t be able to check for security policies and what not.

spiceuser-lxej · 2025-02-07T16:14:51.335Z

The led on the cameras are on, meaning they are powered up.
The lights on the poe switch are on but not blinking, meaning that there’s no data traffic.

I’m not aware of any change in my network, or of any running task at that time.

At this point, I’m suspecting some sort of ddns attack on one of my cameras (that are exposed on the internet since I access them with the app, too). Unfortunately I’m not at home during the day so I cannot check the router logs

ajason · 2025-02-07T17:45:17.919Z

spiceuser-lxej:

Hello, I’m new in the community!

Welcome to Spiceworks!

spiceuser-lxej:

I set up a script that runs a traceroute from my PC (behind switch A) to one of the cameras (on switch B), and until just before 12:00 PM, I get responses in a few ms, but right after that, there’s no reply.

Now, the strange part: if I turn off and on switch B or the router, nothing changes, but if I restart switch A, the cameras come back online immediately! So I don’t understand where the issue is. I thought it could be a hardware failure, but the fact that it happens exactly at 12:00 PM suggests a software cause.

From what I read it sounds like the issue may be related to switch A. I would try your traceroute script from a PC connected to Switch B to the one of the cameras and see if that maintains the connection. This may help narrow down the search for the problem. @notmauricemoss’s suggestions should get you closer still…

robert-zed · 2025-02-07T17:57:44.020Z

One suggestion is to disconnect the switch from the router and have the NAS and PC connect to switch B with no internet access and see what happens.

somedude2 · 2025-02-07T20:12:12.903Z

NotMauriceMoss:

DHCP or lease expiry?

Another vote for that, due to the nice exact time interval. Resetting switch A may be causing a DHCP refresh because some machine on the other side asks for a new lease, and the router now notices something is jambed up and refreshes the tables …

A machine with an undeclared static IP behind switch A will cause havoc…

Also, the wifi ones get different DHCP because, well, WiFi is it’s own animal

(wire shark at one of the camera points would be ideal, but hard to do in practice, probably need to slip a connection into the router and advertise one of the camera IP’s to get a peek at traffic…this is the irritating thing about switches)

spiceuser-lxej · 2025-02-07T22:03:54.560Z

Tonight, when I got back, the cameras were still offline but turned on (the status LEDs were lit, and the indicators on the switch were also on, though they weren’t blinking, meaning there was no data traffic).
However, as soon as I connected my notebook to switch B, the cameras immediately came back online!

It seems that the switch B was in a sort of “power saving mode”, and plugging the notebook in restored it to normal operativity… but as I said it’s an unmanaged switch, so I can’t understand this behaviour.

So I think the next step is isolate the switch A and see if the issue persists…

somedude2 · 2025-02-07T22:17:24.372Z

If the LED’s are on, it’s not in any normal sleep mode even if it has one, poe sleep mode is about powering (connected) devices down.

Plugging your notebook in also caused a DHCP lease request, and also a couple broadcast packets, broadcasts teach routes to switches, …same diag as before, doesn’t sound like looping, you don’t really have any loops, and it would lock up and get stuck anyway.

What POE switch? I know some old linksys switches were buggy and would do weird things and just stop talking…

Also…if your WiFi public? Could there be outside access?

Not sure I would have plugged a notebook into a POE switch, but OK, if you leave it plugged in, does the network connection die at 12:00? That would be simplest step 2

spiceuser-lxej · 2025-02-07T23:17:55.061Z

Both switch A (Tenda) and switch B (Intellinet) are brand new… I did a whole house renovation last year.
I don’t have a public ip but the cameras are exposed on the internet and reachable via app. Are you thinking about something malicious or similar?
Before the renovation I blocked them via firewall, so they were reachable only via nvr, but when setting up the new network I disabled the block

spiceuser-lxej · 2025-02-08T15:05:32.126Z

So, I left the notebook plugged into switch B and the cameras are still perfectly working (it’s 4pm now!). I don’t know if it’s a good or a bad news.
By the way, I tried disconnecting all cameras and one of them appears still connected via wifi, even if it’s not listed in router connected devices list. Maybe it has some setting stored from its initial setup. Can this be the issue?

spiceuser-lxej · 2025-02-09T06:03:19.652Z

The cameras went offline again at 4am, while the notebook was still plugged!
I did some tests: the notebook could still go online and reach devices behind switch A, but no vice versa (so, it couldn’t be reached from switch A).
Now, unplugging/replugging didn’t recall cameras online so I had to power cycle switch A again

adrian_ych · 2025-02-09T12:07:20.169Z

From what I gathered…

This is a home network and using consumer range of products ?
Only Cams on switch-A goes “data” off (not for other devices) but when moved to switch-B does not go off ?
No detailed logs on switch-A ?
No logs on cams ?

Never heard of the switch brands…
Switch-A gives POE power but no data until switch-A reboots…
Sounds like a faulty switch or having faulty ports …

BTW, you DO have a public IP thats issued by the ISP, thats how the cam apps can “find” the cam…so can others (how easily depending on the cam company)…but i not going down that hole as you may need togoogle reviews of that cam & app.

mikegilkim · 2025-02-09T12:54:54.312Z

Very strange indeed.
Seems like you’ve narrowed down the issue that something in Switch A is causing the issue during the time indicated.

Probably an IP Conflict? ping the IP address when the Cameras are offline.
If not, try having the 3 cameras in DHCP and mac-bind it in your router.

hope this might be of help.

spiceuser-lxej · 2025-02-09T13:35:54.584Z

Here I am again after running more tests.

First of all, I need to correct myself: the notebook actually went offline but immediately reconnected via Wi-Fi, which made me think it was still online. So yes, the disconnection issue affects all devices on switch B.

What I did: I connected switch B directly to switch A and blocked all external connections by closing all ports on the firewall.
Despite this, the issue occurred again at exactly 7:00! This time, simply turning switch A off and on wasn’t enough—I had to power cycle everything (router, switch A, and switch B)…

After that, to rule out IP address conflicts, I reset the camera that was also connecting via Wi-Fi, so now it is only connected via PoE.

At this point, I also suspect a hardware issue, but where? From what I understand, a single camera could be triggering a protection mechanism on the switch, causing all ports to be disabled…
So my next attempt will be to disconnect them all and reconnect them one at a time… but the fact that the issue isn’t easily reproducible makes troubleshooting really difficult.

spiceuser-lxej · 2025-02-09T13:44:04.042Z

The cameras are not pingable when offline, and they are not more listed within the connected devices on router.
Another detail: the uplink port on switch B is still blinking, even if others aren’t

I checked /var/log/system on notebook (which runs Ubuntu) but as far as I understand, it’s simply reporting that the ethernet connection has been dropped

Feb  9 03:54:51 Mercurius NetworkManager[1070]: <info>  [1739069690.9729] dhcp4 (wlo1):   gateway 192.168.0.254
Feb  9 03:54:51 Mercurius NetworkManager[1070]: <info>  [1739069690.9729] dhcp4 (wlo1):   lease time 86400
Feb  9 03:54:51 Mercurius NetworkManager[1070]: <info>  [1739069690.9729] dhcp4 (wlo1):   hostname 'Mercurius'
Feb  9 03:54:51 Mercurius NetworkManager[1070]: <info>  [1739069690.9729] dhcp4 (wlo1):   nameserver '8.8.8.8'
Feb  9 03:54:51 Mercurius NetworkManager[1070]: <info>  [1739069690.9729] dhcp4 (wlo1):   nameserver '8.8.4.4'
Feb  9 03:54:51 Mercurius NetworkManager[1070]: <info>  [1739069690.9730] dhcp4 (wlo1):   domain name 'home-life.hub'
Feb  9 03:54:51 Mercurius NetworkManager[1070]: <info>  [1739069690.9730] dhcp4 (wlo1): state changed bound -> bound
Feb  9 03:54:51 Mercurius systemd[1]: Starting Network Manager Script Dispatcher Service...
Feb  9 03:54:51 Mercurius dhclient[1607]: bound to 192.168.0.155 -- renewal in 29525 seconds.
Feb  9 03:54:51 Mercurius dbus-daemon[999]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Feb  9 03:54:51 Mercurius systemd[1]: Started Network Manager Script Dispatcher Service.
Feb  9 03:54:51 Mercurius nm-dispatcher: req:1 'dhcp4-change' [wlo1]: new request (1 scripts)
Feb  9 03:54:51 Mercurius nm-dispatcher: req:1 'dhcp4-change' [wlo1]: start running ordered scripts...
Feb  9 04:02:17 Mercurius systemd[1]: Started Run anacron jobs.
Feb  9 04:02:18 Mercurius anacron[8361]: Anacron 2.3 started on 2025-02-09
Feb  9 04:02:18 Mercurius anacron[8361]: Normal exit (0 jobs run)
Feb  9 04:08:28 Mercurius kernel: [112551.374827] r8169 0000:03:00.0 enp3s0: Link is Down

and then around 7.00

Feb  9 06:57:49 Mercurius systemd[1]: Starting Network Manager Script Dispatcher Service...
Feb  9 06:57:49 Mercurius nm-applet[3234]: gtk_widget_destroy: assertion 'GTK_IS_WIDGET (widget)' failed
Feb  9 06:57:49 Mercurius nm-applet[3234]: gtk_widget_destroy: assertion 'GTK_IS_WIDGET (widget)' failed
Feb  9 06:57:49 Mercurius nm-applet[3234]: Can't set a parent on widget which has a parent
Feb  9 06:57:49 Mercurius dbus-daemon[999]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Feb  9 06:57:49 Mercurius systemd[1]: Started Network Manager Script Dispatcher Service.
Feb  9 06:57:49 Mercurius nm-dispatcher: req:1 'down' [enp3s0]: new request (1 scripts)
Feb  9 06:57:49 Mercurius nm-dispatcher: req:1 'down' [enp3s0]: start running ordered scripts...

jarmbrister · 2025-02-10T14:16:53.689Z

somedude2:

Not sure I would have plugged a notebook into a POE switch…

In case someone runs across this and it causes some FUD, usually plugging in whatever you like to a PoE switch will not cause a problem. Modern equipment generally employs Active PoE and will not send power down the line until the need has been actively checked/negotiated with the connected device. The only time there theoretically could be an issue is if Passive PoE is in the mix. Passive PoE sends power all the time, regardless of what’s connected.

somedude2 · 2025-02-10T16:14:40.802Z

spiceuser-lxej:

dhcp4 (wlo1): lease time 86400

So 24 hour lease -  that is a bit short, but it fits the once per day breakage

dhclient[1607]: bound to 192.168.0.155 -- renewal in 29525 seconds.

renews in 8.2 hours…which is weird, usually that is 1/2 the lease, but ok if it’s
local…it it misses 2 refreshes it fails .4 hours after the lease tho

spiceuser-lxej:

reset the camera that was also connecting via Wi-Fi

Good.that could have been causing loops, not just in routing, but if the camera is a bit stupid, it may have been using the wrong DHCP lease data when switching…but…why is it switching to wifi…

spiceuser-lxej:

power cycle switch A again

Still doesn’t totally blame the switch, because that would also break routing loops, and dropping the interface on the router probably makes it rescan DHCP just in case (some routers do that , some don’t)

make SURE you don’t have a camera somewhere with a static IP set, it will mess up dhcp …

spiceuser-lxej · 2025-02-11T11:55:24.798Z

Well, I tested again with just one camera (which supports only poe) plugged and still went offline after a few hours; I tried also completely disconnecting switch A and still disconnects.
I changed also lan cables: no luck.
All cameras have fixed ip address, outside dhcp range; moreover, as I wrote, the whole setup worked perfectly for several months, then the issue came out of the blue.
So, I gave up and applied for a return, hoping the problem isn’t elsewhere; could a faulty poe camera permanently damage the switch?

ajason · 2025-02-11T13:02:23.705Z

Something else that you could try if you have or could get one is to use a POE injector and connect a camera to the other switch to see if the issue persists.

spiceuser-lxej:

could a faulty poe camera permanently damage the switch?

I would think this would be unlikely.

What is the model of Switch B? What are the camera brands and models? Does it have the Self Healing Network function?

I was looking at the Intellinet switches and wonder if that technology could cause issues:

Self-Healing Network with increased reliability and stability - Intellinet Powered Device Monitor (PDM)

This Power over Ethernet switch has a self-healing PoE function called “Powered Device Monitoring.” This PoE watchdog does a PD alive check on all connected PDs and automatically resets devices that aren’t responding, like PoE security cameras, by cutting power and then turning it back on. The PDM function is especially useful in industrial or surveillance settings where devices connected to the network are important to how it works. As an added bonus, the PD Alive function helps cut down on unnecessary technical support calls and expensive field trips by field-support engineers. This is because the functional reboot can often bring the unresponsive device back online.

Also, did you see this about the Power over Ethernet 802.3at?
For devices that are not 802.3at/af compliant (legacy wireless access points or network cameras), we suggest the use of an Intellinet Network Solutions PoE/PoE+ Splitter.

LuisC · 2025-02-11T21:37:52.035Z

spiceuser-lxej:

; could a faulty poe camera permanently damage the switch?

Never seen it happen in the 13 years I worked on the security integrations side, but I have seen shorts in the wire throw the switch off.