Hey guys,
Got a weird situation going on here, hoping y’all might be able to help. Previous sysadmin left on some very unpleasant terms (like I might get fired for talking to him kind of terms) so asking him is right out sadly. And of course nothing is documented. Anyone got Bingo yet?
Anyways the weird situation in question is I seem to have a number of IP addresses in our main subnet that are being routed by something that no one seems to know exists. Obvious switches aren’t doing it, obvious firewalls aren’t doing it, obvious routers aren’t doing it (yes there are a few, this place is bandaided to the extremes…). We’re trying to cut back on the routing devices to one central one to clear up some of these issues, but we’re getting stymied by this phantom router. We pulled an old Barracuda Spam Filter out and remapped everything we knew about to point directly to the mail server and mail stopped dead. Added the IP of the old Spam Filter as an additional one of the mail server, and mail started flowing again kind of stymied.
At this point I’ve tracked every cable coming from our core switches and can pretty empirically state that the routing devices attached to it are accounted for, and none of them make mention of special rules for things like that spam filter or another block of IPs that do weird things. Moving a device into the IP block and running tracert just gives me back the coreswitch as the first hop, a bunch of stars for the second hop and then the location I was tracing to for the third. Moving the same device out of the block has the same effect except instead of stars for the second it comes back as one of our known routers.
So I guess the question is, how would you go about finding this thing?
7 Spice ups
Is the core switch a layer 3 switch? In other words is it the default gateway for the IP addresses that seem to be routed oddly? Seems to be the case given that it returns an answer with a traceroute. If that is the case can you post a sanitized config of the core switch?
1 Spice up
zuphzuph
(zuphzuph)
3
I’d start digging in host files personally. ¯_(ツ)_/¯ Is there a possible honey pot on the network?
1 Spice up
Yeah the core switch is layer 3 and the default gateway for most things at the moment, including the odd IPs. Attached a cleaned up switch config.
As for the hosts files, I’m not sure that’s of much help. The same machine trying to connect to the same server with just changing the IP address to be inside the block or outside it results in different routing. Literally the only change I made was switching the static IP from one value to another.
SwitchConfig.txt (3.6 KB)
Seems like you’ve inherited a nice headache,
You could set device ip to that subnet and run advanced ip scanner and see what it comes back with.
You could also check routing tables on pcs could be static routes added.
1 Spice up
You have 3 interfaces on the switch with IP addresses assigned, which one is the one with the IP block in question? Also is it the whole block assigned to that interface that has this issue? In other words if the IP block in question is serviced by VLAN C does every IP in that block experience the odd routing issue?
1 Spice up
Based on you config all you need to do is locate there 3 IPs’
ip route xxx.xxx.xxx.xxx 255.255.255.0 192.168.AA.BB
ip route 192.168.DD.## 255.255.255.0 192.168.AA.CC
ip route 192.168.EE.## 255.255.255.0 192.168.AA.DD
BB, CC and DD
Ping and resolve them to their mac addresses.
Locate where they are connected to from switches mac address tables.
Also check your firewall for any inside routed network to IPs from any of local IP ranges.
Most difficult part is usualy if you do not have credentials on those devices to access them
Sometimes it is easy, like with Cisco. Sometimes your hands are tied. Still it shouldn’t be to difficult to figure out by sniffing traffic what is their role.
1 Spice up
Do you perhaps know what the IP Address of this mysterious device is? if so we can grab the MAC of it and infer at least the manufacturer. Also then run Wireshark to do a packet trace / inspection and you might get additional information on it…All of this is predacated of course if you know the IP and or MAC.
Have you checked any of the drop ceilings anywhere for blinking lights? Sometimes you find strange things above the ceiling. Also; is it possible that there might be a rogue device under a desk somewhere in an office or at an end users desk? How about a purposely left rogue device left by this sysadmin? Is it possible that he left something behind that might be causing this? LOOK EVERYWHERE!
How about VoIP Phones, some of these have switches in them that have the ability to do Layer 2 VLAN stuff…Look in the warehouse environments…I once found a wireless camera that was above an indoor garage door that was wreaking havoc with wifi in that area. It was left there and plugged in (painted white to match the walls I might add) by the previous owner of the building…Walk the area…Trust me that might help more than anything.
Thanks,
CK
1 Spice up
As far as I’ve found so far, the issue is occurring with 6 IPs in VLAN A. 5 of which are sequential (.33-.38) and the 6th is off on it’s own at another number quite a ways off those first 5.
192.168.AA.EE is the one that’s the default gateway for most of the building, including the 6 IPs in question.
As for the three other IPs:
192.168.AA.BB - Is a VPN endpoint for an external service we have. It’s only routing inbound traffic from the external IPs (xxx.xxx.xxx.xxx)
192.168.AA.CC - Is a old PBX system. The DD subnet is for the physical blades in the PBX. I’m not entirely sure why it’s routing traffic from that subnet back to the PBX’s interface in the AA subnet. This is probably my least investigated section, but I can’t imagine a PBX routing mail flow.
192.168.AA.DD - Is indeed a router, I’ve been through it’s entire config and it doesn’t make mention anywhere of the IPs I’m having issues with.
And just to keep it up:
192.168.AA.AA is the firewall device. It’s brand new (like 2 months old - younger than my job here), replacing an older much more terrible one. It’s also not responsible here because the issues predate it arriving on site.
I’ve pretty much got access to everything except the option to pull cables randomly and see who screams.
That’s a negative ghost rider, no idea on the IP, it doesn’t identify itself, just dead air on that hop and then reply from the next.
The drop ceiling might not be the worst idea, I’ve just spent the last 3 days looking for a few APs that no one knew about to finally take down the last of our old WiFi network so we could replace it with a much nicer, centrally managed one. Nothing like roaming around the building with a signal analyzer to try and find out where your infrastructure is.
As to the users, while I have had to smack a few for bringing in their home router/aps (because said previous WiFi network was terrible) I’m pretty sure none of them, outside my IT coworkers, could figure out how to route 6 (ish, maybe more) IPs specifically to their device. Been working with them for over 4 years now (not always as IT though) so I’ve got a pretty good handle on their IT competency. No VoIP in this office either, it’s on the books for next year, or so I hear.
I could buy the old sysadmin leaving a device somewhere, part of the reason he got canned was he mounted an Air Fiber system to use the office internet connection to get free internet at his house, and then told his neighbors to do the same… I doubt however it was left to purposely mess with things. Other than the mail routing thing, the other sequential IPs were all unused until recently, which is why the extent of the issue is just coming to light now. Also since we found the mail routing issue we did a complete audit and cable trace of all the infrastructure in the server room. Figured we’d find the phantom router while we were doing that, but obviously, that didn’t pan out as we’re still hunting.
Ok, main question how exactly you detect those ghost IP addresses? Where do you see them.
You could use port monitor session and with wireshark go one by one to track them down.
Could they be used for your remote (dian-in) clients, like for VPN ?
If it’s in the same IP block and VLAN, your next hop would be the end device, as it’s already on the same network and wouldn’t need any routing done. Are you sure you’re putting it in the same block and VLAN? Doesn’t sound like it.
Also, do a sh ip route on the L3 switch to get an idea of where it’s trying to route to reach the IP addresses in question.
