1

All,

I apologize if this has been covered elsewhere, I’m at my wit’s end about the way to resolve this problem. I can post configs if they will help.

My setup: an HP 5406zl at the core, I have a 2910, and 3 2610’s directly connected to the 5406 via fiber. They only factor in here to show that some portion of my config is working. I do not have any issues pinging 'across vlans from any of these ‘remote’ switches.

I have several vlans setup. As a sample of my issue I have a Client VLAN, and a Server VLAN. The 5406 serves as the default gateway for all VLANs in question and for most of the network.

On the 5406, I have 2 ports as a test, Port A is only in the Client vLAN (untagged), Port B is only in the Server vLAN (untagged). The 5406 is setup with IP routing.

If I have a client computer directly connected to Port A, I’m able to ping the default gateway, and I’m able to ping computers in the Server vLAN. I’m also able to get a DHCP address (served from a server residing in Server vlan, 5406, has DHCP helper address setup). This seems to work as I would expect.

Now add to this the ISP’s “layer 2 P2P link”. They provide a device at either side that I do not have access to and I’m not required to route packets to get through. On this P2P link if I plug a computer into either end I have no issues pinging their statically assigned IP addresses. If I plug the P2P link into my 5406 in the “Client VLAN” port ‘A’, I have no issues pinging other devices in the same vlan, however I’m unable to ping devices in the “Server VLAN”. Either from the computer on the far side of the P2P or from the servers. If I attempt to get a DHCP address across the link, I’m unable to.

From the client I’m when on the remote side of the P2P link, I’m unable to ping the default gateway, while I’m able to ping other devices on the same vlan. If I move that same client to the “non-remote” side of the link, again I have no issues pinging anything (server, gateway or otherwise). This is using the exact same port on the core switch and cable that plugs into the ISP’s equipment.

I’m asking for any other eyes that see something I’m missing or suggestions on ways to work around this. The end goal is I need to be able to pass vlan’d traffic across this link. That’s what we were sold, but the ISP says that “both endpoints are up and the tunnel’s up” so it’s good.

I have done several wireshark traces that only seem to confuse me more.

For example in Wireshark, (attempting a ping from the server vlan to the “remote client”) I can see the ping request leave my “server” and I can see the client computer receive the ping request, but there is no ICMP response. Part of me thinks that Windows 7 (the client) location awareness is causing the firewall to not respond, but I saw similar behavior on my Mac computer when I put it in a similar position and again attempts to pull a DHCP address across the link fail.

I appreciate any thoughts and input.

3 Spice ups
2

A quick thought is that the ISP L2 connection is not passing the vlan tags on your packets. I have an L2 MEPL connection here and I have to put devices at each end and run a vpn to get my vlans across it.

3

We have something which sounds very similar here which we use to connect to a nearby remote office.

We are able to treat this effectively as though it is a network cable so we have this set up as a trunk with the VLANs we need tagged at each end to be able to transit through it - in our case our VOIP and Client VLANs.

Not sure if this will help you?

1 Spice up
4

You should be tagging all vlans to the uplink device they are providing,

5

On this: Ask if they’ve enabled ‘QinQ’ on that point-to-point connection:

Otherwise your VLAN tags are just being stripped off and/or ignored/dropped.

6

For testing and proving out, I’m only using one vlan. The client vlan, then depending on the switch to route to alternate vlans.

7

Seems ridiculous that the should sell a “layer 2” link and then we have to VPN across it. I can understand the ‘privacy’ aspect, but if that’s not an issue, they should be providing what they’re marketing.

I’ll have to ask about the QinQ

8

To add further reason to pull my hair out. The link is now allowing me to ping across vlans (i.e. my server can ping the client) and the client can ping the server). It seems to take a while (minutes?) if I disconnect the remote test client for it to re-establish connection.

Still if I try to get a DHCP address from the remote site, I do not see any entries in my monitor port’s wireshark to show that the request made it to the switch.

–update after some additional testing–

I installed a DHCP server on a laptop and connected directly to it with a client. - I was able to get an IP address.

I then tried to get a dhcp address across the ISP’s circuit. The first time I didn’t shut down the computer when transitioning and I think it was able to re-approve it’s IP address. I then rebooted and was not able to get an IP address across the ISP’s circuit.

9

Could you send or post the configurations for the switches at either end of the link and I’ll take a look see if I can see anything?

When I set up the one here I found that I had to set up the IP helper address on the switch at the remote end of the link as well as at the core.

10

@ apdibbo I can do the switch config if it comes to that, after a duplicate test this morning, I’m convinced that there’s an issue with their circuit. Please let me know if you see any flaws in our tests and train of thought.

Previously (not repeated today) I can statically assign an IP on either side and ping between the two devices without any apparent issues.

Today I repeated my DHCP test. I have a laptop that I installed TFTPD32 on to operate as a DHCP server.

Test 1 - Directly connecting the two laptops, I’m able to get a DHCP address without any issues.

Setup for Test 2 - Connect DHCP server directly to ISP’s device, walk to other side of connection, plug in same laptop on far side into ISP’s remote endpoint.

Test 2 - Try to obtain DHCP address across ISP’s circuit, without success.

Additional information. Ran wireshark on both devices for the duration of the tests. On my DHCP server after I disconnect my DHCP client, wireshark sees NO packets from any other mac addresses. To add to this, there was another laptop plugged in on the remote side while I was walking to the remote site, so it should have been seeing packets from the 2nd client.

My test DHCP client once it’s back online appears to be receiving all the broadcast traffic from the DHCP server on the remote site, but it seems that the DHCP server is seeing none of the broadcast traffic from my DHCP client.

Is it possible their connection is setup to only allow broadcast traffic in one direction?

11

That’s a possibility, have you tried reversing your test?

I would speak to your ISP to ascertain if the behaviour that you are seeing is expected and also what you actually have in terms of the capabilities of the circuit i.e. is this a dedicated link for you guys or is it something else?

I would be surprised if they are restricting broadcast traffic as these kind of things are usually sold as “LAN Extensions”

12

Yes I just tried reversing my test. The DHCP then sees the requests and sends an offer, but the client does not see the DHCP offer.

I have spoken to them again. The issue has been escalated to engineering as of yesterday, but status updates aren’t really forthcoming thus far. I gave them the information I’ve acquired so far.

I agree, especially since it’s only in one direction.

13

Very weird, I’m interested to hear what they say

14

Will update if (hopefully when) they give me a resolution.

15

After Engineering initially said that they saw no issues yesterday, I provided them with the wireshark traces that clearly show a lack of traffic in one direction. This morning’s update is that they are investigating a possible issue with the RIP traffic?

It’s like pulling teeth, but at least they seem to acknowledge the issue now.

16

Good to hear, that is often the hardest part :slight_smile:

17

Today they came back and said they had built an exact replica and were not able to reproduce the issue I described. They are sending a technician out to look at it tomorrow. Hopefully we can get somewhere with this.

18

The tech’s came out onsite today with the equipment they normally use to test Fiber point to point connections. They were unable to successfully test loopbacks from both directions. They are confident that the configs are correct, so the only other option is one of the modems being bad. I’m not using them at the moment so they took them back to the office to test and determine the best way to resolve it.

It is such a relief to have them finally admit that there is indeed a problem.

19

Awesome :slight_smile:

20

Ok, so the tech’s were able to get it to work at their office after re-applying the config to the devices (cable modems). They brought them back out onsite and we started with our first level test, just trying to get DHCP across. It worked! So I got a little over zealous and tried to move over to the ‘desired’ setup. It worked but then it seemed like I could get a DHCP address and then i’d loose it pretty quickly. We chased the idea that it was our equipment for a while then reverted back to the original base test (just a laptop on either end with a dhcp server). Unsuccessful. Now all of us were scratching our heads.

The tech’s got back on the phone and after a while the remote tech’s believe there’s an issue with the way the “central hub” devices are configured. the two endpoints connect to two different central hub devices and apparently they’re fighting each other for control of one of the endpoints.

So it still doesn’t work, but now they’ve gone back to investigating issues on the the “central hub”.