1

We have something extremely strange going on with a Fortigate 30e. The client changed ISP and all of a sudden they are no longer able to browse to some websites such as www.yahoo.com or mail.yahoo.com some printer websites. We have narrowed it down to the firewall because we can plug directly into the cable modem and no longer any issues but once we put the firewall back into the picture the issue returns. I’ve contacted Fortigate support and was told because we can ping the websites from the firewall cli we should be able to browse to them, end of supports interest.

We originally thought was something to do with updating the firmware from 5.4.5 all the way up to 6.2.2. We were sure something had become corrupted so I reimaged it with a clean copy of 6.2.2 and the issue still remains. Tracert doens’t show any high latency in the hops. Can ping the websites all day long just cannot browse to them behind the firewall. Any suggestions appreciated.

3 Spice ups
2

Have you reviewed the firewall logs ? There maybe some false positives or even some application firewall settings (like blocking search engines which may include google or yahoo etc).

Then more importantly, you should contact fortigate vedor and fortigate support as there might be some know issues with either their hardware or firmware (eg need to update to 5.9 then to 6.0 then to 6.xx etc) ?

3

Have the rules in the Fortigate been updated to reflect the new IP ?

4

With Fortigate these type things usually happen with web filters confugures in policies. Under logging find web filter and check. You will need to enable security logging on the policies themselves as well and check logging settings to make sure they’re turned on in general terms.

In the web ui go to the fortiguard settings (where you can review all your licenses) and make sure web filtering is working. It relies primarily on DNS which with some ISPs could be problematic.

As a test you can try to edit the policy that controls LAN to WAN traffic and turn off all the security and protection profiles. If that resolves the issue then you can re-enable them one by one to see when the issue returns and dig into the culprit.

1 Spice up
5

Have you changed the local dns forwarder to be the new isp’s or an isp neutral one like 8.8.8.8?

2 Spice ups
6

Joining Martin with his DNS suspicion - with a small twist…

Possibly some firewall rules use FQDN names, while the DNS server defined in the firewall is still that of the old provider (and not allowing queries when you switch over to the new provider).

At the same time your clients and/or DNS forwarder might be correctly configured to the new ISP and/or neutral DNS servers, so the rest ‘somehow’ works.

7

Guess I should of mentioned we tried OpenDNS, Google quad 8 and 8.8.4.4 and the ISPs DNS with all having the same issues. The only ipv4 rule there is, is to allow LAN to WAN, all,all,always,all,accept,enabled, ssl no-inspection. This firewall has never been licensed for web filtering. Security profiles for the LAN to WAN are already disabled.

8

Just to confirm: NAT is enabled on that policy, right?

On the computers that are having issues: what are their DNS servers set to? Check by running ipconfig /all Normally they are provided by a DHCP server. Is the Fortigate acting as your network’s DHCP server? If so, maybe it’s handing out wrong DNS IPs to the clients so the clients are trying to contact a DNS sever that isn’t available anymore. Is this an Active Directory network? If so the DNS setup will be different.

There are some debug commands you can run on the Fortigate to gain visibility on what is going on when websites aren’t loading. The general command on Fortigate’s CLI to run is diagnose sniffer packet any ‘filter’ 4

The filter needs to be replaced by whatever makes sense depending on what you’re looking for (can be IP and port based among other things). The ‘4’ at the end specifies what’s included in the output, in this case it’ll show source interface, destination interface, source port and destination port.

If I were troubleshooting a website not loading, I could check to make sure there was at least traffic to and from the remote IP: diagnose sniffer packet any 'host remote_ip and port 80’ 4

If that doesn’t yield much then additional debug commands are available that dig deeper into the traffic flow and will tell you why a packet was dropped (if that is what is going on to begin with). Will need more info on your environment before I can go into any useful level of detail.

You can accomplish similar things with a tool such as Wireshark on a test computer. It may be helpful to see if traffic leaving the computer is even arriving at the Fortigate and what happens when it’s sent outbound to the Internet.

1 Spice up
9

DHCP and DNS are being handled by the local server. DHCP is not enabled on the firewall. The IPs on the end user devices point back to the server IP for dns. We can ping the websites by their url and IP from what we gathered with a tracert. We then started a sniff on the firewall to test for traffic when pinging the 4 or 5 IP addresses we gathered from the tracert and each showed traffic when we would start a ping. Just a reminder if we remove the firewall from the picture and plug the network straight into the cable modem adjust network settings accordingly, the issue vanishes. So the puck stops at the Fortigate somehow. The device was completely reimaged so the only customization on the firewall is the static IP/static route and a couple trusted networks to access the firewall locally and remotely. One other thing we have wondered is if maybe the MTU value needs to be adjusted so we lowered it from the default value of 1500 to I think we tried 1492 but same issue. I even went to security profiles and disabled Fortiguard category based filtering but didn’t help.

10

One more time: have you changed the DNS setting on the firewall to the DNS of the new provider (or some other public DNS)?

To work, the firewall also needs to resolve DNS queries for various security services and for FQDN’s in firewall rules.

If the firewall is still configured for the old providers DNS server, this won’t allow resolution from a 3rd party network (most ISP’s only allow resolution from their own customers - inside their network). So when the firewall can’t resolve DNS, a lot of things won’t work as it should and you could see effects, just like these you are facing.

Check the DNS setting on the firewall one more time, to be sure, you got it correct.

11

Try to edit the LAN->WAN policy and set Log Allowed Traffic to All Sessions (instead of the default of only Security Events).

Do whatever you need to cause the problem to trigger again, then go to the Log & Report section and see which log categories exist (they are created dynamically when there’s a category to log something on) and go through them to see if you can find any hints.

If you can replicate this website loading issue with a specific website’s IP address (don’t use a large site such as Yahoo or Google because there are so many IPs it’s harder to figure out which IP to filter on), run a sniffer on the Fortigate like so:

diagnose sniffer packet any 'host ipaddress and (port 80 or port 443)' 4

This will show you all inbound and outbound traffic to the IP address. When you can’t load a web page, the idea is to see if you see any outbound traffic and make sure the IPs make sense, then there should be return traffic too. There should be two lines for each: Computer’s LAN IP to remote IP, then NAT is applied and you’ll see Fortigate Public IP to remote IP. And replies coming in will follow the same idea but in reverse.

Wireshark on the computer in question will be helpful to compare, to see if there are any packets dropped. Maybe your outbound request is dropped before it even reaches the remote server.

You can also debug the traffic flow to see exactly what the Fortigate is doing to your packets and if there are errors.

diagnose debug enable
diagnose debug flow filter saddr LAN_IP
diagnose debug flow filter daddr REMOTE_IP
diagnose debug flow trace start 1000

You can run these two commands concurrent to get a mixed output of both types in one console session. Run the diagnose debug commands first, then start the packet filter, and load the troublesome website.

If the Fortigate is blocking something, there are hopefully some clues in the debug output. They are admittedly hard to read so feel free to attach a text file of the results if you don’t mind your IP addresses exposed.

The whole point of this exercise is to determine what is actually happening when a website isn’t loading.

12

Yes tried OpenDNS, Google’s, the new ISP DNS. Nothing helped.

13

So we unfortunately had to remove the static IP from the firewall and change it to dhcp and let it pull an IP from the cable modem. Now all websites work. We spoke with the ISP explaining this circumstance but because the cable modem is customer owned that would not look into it.

14

And that is causing only some websites to not work? That’s an unusual symptom. Normally I see ISPs requiring DHCP in order to register the router’s MAC address and nothing tends to work until that happens. Nice of your ISP to not help out…