Friday, May 30, 2014

Fix Juniper Network Connect on Mac bad route Issue


The Fix:

Run the following "Fix Commands" with root permissions: 

interfacetofix=`netstat -r -n -f inet | grep "0&" | awk '{print $NF}'`
interfacenametofix=`networksetup -listnetworkserviceorder | grep "$interfacetofix" -B1 | tail -2 | head -1 | cut -d " " -f 2-10`
networksetup -setv4off "$interfacenametofix"; networksetup -setdhcp "$interfacenametofix"

networksetup -setnetworkserviceenabled "$interfacenametofix" off; route flush; killall -9 mDNSResponder; route flush; networksetup -setnetworkserviceenabled "$interfacenametofix" on; ifconfig "$interfacetofix" down; ifconfig "$interfacetofix" up

And run these commands again... and again... See the "Typical Fix Walk-through" if you desire more.

NOTE: These commands assume that you are using DHCP to assign an IP address, if you are manually entering in IP addresses on the machine, then please use the alternate set of "Fix Commands".

NOTE: Running these commands will disable networking briefly, this is normal.

For lazy sysadmins, the "Fix commands" that check for a bad route first:

interfacetofix=`netstat -r -n -f inet | grep "0&" | awk '{print $NF}'`
if [[ $interfacetofix == "" ]] ; then echo "No bad Route"; else 
interfacenametofix=`networksetup -listnetworkserviceorder | grep "$interfacetofix" -B1 | tail -2 | head -1 | cut -d " " -f 2-10`
networksetup -setv4off "$interfacenametofix"; networksetup -setdhcp "$interfacenametofix"

networksetup -setnetworkserviceenabled "$interfacenametofix" off; route flush; killall -9 mDNSResponder; route flush; networksetup -setnetworkserviceenabled "$interfacenametofix" on; ifconfig "$interfacetofix" down; ifconfig "$interfacetofix" up
fi

Alternate set of "Fix Commands"- For manually entered IP addresses

interfacetofix=`netstat -r -f inet | grep "0&" | awk '{print $NF}'`
interfacenametofix=`networksetup -listnetworkserviceorder | grep "$interfacetofix" -B1 | tail -2 | head -1 | cut -d " " -f 2-10`
interfaceIP=`networksetup -getinfo "$interfacenametofix" | grep "^IP address:" | awk '{print $NF}'`
interfaceSubnet=`networksetup -getinfo "$interfacenametofix" | grep "Subnet mask:" | awk '{print $NF}'`
interfaceRouter=`networksetup -getinfo "$interfacenametofix" | grep "^Router:" | awk '{print $NF}'`

interfaceDNS=`networksetup -getdnsservers "$interfacenametofix"`
networksetup -setv4off "$interfacenametofix"; networksetup -setmanual "$interfacenametofix" "$interfaceIP" "$interfaceSubnet" "$interfaceRouter"

networksetup -setdnsservers "$interfacenametofix" $interfaceDNS
networksetup -setnetworkserviceenabled "$interfacenametofix" off; route flush; killall -9 mDNSResponder; route flush; networksetup -setnetworkserviceenabled "$interfacenametofix" on; ifconfig "$interfacetofix" down; ifconfig "$interfacetofix" up

Command breakdown:

So what are the above bash commands doing? Glad you asked.

First command:

interfacetofix=`netstat -r -n -f inet | grep "0&" | awk '{print $NF}'`
We want to find out which interface has the bad route attached to it, because that is the interface we want to fix.

netstat -r -n -f inet prints the routing table of just IPv4 addresses.
| grep "0&"  we "pipe" the output of the last command into grep to search for the bad route.
| awk '{print $NF}' we "pipe" that output into awk and print only the last field.
We then place all those commands inside of interfacetofix=`...` to assign the variable interfacetofix the name of the interface with the bad route.

Second command:

interfacenametofix=`networksetup -listnetworkserviceorder | grep "$interfacetofix" -B1 | tail -2 | head -1 | cut -d " " -f 2-10`
Now we know what interface needs to be fixed, but Max OS X gives it a friendly name, and we need to know that name.

networksetup -listnetworkserviceorder prints all of the friendly interface names along with the device names.
| grep "$interfacetofix" -B1  we "pipe" the output of the last command into grep to search for the interface name by sending the non-friendly interface name and return the line before the matching line as the line before contains the friendly interface name.
| tail -2 | head -1 we "pipe" that output taking the last of top two lines (in case you renamed the friendly interface name with the interface name).
| cut -d " " -f 2-10 we "pipe" that output and remove the preceding interface order number so that we only get the friendly interface name (this command supports spaces in the interface name too).
We then place all those commands inside of interfacenametofix=`...` to assign the variable interfacenametofix the friendly name of the interface with the bad route.

Third command:

networksetup -setv4off "$interfacenametofix"; networksetup -setdhcp "$interfacenametofix"
Now we know the friendly name of the interface that needs to be fixed, so we tell the computer to disable the IPv4 protocol, and the immediately tell the system to enable that same interface with a IPv4 DHCP address.

networksetup -setv4off "$interfacenametofix" tells the computer to disable the IPv4 protocol
; networksetup -setdhcp "$interfacenametofix" tells the computer to enable the IPv4 protocol using a DHCP address.
Note the semicolon separating the two commands above... that's right these are actually two separate commands that you are running. The semicolon allows us to run a command, and then once that first command is finished run this second command. We could have separated these commands into two lines, but then if we were connected remotely, the first command would disconnect us and we would not be able to reconnect as the IPv4 network is turned off.

Fourth Command:

networksetup -setnetworkserviceenabled "$interfacenametofix" off; route flush; killall -9 mDNSResponder; route flush; networksetup -setnetworkserviceenabled "$interfacenametofix" on; ifconfig "$interfacetofix" down; ifconfig "$interfacetofix" up
While not strictly necessary, this command (it's actually a series of commands separated by semicolons) will do some basic maintenance to reset issues cause by the bad route. It will turn off the interface with the bad route. Then flush the routing table (which ironically by itself cannot get ride of the bad route in question in this post) which should clean up any other bad routing you may have. Then it flushes DNS cache. Then it flushes the routing table again (yes, running this a second time actually makes a difference). Then it will turn on the interface. Finally it will "bounce" (turn off and then turn on) the interface with the bad route using a second method.

The full story:

How to tell if I am one of the lucky ones with a bad route caused by Juniper Network Connect:

Some websites/connections to servers are not working, you will be able to ping some sites, but not all. Rebooting fixes the issue, but the next time you launch Network Connect then it's messed up again.

You try to telnet/ping a host and you get a "Cannot allocate memory" error.
$ telnet www.google.com 80
Trying 74.125.141.104...
telnet: connect to address 74.125.141.104: Cannot allocate memory
telnet: Unable to connect to remote host



To verify that the problem was caused by Juniper's "Network Connect.app", run the following command in the terminal:
netstat -r -f inet
and look for something similar to the underlined in the following output:
"
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
0&0x68             link#4             UCSc            4        0     en0
default            192.168.1.1        UGSc           21        4     en0
10.1.100.1         link#1             UHS             2    20206     lo0

"
If you see "0&0x68", then you have a bad route. Network routes to should not look like hex addresses... that's just not cool Juniper.

All of the things you have probably tried, or should have tried:

Running sudo route flush will not work to fix this problem, but nice try.

Running sudo route delete "0&0x68" will not work to fix this problem, again nice try.

Rebooting, yes it clears the issue, until Network Connect launches again... annoying almost-fix.

Doing the many steps outlined here will fix the issue, but you can't do that remotely, and most times you must repeat the steps when starting Network Connect... ok good Googleing here.

What conditions happen to cause Network Connect to create a bad route?

Mac Laptop + Juniper Network Connect + Changing networks while Network Connect is connected and the computer is asleep

Mac Laptop + Juniper Network Connect + running a hypervisor

Mac Laptop + Juniper Network Connect + The stars align


Still Broken?:

Typical Fix Walk-through:

Launch "Network Connect.app" (you bad route journey begins here)
Run the "Fix commands"
From "Network Connect.app" in the "Sign in at:" URI, hit enter (the network connect website should load now)
Have the user sign in
If using two-factor authentication, run the "Fix commands" again now
Continue sign-in
If everything looks good: Sign out of "Network Connect.app" and then quit "Network Connect.app".
The next time you launch Network Connect no bad routes should be created.

What to do if the "Fix Commands" didn't work:

1) Run these commands multiple times while connecting with network connect.
2) Are you sure you are running these commands as root
3) You are running all three commands right? Hit enter a few times after that last command.
4) Try it in bash, it should work in zsh, but I have yet to try it outside of bash
5) Send me your netstat -r -n -f inet command output and I'll take some time to see what's up.6) Try doing the manual steps as outlined here
7) Reboot and don't launch Network Connect (uncheck "Reopen windows when logging back in" in the restart dialogue box).
8) Purchase another VPN solution.

3 comments:

  1. Thanks for posting this, especially the detailed breakdown and background!

    ReplyDelete
  2. You saved my ass! I was pretty desperate figuring out why my Juniper VPN client wasn't working anymore. You're my hero man.

    ReplyDelete
  3. You can either make peace with the restrictions and security risks that come with free VPN services or you can save your data, online privacy, internet freedom and your money by opting for a premium and professional VPN service Every teenager with a laptop in a coffee shop is trying to hack accounts over public WiFis these days, and a VPN is your last line of defensetop10-bestvpn

    ReplyDelete