Nitish Kumar's Blog

Layman's take on technology | Doesn't need to have a super brain | Just do your bit

How to: (Linux router utilizing multiple uplink providers)

with 14 comments


Utilizing Multiple DSL Gateways to get combined into Single Gateway

Multiple DSL connection often exist in corporate scenario due to higher gap in prices between DSL plan and Leased Line plans and we also find people struggling with connectivity issues due to downlink with any of the DSL connection, while other connections of them might be working due to failover shifting etc. After struggling with the same issue for a while, few internet resources helped us to devise a solution for the same. A Linux based Software router utilizing all the connections at the same time. Before the implementation details, let’s put a FAQ over here.

Que: Why we need a Linux based router, when already our DSLs are acting as hardware based router from our service provider?

Answer: There are multiple reasons for the same:

  1. Connection Reliability & Failover handling: You might be having multiple connections to internet, but still might be struggling due to downlink with any of the uplink as switching mechanism from one uplink is sure going to cumbersome in any environment. Automatic failover handling makes it extremely attractive solution for any demanding environment. Somewhere it scores higher over a dedicated leased due to its failover handling.
  2. Firewall/ Proxy: The same solution could provide us a mechanism to utilize our uplink connection as no more transparent to our inner network (as each request with pass through Linux router) henceforth could control the way traffic get utilized. It would be a central gateway for outgoing request.
  3. Better throughput & Load balancing: Theoretically and practically, if one has say three modems of strength 512 Kbps each, then this solution doesn’t provide the combined strength of 1.5 Mbps, but one could attain a better throughput for multiple connections like torrents and multiple applications utilizing and enjoy connections through each of the DSL at the same time.

Hardware/ Software requirements:

Minimum requirements might vary, but here I am providing the requirements, which I felt as somewhat standard.

  • Processor: P4 2.4 GHz or more.
  • RAM:
    512 MB or more.

There are additional benefit of engaging two or more Ethernet card over the same machine, say two; one handling the requests and the other one communicating with DSLs.

For security reasons, one may wish to isolate the DSLs from the inner network and in that case, you will need a separate Switch having only DSLs and this router connected to the same and no more connection to rest of the network. Rest of the network will be connected to other Ethernet card configured in preferably different subnet.

Step-By-Step Procedure :-

Here I am presenting the way, we did the things in most simplest and layman way, without going into far technical details. Sure there might be better or alternate or customized ways suiting your own environment. One may always tweak over any solution.

Here are the steps:

  1. Machine: Get a machine ready with Either of Latest Fedora, RHEL or latest of CentOS with standard installation covering all the important packages.
  2. IP Forwarding: The OS need to has IP forwarding enabled. Its disabled by default. For checking the same, you may type the following:

    cat /proc/sys/net/ipv4/ip_forward

    If output is 1, then nothing to do and if output is 0, then it needs to be ON.

    For permanently putting IP Forwarding as ON, you need to change the value of net.ipv4.ip_forward to 1 from 0 in the file

    /etc/sysctl.conf. The changes could take affect by either a reboot or by the command

    sysctl –p /etc/sysctl.conf

  3. Configure IPs for both the Ethernet cards. I am keeping mine in the same subnet. Keeping all the IPs in same subnet.

    Say my DSL IPs are [MY_DSL1_IP], [MY_DSL2_IP], [MY_DSL3_IP] and two IPs of this router are [Ethernet1_IP] and [Ethernet2_IP]. I am keeping my subnet as 172.16.x.x/16

  4. In next, you need to create multiple profiles of one Ethernet card (take eth1) to communicate with each of DSL (in our case three different profiles with three different IPs).

    Make copies of file ifcfg-eth1 as ifcfg-eth1:0, ifcfg-eth1:1 and ifcfg-eth1:2. [Location /etc/sysconfig/network-scripts]

    cp ifcfg-eth1 ifcfg-eth1:0
    cp ifcfg-eth1 ifcfg-eth1:1
    cp ifcfg-eth1 ifcfg-eth1:2

    Make changes in files as given below:

    Content of ifcfg-eth1

    # Accton Technology Corporation SMC2-1211TX

    DEVICE=eth1
    ONBOOT=yes
    BOOTPROTO=none
    HWADDR=00:10:b5:fd:7f:e7
    NETMASK=255.255.0.0
    IPADDR=[Ethernet2_IP]

    TYPE=Ethernet
    USERCTL=no
    IPV6INIT=no
    PEERDNS=yes

    Content of ifcfg-eth1:0

    # Accton Technology Corporation SMC2-1211TX

    DEVICE=eth1:0
    ONBOOT=yes
    BOOTPROTO=none
    HWADDR=00:10:b5:fd:7f:e7
    NETMASK=255.255.0.0
    IPADDR=[Profile0_IP]

    TYPE=Ethernet
    USERCTL=no
    IPV6INIT=no
    PEERDNS=yes

    Content of ifcfg-eth1:1

    # Accton Technology Corporation SMC2-1211TX

    DEVICE=eth1:1
    ONBOOT=yes
    BOOTPROTO=none
    HWADDR=00:10:b5:fd:7f:e7
    NETMASK=255.255.0.0
    IPADDR=[Profile1_IP]
    TYPE=Ethernet
    USERCTL=no
    IPV6INIT=no
    PEERDNS=yes

    Content of ifcfg-eth1:2

    # Accton Technology Corporation SMC2-1211TX

    DEVICE=eth1:2
    ONBOOT=yes
    BOOTPROTO=none
    HWADDR=00:10:b5:fd:7f:e7
    NETMASK=255.255.0.0
    IPADDR=[Profile2_IP]

    TYPE=Ethernet
    USERCTL=no
    IPV6INIT=no
    PEERDNS=yes

  5. Add the entries in your rt_tables file for your DSLs

    Just do vi /etc/iproute2/rt_tables and put enties, so that the file contents will look like something as given below:

1 dsl1
2 dsl2
3 dsl3

#
# reserved values
#
255    local
254    main
253    default
0    unspec
#
# local
#
#1    inr.ruhep

Now most important step, place the routes in the rc.local file. Just do vi /etc/rc.d/rc.local and put enties, so that the file contents will look like something as given below:

#!/bin/sh

#

# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don’t
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
/sbin/ip route add 172.16.0.0/16 dev eth1:0 src [Profile0_IP] table dsl1
/sbin/ip route add 172.16.0.0/16 dev eth1:1 src [Profile1_IP] table dsl2
/sbin/ip route add 172.16.0.0/16 dev eth1:2 src [Profile2_IP] table dsl3
/sbin/ip route add default via [MY_DSL1_IP] table dsl1
/sbin/ip route add default via [MY_DSL2_IP] table dsl2
/sbin/ip route add default via [MY_DSL3_IP] table dsl3
/sbin/ip rule add from [MY_DSL1_IP] table dsl1
/sbin/ip rule add from [MY_DSL2_IP] table dsl2
/sbin/ip rule add from [MY_DSL3_IP] table dsl3
/sbin/ip route add default scope global nexthop via [MY_DSL1_IP] dev eth1:0 weight 1 nexthop via [MY_DSL2_IP] dev eth1:1 weight 1 nexthop via [MY_DSL3_IP] dev eth1:2 weight 1
/usr/bin/nohup usr/local/bin/gwping >>/var/log/gwping.log 2>&1

As about the last line and gwping, then it will come in later step.

  1. The next part is to place a script, which will take care of failover shifting and logging. What it basically do is to ping some public website through each of DSL, determining that when any of DSL went down and switching routing rules accordingly.

    Here is our customized script saved as /usr/local/bin/gwping:

    GWCMD="/sbin/ip route replace default scope global"

    # Conventionally 0 indicates success in this script.

    # Time between checks in seconds

    SLEEPTIME=10

    #IP Address or domain name to ping. The script relies on the domain being

    #pingable and always available

    TESTIP=http://www.yahoo.com

    #Ping timeout in seconds

    TIMEOUT=2

    # External interfaces

    EXTIF1=eth1:0

    EXTIF2=eth1:1

    EXTIF3=eth1:2

    #IP address of external interfaces. This is not the gateway address.

    IP1=[Profile0_IP]

    IP2=[Profile1_IP]

    IP3=[Profile2_IP]

    #Gateway IP addresses. This is the first (hop) gateway, could be your router IP

    #address if it has been configured as the gateway

    GW1=[MY_DSL1_IP]

    GW2=[MY_DSL2_IP]

    GW3=[MY_DSL3_IP]

    # Relative weights of routes. Keep this to a low integer value.

    W1=1

    W2=1    

    W3=1

    # Broadband providers name; use your own names here.

    NAME1=dsl1

    NAME2=ds12

    NAME3=dsl3

    #No of repeats of success or failure before changing status of connection

    SUCCESSREPEATCOUNT=4

    FAILUREREPEATCOUNT=1

    # Do not change anything below this line

    # Last link status indicates the macro status of the link we determined. This is down initially to force routing change upfront. Don’t change these values.

    LLS1=1

    LLS2=1

    LLS3=1

    # Last ping status. Don’t change these values.

    LPS1=1

    LPS2=1

    LPS3=1

    # Current ping status. Don’t change these values.

    CPS1=1

    CPS2=1

    CPS3=1

    # Change link status indicates that the link needs to be changed. Don’t change these values.

    CLS1=1

    CLS2=1

    CLS3=1

    # Count of repeated up status or down status. Don’t change these values.

    COUNT1=0

    COUNT2=0

    COUNT3=0

    while : ; do

    /bin/ping -W $TIMEOUT -I $IP1 -c 1 $TESTIP > /dev/null 2>&1

    RETVAL=$?

    if [ $RETVAL -ne 0 ]; then

    echo $NAME1 Down

    CPS1=1

    else

    CPS1=0

    fi

    if [ $LPS1 -ne $CPS1 ]; then

    echo Ping status changed for $NAME1 from $LPS1 to $CPS1

    COUNT1=1

    else

    if [ $LPS1 -ne $LLS1 ]; then

    COUNT1=`expr $COUNT1 + 1`

    fi

    fi

    if [[ $COUNT1 -ge $SUCCESSREPEATCOUNT || ($LLS1 -eq 0 && $COUNT1 -ge $FAILUREREPEATCOUNT) ]]; then

    echo Uptime status will be changed for $NAME1 from $LLS1

    CLS1=0

    COUNT1=0

    if [ $LLS1 -eq 1 ]; then

    LLS1=0

    else

    LLS1=1

    fi

    else

    CLS1=1

    fi

    LPS1=$CPS1

    /bin/ping -W $TIMEOUT -I $IP2 -c 1 $TESTIP > /dev/null 2>&1

    RETVAL=$?

    if [ $RETVAL -ne 0 ]; then

    echo $NAME2 Down

    CPS2=1

    else

    CPS2=0

    fi

    if [ $LPS2 -ne $CPS2 ]; then

    echo Ping status changed for $NAME2 from $LPS2 to $CPS2

    COUNT2=1

    else

    if [ $LPS2 -ne $LLS2 ]; then

    COUNT2=`expr $COUNT2 + 1`

    fi

    fi

    if [[ $COUNT2 -ge $SUCCESSREPEATCOUNT || ($LLS2 -eq 0 && $COUNT2 -ge $FAILUREREPEATCOUNT) ]]; then

    echo Uptime status will be changed for $NAME2 from $LLS2

    CLS2=0

    COUNT2=0

    if [ $LLS2 -eq 1 ]; then

    LLS2=0

    else

    LLS2=1

    fi    

    else

    CLS2=1

    fi

    LPS2=$CPS2

    /bin/ping -W $TIMEOUT -I $IP3 -c 1 $TESTIP > /dev/null 2>&1

    RETVAL=$?

    if [ $RETVAL -ne 0 ]; then

    echo $NAME3 Down

    CPS3=1

    else

    CPS3=0

    fi

    if [ $LPS3 -ne $CPS3 ]; then

    echo Ping status changed for $NAME3 from $LPS3 to $CPS3

    COUNT3=1

    else

    if [ $LPS3 -ne $LLS3 ]; then

    COUNT3=`expr $COUNT3 + 1`

    fi

    fi

    if [[ $COUNT3 -ge $SUCCESSREPEATCOUNT || ($LLS3 -eq 0 && $COUNT3 -ge $FAILUREREPEATCOUNT) ]]; then

    echo Uptime status will be changed for $NAME3 from $LLS3

    CLS3=0

    COUNT3=0

    if [ $LLS3 -eq 1 ]; then

    LLS3=0

    else

    LLS3=1

    fi

    else

    CLS3=1

    fi

    LPS3=$CPS3

    # Changeover Logic here

    if [[ $CLS1 -eq 0 || $CLS2 -eq 0 || $CLS3 -eq 0 ]]; then

    if [ $LLS1 -eq 0 ]; then

    echo Adding $NAME1

    GWCMD="$GWCMD nexthop via $GW1 dev $EXTIF1 weight 1"

    fi

    if [ $LLS2 -eq 0 ]; then

    echo Adding $NAME2

    GWCMD="$GWCMD nexthop via $GW2 dev $EXTIF2 weight 1"

    fi

    if [ $LLS3 -eq 0 ]; then

    echo Adding $NAME3

    GWCMD="$GWCMD nexthop via $GW3 dev $EXTIF3 weight 1"

    fi

    echo "Change execute now \n $GWCMD"

    $GWCMD

    GWCMD="/sbin/ip route replace default scope global"

    fi

    sleep $SLEEPTIME

    done

  2. The last place is to make DNS entries. Just do a vi /etc/resolv.conf and type the following:

    nameserver 208.67.222.222

    I am using OpenDNS IP as DNS here. One may wish to have any other reliable one.

  3. Oops!! forgot one more thing, do a chmod 777 /var/log/gwping.log

    Now, you just need to reboot the same machine and then use [Ethernet1_IP] as Gateway in your production machines/ Proxy/ Firewall and here you go.

Waiting for your comments. Soon will post the HowTo, if one wishes to get the same thing done without putting one additional ethernet card in action means with one card only.

PDF Version

About these ads

14 Responses

Subscribe to comments with RSS.

  1. Hi
    How do I exclude Ip address from this script.
    Each time the script changes to new gateway my Voip phone cuts out..

    I also have the same problem when I’m running a shh session.

    Ta
    Steve

    Steve

    August 24, 2009 at 3:38 pm

  2. Sorry, but this problem has to be remained. Even we are facing the same for our normal pop3 connections or secure channels. One way to avoid similar issue is to keep the checking interval more longer, say 5 mins or 10 mins as per your needs.

    Nitish Kumar

    August 24, 2009 at 11:59 pm

  3. GWCMD=”/sbin/ip route replace default scope global”

    # Conventionally 0 indicates success in this script.

    # Time between checks in seconds

    SLEEPTIME=10

    #IP Address or domain name to ping. The script relies on the domain being

    #pingable and always available

    TESTIP=www.yahoo.com

    #Ping timeout in seconds

    TIMEOUT=2

    # External interfaces

    EXTIF1=eth1

    EXTIF2=eth2

    #IP address of external interfaces. This is not the gateway address.

    IP1=192.168.0.11

    IP2=192.168.1.3

    #Gateway IP addresses. This is the first (hop) gateway, could be your router IP

    #address if it has been configured as the gateway

    GW1=192.168.0.150

    GW2=192.168.1.1

    # Relative weights of routes. Keep this to a low integer value.

    W1=1

    W2=1

    # Broadband providers name; use your own names here.

    NAME1=rt_link1

    NAME2=rt_link2

    #No of repeats of success or failure before changing status of connection

    SUCCESSREPEATCOUNT=4

    FAILUREREPEATCOUNT=1

    # Do not change anything below this line

    # Last link status indicates the macro status of the link we determined. This is down initially to force routing change upfront. Don’t change these values.

    LLS1=1

    LLS2=1

    # Last ping status. Don’t change these values.

    LPS1=1

    LPS2=1

    # Current ping status. Don’t change these values.

    CPS1=1

    CPS2=1

    # Change link status indicates that the link needs to be changed. Don’t change these values.

    CLS1=1

    CLS2=1

    # Count of repeated up status or down status. Don’t change these values.

    COUNT1=0

    COUNT2=0

    while : ; do

    /bin/ping -W $TIMEOUT -I $IP1 -c 1 $TESTIP > /dev/null 2>&1

    RETVAL=$?

    if [ $RETVAL -ne 0 ]; then

    echo $NAME1 Down

    CPS1=1

    else

    CPS1=0

    fi

    if [ $LPS1 -ne $CPS1 ]; then

    echo Ping status changed for $NAME1 from $LPS1 to $CPS1

    COUNT1=1

    else

    if [ $LPS1 -ne $LLS1 ]; then

    COUNT1=`expr $COUNT1 + 1`

    fi

    fi

    if [[ $COUNT1 -ge $SUCCESSREPEATCOUNT || ($LLS1 -eq 0 && $COUNT1 -ge $FAILUREREPEATCOUNT) ]]; then

    echo Uptime status will be changed for $NAME1 from $LLS1

    CLS1=0

    COUNT1=0

    if [ $LLS1 -eq 1 ]; then

    LLS1=0

    else

    LLS1=1

    fi

    else

    CLS1=1

    fi

    LPS1=$CPS1

    /bin/ping -W $TIMEOUT -I $IP2 -c 1 $TESTIP > /dev/null 2>&1

    RETVAL=$?

    if [ $RETVAL -ne 0 ]; then

    echo $NAME2 Down

    CPS2=1

    else

    CPS2=0

    fi

    if [ $LPS2 -ne $CPS2 ]; then

    echo Ping status changed for $NAME2 from $LPS2 to $CPS2

    COUNT2=1

    else

    if [ $LPS2 -ne $LLS2 ]; then

    COUNT2=`expr $COUNT2 + 1`

    fi

    fi

    if [[ $COUNT2 -ge $SUCCESSREPEATCOUNT || ($LLS2 -eq 0 && $COUNT2 -ge $FAILUREREPEATCOUNT) ]]; then

    echo Uptime status will be changed for $NAME2 from $LLS2

    CLS2=0

    COUNT2=0

    if [ $LLS2 -eq 1 ]; then

    LLS2=0

    else

    LLS2=1

    fi

    else

    CLS2=1

    fi

    LPS2=$CPS2

    # Changeover Logic here

    if [[ $CLS1 -eq 0 || $CLS2 -eq 0 ]]; then

    if [ $LLS1 -eq 0 ]; then

    echo Adding $NAME1

    GWCMD=”$GWCMD nexthop via $GW1 dev $EXTIF1 weight 1″

    fi

    if [ $LLS2 -eq 0 ]; then

    echo Adding $NAME2

    GWCMD=”$GWCMD nexthop via $GW2 dev $EXTIF2 weight 1″

    fi
    echo “Change execute now \n $GWCMD”

    $GWCMD

    GWCMD=”/sbin/ip route replace default scope global”

    fi

    sleep $SLEEPTIME

    done

    in the booting its stuck at “starting apt”

    Tushar

    November 18, 2010 at 4:27 pm

    • There must be something wrong in the script.

      Remove the entry of script from /etc/rc.d/rc.local

      And then try to run the script manually by ./script_name to see if it works.

      Nitish Kumar

      November 18, 2010 at 5:03 pm

      • thanks it’s work fine when i run the scrip manually but its default goes to second link(rt_link2) 192.168.1.1. I want the default link is frist link(rt_link1) 192.168.0.150 the rc.local file are as follows

        /etc/rc.local

        /sbin/ip route add 192.168.0.0/24 dev eth1 src 192.168.0.11 table rt_link1
        /sbin/ip route add 192.168.0.1/24 dev eth2 srv 192.168.1.3 table rt_link2
        /sbin/ip route add default via 192.168.0.150 table rt_link1
        /sbin/ip route add default via 192.168.1.1 table rt_link2
        /sbin/ip rule add from 192.168.0.150 table rt_link1
        /sbin/ip rule add from 192.168.1.1 table rt_link2
        /sbin/ip route add default scop global nexthop via 192.168.0.150 dev eth1 weight 1 nexthop via 192.168.1.1 dev eth2 weight 1
        #/usr/bin/nohup /usr/local/bin/gwping >> /var/log/gwping.log 2>&1

        Tushar

        November 19, 2010 at 10:49 am

  4. Hi Nithish,

    This is shiva i have cleared in routing with multiple internet connections using virtual IP’s. But i have one doubt on this..
    Need to split the connections for client users like 172.16.0.0 to 172.16.0.13 – DSL_IP1 & 172.16.0.14 to 176.16.0.24 – DSL_IP2 like these i have tried in squid TCP_OUTGOING_ADDRESS but it was not worked yet any other way to split up the users.

    Shiva

    September 29, 2011 at 10:24 pm

    • That may be done by routing only, not on squid level. Either use external router or make the rules in iptables accordingly 

      Nitish Kumar

      September 30, 2011 at 9:02 am

      • Hello Nithish,

        I have tried using SNAT rule too but it was not worked by using network alias. Plz send me the step by step configuration as below mentioned firewall designed.

        eth0 : 1.1.25.1
        eth1: 1.1.6.7 and this gateway as 1.1.6.6
        eth1:1 1.1.9.10 and this gateway as 1.1.9.9
        eth1:2 1.1.10.11 and this gateway as 1.1.10.10

        we are using more than 40 systems. using the network source as 1.1.25.50 to 1.1.25.90 i want to split up the gateways like 1.1.25.50 to 1.1.25.60 with 1.1.6.6 &&& 1.1.25.61 to 1.1.25.70 with 1.1.9.9 &&& 1.1.25.71 to 1.1.25.90 with 1.1.10.10 gateways. This was my firewall designed. Please send me the clear solutions for this. Regarding your help…

        Regards,
        Shiva

        Shiva

        October 1, 2011 at 8:29 am

  5. Hello Nithish,

    I have tried using SNAT rule too but it was not worked by using network alias. Plz send me the step by step configuration as below mentioned firewall designed.

    eth0 : 1.1.25.1
    eth1: 1.1.6.7 and this gateway as 1.1.6.6
    eth1:1 1.1.9.10 and this gateway as 1.1.9.9
    eth1:2 1.1.10.11 and this gateway as 1.1.10.10

    we are using more than 40 systems. using the network source as 1.1.25.50 to 1.1.25.90 i want to split up the gateways like 1.1.25.50 to 1.1.25.60 with 1.1.6.6 &&& 1.1.25.61 to 1.1.25.70 with 1.1.9.9 &&& 1.1.25.71 to 1.1.25.90 with 1.1.10.10 gateways. This was my firewall designed. Please send me the clear solutions for this. Regarding your help…

    Regards,
    Shiva

    Shiva

    October 4, 2011 at 9:38 pm

  6. Hello Nitish,

    i want to try out this on my centos 6, with 2x wan.
    Is there anything that i have to considere?
    I red all post, it seems there are still trouble.

    regards
    Nitha

    Nitha

    November 30, 2011 at 7:06 pm

  7. hello, Mr. Kumar, i wanna ask about content of ifcfg-eth1:0 or else in line IPADDR=[Profile0_IP], what should i type on it [Profile0_IP]. i really dont know about it, hope it will help me for somthin. thx

    Freddy

    September 23, 2012 at 9:58 pm

    • and, can i use my physical eth device like eth2 or eth3 to applying this configuration.

      welcome2u

      September 23, 2012 at 10:01 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 2,851 other followers

%d bloggers like this: