Wednesday, December 10, 2008

Per-Traffic Load Balancing

Contents

[hide]

[edit] Introduction

Bandwidth management is an essential part of every day operation for typical ISP's, business, and even everyday home users. There are many different types of management tools available to RouterOS users, QoS, rate-limitng, packet-limiting, to name few

I personally operate a wireless ISP in an area that has no other type of conventional high-speed internet conection (ie. cable, fibre or DSL). Not having access to fibre myself, I am in a situation where the single fastest backbone connection I can get does not provide enough bandwidth for me to have only one connection. As a result of this limitation load-balancing multiple internet backbone connections is very important.

In the past I have used ECMP, persisten per connection styled load-balancing (see http://wiki.mikrotik.com/wiki/Load_Balancing), as well as various other methods However, I found all of then lacking in various different areas (not load-balancing correctly, broken large HTTP downloads, IM problems, to name a few issues). I then investigated a way to give me more control over my bandwidth while minimizing the potential problems. The end result was a per-traffic type of load-balancing. This tutorial is designed address that one specific area in depth, at a later date, I may expand/add additional info on fail-over, and other topics

[edit] Functions of RouterOS used

  • Firewall mangle rules
  • Firewall address-lists
  • Routing

[edit] Step 1 - How to break up to traffic

Before you even log onto your RouterOS box, you should have an idea of how you want to divide your traffic, and have an understanding of what traffic can and what traffic can't be broken up.

Here is an example of what you may want to attempt to separate (in no particular order)

  • HTTP traffic (port 80)
  • SSL traffic (port 443)
  • POP3 traffic (port 110)
  • SMTP traffic (port 25)
  • P2P traffic (various port)
  • Unknown traffic (various port)

After making a list of the traffic type, and the ports they operate on, you need to look at the list and decide if it is at all even capable of being forced out a certain internet connection.exión a internet.

Using the above list as an example here is what I came up with

  • HTTP traffic (no problems found yet)
  • SSL traffic (some issues, for normal SSL websites and 90% of all software it is not a problem, I'll explain why issues can arise later)
  • POP3 traffic (no problems found yet)
  • SMTP traffic (no problems found yet)
  • P2P traffic (must go out same internet connection as Unknown Traffic, I'll explain why this is later)
  • Unknown traffic (must go out same internet connection as P2P traffic, I'll explain why this is later)

Now for a quick explanation of why there can be some issues depending on the traffic type. Some websites/programs do not play friendly with multiple requests from different IP addresses, this is the reason why ECMP has so many problems. I will provide a simple solution to the rare conditions where load-balancing can't be completed.

Again using the above as an example lets explain where and why issues can be arise. SSL - The beauty of a website is that are separate requests for different data, ie. loading pictures from 3 different sources would be a request to each respective server. The result is if a we have a website that uses both SSL and HTTP traffic we know that in most cases the website will just answer the request without care of the originating IP address. However, and this is a special case, if the website/program developer checks where the requests are originating and they find that the IP are different, they may not succesfully answer those requests (this can be by design or by accident). I had 2 cases of this, in both cases it was secured medical websites that where using it as a method of protecting their data.

P2P and Unknown - I am addressing these together because the issue is one and the same. RouterOS doesn't identify P2P based on any single condition, but instead analyzes the packets! This means RouterOS needs time to watch the data before it realizes that is in fact P2P traffic. As a result RouterOS doesn't know the data is P2P until AFTER the connection is made. This is important because the only way to send data out a specific internet connection you must know that traffic is BEFORE the connection is made. So as with P2P traffic, unknown traffic is just that, it is unknown. By marking the unknown traffic though you can control what internet connection is used for both P2P and the left over unknown traffic (very useful!)

[edit] Step 2 - Setting Up the Network

Before we get to the real point of this tutorial we need a fictious network we can use as an example.

  • client computers (172.18.1.0/24)
  • Internet Gateways (10.0.1.1/24, 10 0.2.1/24)
  • RouterOS IPs (10.0.1.2/24, 10.0.2.2/24)

Network Map

Asuming that the IPs, default routes, and DNS settings are already in place the following allow users to get internet access.

Create Address-List for permitted use of internet 

/ ip firewall address-list
add list="Salida_Internet" address=172.18.1.0/24 comment="" disabled=no

Create Address-List to bypass load-balancing 

/ ip firewall address-list
add list="WAN-01" address=172.18.1.24/32 comment="" disabled=no
add list="WAN-02" address=172.18.1.76/32 comment="" disabled=no

Apply 'Masquerading'to the traffic leaving the WAN interfaces 

/ ip firewall nat
add chain=srcnat action=masquerade out-interface="WAN - 01" src-address-list="Allowed - Internet" comment="Gateway 10.0.1.1/24" disabled=no
add chain=srcnat action=masquerade out-interface="WAN - 02" src-address-list="Allowed - Internet" comment="Gateway 10.0.2.1/24" disabled=no

Clients should be able to browse the internet, however only one internet connection would be used (the current default route in RouterOS)

[edit] Step 3 - Using RouterOS's Mangle Tool to mark specific traffic

This following is the necessary RouterOS commands to mark particular traffic for a certain route, we are using the same example for traffic types as in the beginning, HTTP, SSL, POP3, SMTP, P2P, and Unknown 

/ ip firewall mangle
add chain=prerouting action=mark-routing new-routing-mark="WAN-01" src-address-list="WAN-01" passthrough=no comment="" disabled=no
add chain=prerouting action=mark-routing new-routing-mark="WAN-02" src-address-list="WAN-02" passthrough=no comment="" disabled=no
add chain=prerouting action=mark-routing new-routing-mark="HTTP traffic" passthrough=no dst-port=80 protocol=tcp comment="" disabled=no
add chain=prerouting action=mark-routing new-routing-mark="SSL traffic" passthrough=no dst-port=443 protocol=tcp comment="" disabled=no
add chain=prerouting action=mark-routing new-routing-mark="POP3 traffic" passthrough=no dst-port=110 protocol=tcp comment="" disabled=no
add chain=prerouting action=mark-routing new-routing-mark="SMTP traffic" passthrough=no dst-port=25 protocol=tcp comment="" disabled=no
add chain=prerouting action=mark-routing new-routing-mark="P2P traffic" passthrough=no p2p=all-p2p comment="" disabled=no
add chain=prerouting action=mark-routing new-routing-mark="Unknown traffic" passthrough=no comment="" disabled=no

The first two lines provide a method of marking particular clients to bypass the load-balancing and use only a certain internet connection for all of their traffic. The following lines with mark traffic based on the dst-port, notice how we are not passing though, also notice that we are marking all traffic, even if not known, this always for a different internet connection to be specified fot the Unknown and P2P instead of using the router's default route. I also mark P2P separately even though it must go out the same internet connection as Unknown traffic. I do this for a couple of reasons, one is that I could easily stop all traffic by simply disabling a route, and the other is in my RouterOS configuration I use a lot of QoS, it very easy to remember how everything is configured if the QoS mirrors the load-balancing

So now we are marking traffic for their respectives routes, next is to add those actual routes.

[edit] Step 4 - Using the routing functions of RouterOS to force traffic ou certain internet connections

This following is the necessary RouterOS commands to provide routes for the marked HTTP, SSL, POP3, SMTP, P2P, and Unknown Traffic 

/ ip route
add dst-address=0.0.0.0/0 gateway=10.0.1.1 scope=255 target-scope=10 routing-mark="WAN - 01" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.2.1 scope=255 target-scope=10 routing-mark="WAN - 02" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.2.1 scope=255 target-scope=10 routing-mark="HTTP traffic" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.2.1 scope=255 target-scope=10 routing-mark="SSL traffic" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.2.1 scope=255 target-scope=10 routing-mark="POP3 traffic" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.1.1 scope=255 target-scope=10 routing-mark="SMTP traffic" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.1.1 scope=255 target-scope=10 routing-mark="P2P traffic" comment="" disabled=no
add dst-address=0.0.0.0/0 gateway=10.0.1.1 scope=255 target-scope=10 routing-mark="Unknown traffic" comment="" disabled=no

The first two lines provide the routes necessary to give the clients that are not being load-balanced via traffic type the correct gateway to the internet. The remaining lines are the routes necessary to provide the appropiate gateway based on traffic type

[edit] Step 5 - Review what you've created

What did you do: * You intelligently broke up your internet traffic into different types * Your marked that traffic using the Mangle Tool of RouterOS * You created a bypass list to allow certain IP to bypass the Per-Traffic Load-Balancing * You assigned particular routes based on the traffic you marked with Mangle

[edit] Conclusion

What you have done is very powerful and this tutorial provides just the tip of the iceberg so to say. The traffic types I have listed here are only a small amout of the total and you may want to add many others (ie. DNS, Terminal Services, ICMP, etc). This can be used is a great deal of different ways be it in a small offfice enviroment (separating your email usage from your browsing) or a large wireless ISP (for load-balancing). Some tips and final thougths

1. Think though what you want to achieve before you start 2. Pay attention to the traffic of type on each line (is it incoming intensive or outgoing intensive) 3. Finally experiment, nothing better then finding better, faster, and more intelligent ways to improve your services with a little capital costs

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)