In my last post, I mentioned that performance Engineer needs to have basic understanding as how load balancers work, its required skill for any performance engineer who is working in big enterprise environment, as without understanding this,your performance tests results are useless as response time also depends on how traffic is routed to the backend servers.So in this post, I am going to try and explain as how I make sure that my tests also stress the load balancer in the meaning full way as close to the production behavior.
Load balancers in majority of the cases identify clients with either a source IP or by setting the persistent cookie header to the client request.Source IP is the IP of the client who is using the applications.In the corporate environment, most load balancers uses the source IP to identify the client and route the requests because most of the environment infrastructure is known to all the concerned teams and revealing IP Address is not considered as the risk.Load Balancers reads the clients IP and based on the load balancing algorithm forwards the requests to the appropriate backend servers.We can also have source IP Stickiness to the instance of the servers also referred as server affinity.
Load balancers depending on the algorithm and requirements of the applications, can also implement the cookie based solutions. Load balancers in this case adds the cookies to the client requests before routing it to the appropriate backend servers.So each request going back and forth will be carrying this cookie with it.So as long as cookies are present in the requests, connections of the clients are maintained to the respective instances of the backend servers.Cookie based implementation also helps in maintaining the connection balance across the servers in the backend in case for some reasons source IP based persistence fails.
I feel these are two most commonly used ways for implementing load balancing solutions and almost all load balancing vendors support these solutions.
Now the question here is how should we test Load Balancers during Performance testing ? How do we ensure that Load balancer is working the way it should work ?
In case if you do not know anything about Load Balancers, then this might be tricky stuff for you ,but if you know algorithm and how Load Balancer is implementing its algorithm, then this might be a cake walk for you.
Below are some of the steps I recommend one should perform during performance testing whenever you have Load Balancers in your environments,
1. Identify how your Load Balancer is implementing the Load Balancing.How does it spread the requests across backend farm of servers.Understand the algorithm and its implementing.
2.IP Spoofing is required.Sometimes back, I had written about IP Spoofing here. Feel free to spend some time reading it.We need as many IP as the number of users we are testing for depending on the algorithm of the Load balancer.If Algorithm is using Source IP, then it will route the requests based on Source IP.
3. Once IP Spoofing is enabled.Scripts should and must output the IP Address of the Vuser.I am aware most load testing tools provide you a interface where in you can out put the IP address of the user.Below is the piece of code which can be used in LoadRunner,
ip = lr_get_vuser_ip();
lr_output_message(“The IP address is %s”, ip);
lr_output_message(“IP spoofing disabled”);
4. If the load balancing is done via cookies, make sure you identify and not comment out the cookies set by the Load Balancers,some tools store persistent cookies in memory, this don’t show up in your scripts,they are handled automatically by the tool.LoadRunner falls in this category of tools which handles persistent cookies automatically.You need to use advanced logging to see those cookies going back and forth.If possible log the cookies to the log along with timestamp, this will help you to troubleshoot issues in complex and shared environments.
5. I often make sure that I capture run time stats as where the client request is routed to in the back end farm.I do this by using this technique. Feel free to spend sometime reading that as well. All you got to do is correlate the headers values which displays the server information and write it to the log with timestamp.This will really help you in runtime debugging .
6. If your sole objective is to test Load Balancer , make sure you run only single script and not multiple scripts, as this will rule out the connection imbalance issues normally called as network bottlenecks.Not all connection exit gracefully across all layers at TCP Stack.
Sometimes I have seen lot of folks using webserver logs and retrieving number of hits from there for the duration of the test run as measurable way of testing load balancers, I agree that in some cases its correct and sometimes it could be wrong in case your application uses some kind of caching mechanism,we need to remember always that our runtime settings for the scenario has a strong influence on the hits per second.In most of the applications, static and non resource contents like images/js/css files are cached.If you are using webservers logs to analyze load balancers work, then make sure you do comparison only for POST requests and not for GET requests.Here is reasons as why POST requests makes sense.
Once you have all these stats, it will make your job extremely easy to identify and say whether load balancer is doing its job correctly or not.
Hope this helps.
Tags: Load Balancer