What Is ArvanCloud WAF and How It Works?
WAF is an application level firewall (or the seventh layer of network) designed and implemented to monitor, filter or block the data packets of web applications. Web Application Firewall (WAF) can be considered a tool to detect and prevent web attacks. The typical approach of a WAF is to use a set of rules to detect attacks; however, other approaches such as whitelisting or anomaly detection can also be utilized. Different WAF approaches are as follows:
- Pattern-based approach: Here, predetermined patterns are uploaded in WAF and the sent requests are matched with these patterns. In case a request corresponds with one of these patterns, it is blocked by WAF.
- Behavior-based approach: Knowing the users’ ordinary behaviors, WAF attempts to detect anomalies as attacks and prevent them.
- Score-based approach: In this case, each rule has a score. If a request corresponds with one of the rules, the total score of the corresponded rules are computed, and if the obtained value reaches the predetermined threshold, WAF blocks the request.
By inspecting all packets sent to the web server, WAF detects packets suspect of being an attack and prevents them from accessing the main server. Web application firewalls are typically implemented as a reverse proxy, but router and bridge modes are also available. WAF can be implemented as cloud or via hardware. The following table compares these two types of WAF:
Another solution is to use hybrid WAF, such that general attacks are detected by cloud WAF and more sophisticated and special attacks by hardware WAF.
Wrong Expectations from WAF
WAF is not an alternative to secure programming. To put it other words; no matter how strong is WAF, infiltration is probable if your program is vulnerable. WAF is generally one of the security mechanisms. By utilizing security technologies, these methods attempt to reduce threats.
Security mechanisms, on the other hand, seek to reduce vulnerability points using certain methods. Among security mechanisms we can refer to Secure Software Development Life Cycle. WAF in no way reduces or even reveals the vulnerabilities of your programs; rather, it decreases the possibility of using these vulnerabilities by hackers, and in case an attack occurs or is detected, it alerts the user to prevent infiltration. Of course, none of these two mechanisms negates the other, and both should be used simultaneously to provide more security.
As said before, WAF is designed to prevent attacks in application layer, and is useless in other areas such as network security. It should not be confused with Next Generation Firewall. Also, WAF is not a perfect solution. That is, depending on the set of WAF rules, it may not be able to detect certain attacks and some of the true requests may be detected as an attack. Moreover, many logical vulnerabilities have no pattern suitable to be detected and prevented.
How ArvanCloud WAF works?
When it was decided in ArvanCloud to offer WAF services, a research phase was first defined to find the best open source option. In initial searches, we could find the ModSecurity, the module that was first designed for Apache, then supported by Nginx. The number of papers and firms using ModSecurity (including Nginx Plus) was so high that we assuredly thought that we have the best option available for open source web firewall. Afterwards, we decided to implement it on our own servers. Implementation was simple and we just needed to install ModSecurity and load its connector module in Nginx. After launching the connecting module, it was the time to execute the intended tests on WAF service to verify its performance before releasing it. All tests were successfully done, and finally, we had to do benchmark testing. Given the high number of requests on ArvanCloud servers, we required a WAF service with latency less than 1 millisecond. Unfortunately, the benchmark testing results of ModSecurity were disappointing and could not satisfy our needs.
Abandoning ModSecurity, we searched for other alternatives that could be executed by OpenResty. Therefore, we concentrated on WAF services implemented by Lua language. Lua language enabled us to easily change WAF source code without any secondary compilation, and investigate the made modifications. Our option was lua-resty-waf which, according to its developer, had latency less than 1 millisecond. Although the implementation of lua-resty-waf took time more than ModSecurity, the benchmark testing results were extraordinarily better than ModSecurity, after it was launched and tested. Hence, we succeeded to find a WAF service that could satisfy our needs. However, an important question was raised: Is this service, accomplished as an academic project, ready to be executed in Production mode?
To find out the answer, we created a fork from the main project and wrote more tests to examine the WAF service performance. Several fundamental bugs were detected by these tests. For example, the service did not examine the body of POST requests. By applying bugfixes, all tests were passed, but the latency was increased (became more than 1 millisecond) due to adding the bugfixes. Next, we decided to optimize the WAF service.
How Did We Optimize Lua-resty-WAF Service?
CPU Profiling is one of the most significant methods of analyzing a running program. One of the main purposes of profiling a program is to improve its performance. It includes three stages in general:
- Profiling the program and extracting statistical information (detection stage)
- Rewriting the program code according to the obtained information (improvement stage)
- Re-profiling the program and comparing the information before and after rewriting (review stage)
Despite benchmark testing, profiling a program can provide more accurate information about the components of the program. This significantly helps detecting and rewriting the sections that took more time to be implemented.
In benchmark testing, it is not possible to detect the function that needed more time to be executed, and merely the time required for running the whole program can be measured. In contrast, profiling enables us to easily detect the time-consuming function and rewrite it if possible.
We used SystemTap to optimize lua-resty-waf. To exploit the tool, we had to rebuild the Linux kernel by debug-related packages. After rebuilding the Linux kernel and installing the SystemTap, we were able to profile the WAF service. We changed the SystemTap output of WAF service profiling into a bar chart using FlameGraph, so that it could be examined more conveniently. The following is an example of these charts that were used to inspect the WAF service:
The wider the bar is, the more CPU processing time is spent for that function.
We repeatedly did the same and changed the code every time and inspected the results. At last, we could increase the WAF service performance by 40%. Therefore, the latency of WAF service was at an acceptable level. Once again, we performed the tests on the WAF service to make sure that our modifications have no destructive effect on WAF service performance. Then we launched the WAF service on our Production servers to protect our users from online attacks. Despite the long time spent for WAF service optimization process, the result was satisfactory: achieving a reliable service with latency less than 1 millisecond. Ultimately, we released the results of our attempts on lua-resty-waf in Github.
An example of ArvanCloud WAF performance
Suppose that dvwa.com uses ArvanCloud cloud security WAF. All requests sent to this website arrive at ArvanCloud WAF. In this product, we utilize the score-based approach. Suppose that an SQLI attack is sent to this website in the following request:
In ArvanCloud WAF, the following rules for this request are matched:
Notice that in the last rule, the request score is calculated, and since it is greater than the determined threshold, the request is blocked.