A reader recently told me about a problem with the Sophos UTM web application firewall.
Due to the Corona crisis, many of the employees have switched to the home office and have since been using more connections to the Exchange server, which are protected by the WAF of the Sophos UTM. Since there was a higher load on the UTM during the day due to the many home office users, connections that ran via the WAF became very slow. During the day, loading times of over 60 seconds occurred. However, the CPU and RAM utilization of the UTM did not show any bottlenecks.
The reader then opened a ticket with Sophos Support, who were able to determine that the WAF has too few free worker processes and therefore timeouts and long loading times occur when accessing the services via WAF.
In this case, the following message was found in the reverse proxy log of the UTM (/var/log/reverseproxy.log):
2020:03:31-17:24:14 vpn2-1 httpd[8532]: [mpm_worker:error] [pid 8532:tid 4147619520] AH00288: scoreboard is full, not at MaxRequestWorkers
According to Sophos Support, the WAF of the UTM has 800 worker processes available; if all worker processes are busy, the problem described above occurs.
The problem was finally solved by increasing the number of worker processes (see end of article). However, this was not the reason why I received a nice mail from the reader. Here is an excerpt from the email:
I came across this article while searching Sophos:
https://community.sophos.com/kb/en-us/123512
There is a status page for the WAF, but it can only be accessed via an SSH tunnel.
I am currently tinkering with Powershell and trying to parse the page to output it to PRTG, but I haven't found anything yet on how to build an SSH tunnel from the script.
WAF, PRTG and PowerShell then aroused my interest and together we built a sensor for PRTG (many thanks at this point!), which displays the performance data of the UTM in PRTG.
The WAF overview page looks like this, for example:
A small PowerShell sensor, which establishes an SSH tunnel to the UTM and reads the data from the status page, now displays the values in PRTG:
The screenshot shows a first attempt for the sensor, the current version of the sensor can now display the following values:
- Busy Workers
- Idle Workers
- Requests per Second
- Bytes per second
- Duration per request
- Cache Index Usage
- Cache Usage
If someone would also like to display the performance data of the WAF in PRTG, the sensor can be downloaded here:
The sensor requires the PowerShell module Posh-SSH. If there are problems with the PowerShell module in connection with Windows Server 2019, please here look. Lines 2 to 4 in the script must be adjusted accordingly.
Note: The number of worker processes of the UTM can be increased in the file "/var/storage/chroot-reverseproxy/usr/apache/conf/mpm.conf":
In this case, however, you should keep an eye on the CPU and RAM utilization of the UTM.