Regarding nwaf free module

gnusys

Member
I am a developer providing an Nginx plugin to the popular cPanel control panel and has been providing Nemesida( free) as an option along with Mod_sec3 for installation along with the Nginx RPM's I provide. I had a couple of questions

1. Do you have any plans to revoke the free module in future? I am asking this because I know that monetization is a requirement for any company and its fair too.
Also since its not a fully Open Source project, I am sure you alone is putting a lot of effort in this and unless it gets rewarded it is not sustainable

2. Since my plugin is mostly used in the web hosting industry, it is common to have servers with more than 4000 vhost .Nginx config test, reload,restart etc slows down a lot with such high number of vhost and mod_sec3 cannot even be used with vhost numbers higher than 20-30 .I have seen Nemesida is better, but enabling Nemesida slows down Nginx reload/restart/config test etc, so far it is acceptable upto 30-40 vhosts and I havent been able to test this on anything beyond as the servers are all production and cant compromise on uptime. Have you considered the impact of loading the plugin on such high number of configs ( lots of config files to parse )

3. Is there a way I can have the nwaf log be written to a custom log file and not the standard nginx error log?

4. Is there a way to have nwaf off by default and can be turned up on a per vhost level in the server {} context
 

rr

Administrator
Hello,

1. No, we do not have a plan to revoke the Nemesida WAF Free. But this is part of a commercial product, so we do not want to disclose the source code in order to avoid illegal use of the full version.

2. We will try to provide the measurements tomorrow.

3. There is no ways, because the dyn. module of the Nemesida WAF does not use his own log, only NGINX *.error.log (but you can customise it for every virtual host)

4. This option will appear in the next releases, stay tuned.
 

gnusys

Member
Thank you for the update

1. It is reassuring that you do not have plans to revoke the free status of the module . It gives me the confidence to advertise or push the module as part of the Nginx build I am offering

2. You could check https://github.com/SpiderLabs/ModSecurity/issues/1663 . Generally, the reload/restart/configtest time is proportional to the number of vhost loaded in Nginx and the Nginx memory usage is also directly proportional, for example on servers with 4000 vhost , the normal memory usage would be around 10GB (1 master+ single worker). This comes to around 15 GB of memory usage on a graceful reload as Nginx spawns a new worker process and gracefully shutdown the old. I will try to get some measurement done on my end too
 

support

Member
Staff member
The table shows the time in milliseconds.
systemctl start nginx​
nginx -t​
systemctl reload nginx​
With Nemesida WAF​
~ 1000 VHOSTS​
2283​
2185​
17​
3 VHOSTS​
381​
380​
12​
Without Nemesida WAF​
~ 1000 VHOST​
53​
10​
13​
3 VHOSTS​
35​
11​
12​
 

gnusys

Member
Hi, there is something wrong with the test you did I believe. I just tried Nemesida on a server with 800 vhost which has multiple Included files and

without nemesida


Code:
# time nginx -t



nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

nginx: configuration file /etc/nginx/nginx.conf test is successful



real    0m12.311s

user    0m10.934s

sys     0m1.320s

But as soon as I enable Nemesida , the nginx -t gets stuck forever( probably it might complete in 10-15 minutes..but make it highly unusable)

If you can provide me with your private email address I would be glad to provide a zip of the sites-enabled folder so you can check this on your end

Ref: https://serverfault.com/questions/9...nfig-files-very-slow-to-reload-nginx-s-reload
 

gnusys

Member
Just as an overview the server has around 10k include files, but without Nemesida it reloads Nginx in around 12 seconds

Code:
sites-enabled]# grep "include" *|wc -l
9225
 

gnusys

Member
To further provide some input. I did a strace nginx -t and here is where it slows down

Code:
open("/sys/class/dmi/id/product_uuid", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0400, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f64b7939000
read(5, "39DACD2A-F8D3-4EDE-A14C-67854503"..., 4096) = 37
close(5)                                = 0
munmap(0x7f64b7939000, 4096)            = 0
gettid()                                = 21890
write(3, "2020/07/11 08:35:48 [info] 21890"..., 227) = 227
open("/sys/class/dmi/id/product_uuid", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0400, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f64b7939000
read(5, "39DACD2A-F8D3-4EDE-A14C-67854503"..., 4096) = 37
close(5)                                = 0
munmap(0x7f64b7939000, 4096)            = 0
gettid()                                = 21890
write(3, "2020/07/11 08:35:48 [info] 21890"..., 107) = 107

######################################################

AFTER THIS POINT THERE IS A HUGE DELAY FOR ANY OUTPUT


#######################################################

chown("/etc/nginx/nwaf/conf/global/", 99, 99) = 0
chown("/etc/nginx/nwaf/conf/global/nwaf.conf", 99, 99) = 0
brk(NULL)                               = 0x4df1000
brk(0x4e8e000)                          = 0x4e8e000
brk(NULL)                               = 0x4e8e000
brk(0x4f0e000)                          = 0x4f0e000
brk(NULL)                               = 0x4f0e000
brk(0x4f8e000)                          = 0x4f8e000
brk(NULL)                               = 0x4f8e000
brk(0x4ff0000)                          = 0x4ff0000
brk(NULL)                               = 0x4ff0000
brk(0x5051000)                          = 0x5051000
brk(NULL)                               = 0x5051000
brk(0x50b3000)                          = 0x50b3000
mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f64af7fd000
brk(NULL)                               = 0x50b3000
brk(0x50da000)                          = 0x50da000
munmap(0x7f64af7fd000, 4198400)         = 0
brk(NULL)                               = 0x50da000
brk(0x54f3000)                          = 0x54f3000
brk(NULL)                               = 0x54f3000
brk(0x551e000)                          = 0x551e000
close(4)                                = 0
write(2, "nginx: the configuration file /e"..., 65nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
) = 65
 

gnusys

Member
So I think the delay is happening when Nginx allocate memory for the Nemesida WAF ( its the same issue with mod_sec v3 too )
 

support

Member
Staff member
Thank you for the update

1. It is reassuring that you do not have plans to revoke the free status of the module . It gives me the confidence to advertise or push the module as part of the Nginx build I am offering

2. You could check https://github.com/SpiderLabs/ModSecurity/issues/1663 . Generally, the reload/restart/configtest time is proportional to the number of vhost loaded in Nginx and the Nginx memory usage is also directly proportional, for example on servers with 4000 vhost , the normal memory usage would be around 10GB (1 master+ single worker). This comes to around 15 GB of memory usage on a graceful reload as Nginx spawns a new worker process and gracefully shutdown the old. I will try to get some measurement done on my end too
Hello,
you can send your test configs to info@pentestit.ru.
 

rr

Administrator
I am a developer providing an Nginx plugin to the popular cPanel control panel and has been providing Nemesida( free) as an option along with Mod_sec3 for installation along with the Nginx RPM's I provide. I had a couple of questions

1. Do you have any plans to revoke the free module in future? I am asking this because I know that monetization is a requirement for any company and its fair too.
Also since its not a fully Open Source project, I am sure you alone is putting a lot of effort in this and unless it gets rewarded it is not sustainable

2. Since my plugin is mostly used in the web hosting industry, it is common to have servers with more than 4000 vhost .Nginx config test, reload,restart etc slows down a lot with such high number of vhost and mod_sec3 cannot even be used with vhost numbers higher than 20-30 .I have seen Nemesida is better, but enabling Nemesida slows down Nginx reload/restart/config test etc, so far it is acceptable upto 30-40 vhosts and I havent been able to test this on anything beyond as the servers are all production and cant compromise on uptime. Have you considered the impact of loading the plugin on such high number of configs ( lots of config files to parse )

3. Is there a way I can have the nwaf log be written to a custom log file and not the standard nginx error log?

4. Is there a way to have nwaf off by default and can be turned up on a per vhost level in the server {} context
We just released the Nemesida WAF 4.2.294 with the nwaf_host_enable parameter for activating the Nemesida WAF functionality for the listed virtual hosts.
 

gnusys

Member
Just checked the feature and I put

nwaf_host_enable domain1.com; # domain1.com and domain2.com hosted on the same server

in the http_context and tested


and it throws a 403 error and the request is blocked by the WAF

Wasnt the waf supposed to be active only on domain1.com with the above setting and not block request to domain2.com?
#######
Also, let me know if multiple declarations work like for example( it doesn't throw an error)

nwaf_host_enable dom1.com;
nwaf_host_enable dom2.com;
nwaf_host_enable dom3.com;

Or do all the domains that need waf active be declared in a single nwaf_host_enable separated by comma( which is highly non-practical with a server hosting a lot of vhost)
 

support

Member
Staff member
Hello,
we tried to reproduce your problem with the "nwaf_host_enable" parameter. This option works correctly, according to the documentation. You can use it repeatedly.
 

gnusys

Member
Yes sorry, I tested again and it works fine. Multiple declarations work too

Did you get a chance to dig deeper into the reload/restart time with a high number of vhosts? I tested the cpanel server again with stock Nginx rpm's from nginx.org ( since my build had some extra modules and I wanted to rule out them ) and the stock nginx rpm also has the time issue and as I already mentioned mod_sec v3 has the same issue, so it might have to do something with loading all the rules.bin data into each vhost and not as a common one.
 

rr

Administrator
Yes sorry, I tested again and it works fine. Multiple declarations work too

Did you get a chance to dig deeper into the reload/restart time with a high number of vhosts? I tested the cpanel server again with stock Nginx rpm's from nginx.org ( since my build had some extra modules and I wanted to rule out them ) and the stock nginx rpm also has the time issue and as I already mentioned mod_sec v3 has the same issue, so it might have to do something with loading all the rules.bin data into each vhost and not as a common one.
Long server reload/restart with the many virtual hosts problem solved in Nemesida WAF 4.2.458. Please, check it.
 

gnusys

Member
Long server reload/restart with the many virtual hosts problem solved in Nemesida WAF 4.2.458. Please, check it.
I can confirm it is fixed. I just provisioned a small Cpanel server and filled it with 1500 vhost and tested and nwaf loading doesn't take any extra time

####
# time nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

real 0m10.423s
user 0m5.181s
sys 0m5.242s

####

Out of Academic interest, what was the issue and how did you fix it? Possibly the same fix can be adopted by other WAF designers too ( like I mentioned mod_sec3 has the same issue). I know its a closed product, so if you do not want to provide more details, it's perfectly fine
 

rr

Administrator
I can confirm it is fixed. I just provisioned a small Cpanel server and filled it with 1500 vhost and tested and nwaf loading doesn't take any extra time

####
# time nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

real 0m10.423s
user 0m5.181s
sys 0m5.242s

####

Out of Academic interest, what was the issue and how did you fix it? Possibly the same fix can be adopted by other WAF designers too ( like I mentioned mod_sec3 has the same issue). I know its a closed product, so if you do not want to provide more details, it's perfectly fine
Thanks for your feedback. The problem was in the procedure for generating a list of virtual hosts to create a unique list that we use when interacting with the machine learning module.
 
Top