Regarding nwaf free module

G

gnusys

Active member
I am a developer providing an Nginx plugin to the popular cPanel control panel and has been providing Nemesida( free) as an option along with Mod_sec3 for installation along with the Nginx RPM's I provide. I had a couple of questions

1. Do you have any plans to revoke the free module in future? I am asking this because I know that monetization is a requirement for any company and its fair too.
Also since its not a fully Open Source project, I am sure you alone is putting a lot of effort in this and unless it gets rewarded it is not sustainable

2. Since my plugin is mostly used in the web hosting industry, it is common to have servers with more than 4000 vhost .Nginx config test, reload,restart etc slows down a lot with such high number of vhost and mod_sec3 cannot even be used with vhost numbers higher than 20-30 .I have seen Nemesida is better, but enabling Nemesida slows down Nginx reload/restart/config test etc, so far it is acceptable upto 30-40 vhosts and I havent been able to test this on anything beyond as the servers are all production and cant compromise on uptime. Have you considered the impact of loading the plugin on such high number of configs ( lots of config files to parse )

3. Is there a way I can have the nwaf log be written to a custom log file and not the standard nginx error log?

4. Is there a way to have nwaf off by default and can be turned up on a per vhost level in the server {} context
 
Hello,

1. No, we do not have a plan to revoke the Nemesida WAF Free. But this is part of a commercial product, so we do not want to disclose the source code in order to avoid illegal use of the full version.

2. We will try to provide the measurements tomorrow.

3. There is no ways, because the dyn. module of the Nemesida WAF does not use his own log, only NGINX *.error.log (but you can customise it for every virtual host)

4. This option will appear in the next releases, stay tuned.
 
Thank you for the update

1. It is reassuring that you do not have plans to revoke the free status of the module . It gives me the confidence to advertise or push the module as part of the Nginx build I am offering

2. You could check https://github.com/SpiderLabs/ModSecurity/issues/1663 . Generally, the reload/restart/configtest time is proportional to the number of vhost loaded in Nginx and the Nginx memory usage is also directly proportional, for example on servers with 4000 vhost , the normal memory usage would be around 10GB (1 master+ single worker). This comes to around 15 GB of memory usage on a graceful reload as Nginx spawns a new worker process and gracefully shutdown the old. I will try to get some measurement done on my end too
 
The table shows the time in milliseconds.
​
systemctl start nginx​
nginx -t​
systemctl reload nginx​
With Nemesida WAF​
~ 1000 VHOSTS​
2283​
2185​
17​
3 VHOSTS​
381​
380​
12​
Without Nemesida WAF​
~ 1000 VHOST​
53​
10​
13​
3 VHOSTS​
35​
11​
12​
 
Hi, there is something wrong with the test you did I believe. I just tried Nemesida on a server with 800 vhost which has multiple Included files and

without nemesida


Code: 
# time nginx -t



nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

nginx: configuration file /etc/nginx/nginx.conf test is successful



real    0m12.311s

user    0m10.934s

sys     0m1.320s


But as soon as I enable Nemesida , the nginx -t gets stuck forever( probably it might complete in 10-15 minutes..but make it highly unusable)

If you can provide me with your private email address I would be glad to provide a zip of the sites-enabled folder so you can check this on your end

Ref: https://serverfault.com/questions/9...nfig-files-very-slow-to-reload-nginx-s-reload
 
Just as an overview the server has around 10k include files, but without Nemesida it reloads Nginx in around 12 seconds

Code: 
sites-enabled]# grep "include" *|wc -l
9225
 
To further provide some input. I did a strace nginx -t and here is where it slows down

Code: 
open("/sys/class/dmi/id/product_uuid", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0400, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f64b7939000
read(5, "39DACD2A-F8D3-4EDE-A14C-67854503"..., 4096) = 37
close(5)                                = 0
munmap(0x7f64b7939000, 4096)            = 0
gettid()                                = 21890
write(3, "2020/07/11 08:35:48 [info] 21890"..., 227) = 227
open("/sys/class/dmi/id/product_uuid", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0400, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f64b7939000
read(5, "39DACD2A-F8D3-4EDE-A14C-67854503"..., 4096) = 37
close(5)                                = 0
munmap(0x7f64b7939000, 4096)            = 0
gettid()                                = 21890
write(3, "2020/07/11 08:35:48 [info] 21890"..., 107) = 107

######################################################

AFTER THIS POINT THERE IS A HUGE DELAY FOR ANY OUTPUT


#######################################################

chown("/etc/nginx/nwaf/conf/global/", 99, 99) = 0
chown("/etc/nginx/nwaf/conf/global/nwaf.conf", 99, 99) = 0
brk(NULL)                               = 0x4df1000
brk(0x4e8e000)                          = 0x4e8e000
brk(NULL)                               = 0x4e8e000
brk(0x4f0e000)                          = 0x4f0e000
brk(NULL)                               = 0x4f0e000
brk(0x4f8e000)                          = 0x4f8e000
brk(NULL)                               = 0x4f8e000
brk(0x4ff0000)                          = 0x4ff0000
brk(NULL)                               = 0x4ff0000
brk(0x5051000)                          = 0x5051000
brk(NULL)                               = 0x5051000
brk(0x50b3000)                          = 0x50b3000
mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f64af7fd000
brk(NULL)                               = 0x50b3000
brk(0x50da000)                          = 0x50da000
munmap(0x7f64af7fd000, 4198400)         = 0
brk(NULL)                               = 0x50da000
brk(0x54f3000)                          = 0x54f3000
brk(NULL)                               = 0x54f3000
brk(0x551e000)                          = 0x551e000
close(4)                                = 0
write(2, "nginx: the configuration file /e"..., 65nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
) = 65
 
So I think the delay is happening when Nginx allocate memory for the Nemesida WAF ( its the same issue with mod_sec v3 too )
 
Here is a perf record and perf report for nginx -t

tug4jR9.png
 
gnusys said:
Thank you for the update

1. It is reassuring that you do not have plans to revoke the free status of the module . It gives me the confidence to advertise or push the module as part of the Nginx build I am offering

2. You could check https://github.com/SpiderLabs/ModSecurity/issues/1663 . Generally, the reload/restart/configtest time is proportional to the number of vhost loaded in Nginx and the Nginx memory usage is also directly proportional, for example on servers with 4000 vhost , the normal memory usage would be around 10GB (1 master+ single worker). This comes to around 15 GB of memory usage on a graceful reload as Nginx spawns a new worker process and gracefully shutdown the old. I will try to get some measurement done on my end too
Click to expand...
Hello,
you can send your test configs to info@pentestit.ru.
 
gnusys said:
I am a developer providing an Nginx plugin to the popular cPanel control panel and has been providing Nemesida( free) as an option along with Mod_sec3 for installation along with the Nginx RPM's I provide. I had a couple of questions

1. Do you have any plans to revoke the free module in future? I am asking this because I know that monetization is a requirement for any company and its fair too.
Also since its not a fully Open Source project, I am sure you alone is putting a lot of effort in this and unless it gets rewarded it is not sustainable

2. Since my plugin is mostly used in the web hosting industry, it is common to have servers with more than 4000 vhost .Nginx config test, reload,restart etc slows down a lot with such high number of vhost and mod_sec3 cannot even be used with vhost numbers higher than 20-30 .I have seen Nemesida is better, but enabling Nemesida slows down Nginx reload/restart/config test etc, so far it is acceptable upto 30-40 vhosts and I havent been able to test this on anything beyond as the servers are all production and cant compromise on uptime. Have you considered the impact of loading the plugin on such high number of configs ( lots of config files to parse )

3. Is there a way I can have the nwaf log be written to a custom log file and not the standard nginx error log?

4. Is there a way to have nwaf off by default and can be turned up on a per vhost level in the server {} context
Click to expand...

We just released the Nemesida WAF 4.2.294 with the nwaf_host_enable parameter for activating the Nemesida WAF functionality for the listed virtual hosts.
 
Just checked the feature and I put

nwaf_host_enable domain1.com; # domain1.com and domain2.com hosted on the same server

in the http_context and tested


and it throws a 403 error and the request is blocked by the WAF

Wasnt the waf supposed to be active only on domain1.com with the above setting and not block request to domain2.com?
#######
Also, let me know if multiple declarations work like for example( it doesn't throw an error)

nwaf_host_enable dom1.com;
nwaf_host_enable dom2.com;
nwaf_host_enable dom3.com;

Or do all the domains that need waf active be declared in a single nwaf_host_enable separated by comma( which is highly non-practical with a server hosting a lot of vhost)
 
Hello,
we tried to reproduce your problem with the "nwaf_host_enable" parameter. This option works correctly, according to the documentation. You can use it repeatedly.
 
Yes sorry, I tested again and it works fine. Multiple declarations work too

Did you get a chance to dig deeper into the reload/restart time with a high number of vhosts? I tested the cpanel server again with stock Nginx rpm's from nginx.org ( since my build had some extra modules and I wanted to rule out them ) and the stock nginx rpm also has the time issue and as I already mentioned mod_sec v3 has the same issue, so it might have to do something with loading all the rules.bin data into each vhost and not as a common one.
 
gnusys said:
Yes sorry, I tested again and it works fine. Multiple declarations work too

Did you get a chance to dig deeper into the reload/restart time with a high number of vhosts? I tested the cpanel server again with stock Nginx rpm's from nginx.org ( since my build had some extra modules and I wanted to rule out them ) and the stock nginx rpm also has the time issue and as I already mentioned mod_sec v3 has the same issue, so it might have to do something with loading all the rules.bin data into each vhost and not as a common one.
Click to expand...

Long server reload/restart with the many virtual hosts problem solved in Nemesida WAF 4.2.458. Please, check it.
 
rr said:
Long server reload/restart with the many virtual hosts problem solved in Nemesida WAF 4.2.458. Please, check it.
Click to expand...
I can confirm it is fixed. I just provisioned a small Cpanel server and filled it with 1500 vhost and tested and nwaf loading doesn't take any extra time

####
# time nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

real 0m10.423s
user 0m5.181s
sys 0m5.242s

####

Out of Academic interest, what was the issue and how did you fix it? Possibly the same fix can be adopted by other WAF designers too ( like I mentioned mod_sec3 has the same issue). I know its a closed product, so if you do not want to provide more details, it's perfectly fine
 
gnusys said:
I can confirm it is fixed. I just provisioned a small Cpanel server and filled it with 1500 vhost and tested and nwaf loading doesn't take any extra time

####
# time nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

real 0m10.423s
user 0m5.181s
sys 0m5.242s

####

Out of Academic interest, what was the issue and how did you fix it? Possibly the same fix can be adopted by other WAF designers too ( like I mentioned mod_sec3 has the same issue). I know its a closed product, so if you do not want to provide more details, it's perfectly fine
Click to expand...

Thanks for your feedback. The problem was in the procedure for generating a list of virtual hosts to create a unique list that we use when interacting with the machine learning module.
 
You must log in or register to reply here.
Back
Top