shared memory zone "nwaf_shared" was locked by

gnusys

Active member
One server with nwaf installed had nginx stalled with the following in the nginx error log

2021/05/19 12:30:48 [alert] 1042#1042: worker process 17236 exited on signal 11
2021/05/19 12:30:48 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17236
2021/05/19 12:30:48 [alert] 1042#1042: worker process 17301 exited on signal 11
2021/05/19 12:30:48 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17301
2021/05/19 12:30:48 [alert] 1042#1042: worker process 17368 exited on signal 11
2021/05/19 12:30:48 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17368
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17434 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17434
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17500 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17500
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17566 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17566
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17633 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17633
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17698 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17698
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17764 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17764
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17829 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17829
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17895 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17895
2021/05/19 12:30:49 [alert] 1042#1042: worker process 17960 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 17960
2021/05/19 12:30:49 [alert] 1042#1042: worker process 18025 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 18025
2021/05/19 12:30:49 [alert] 1042#1042: worker process 18090 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 18090
2021/05/19 12:30:49 [alert] 1042#1042: worker process 18156 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 18156
2021/05/19 12:30:49 [alert] 1042#1042: worker process 18222 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 18222
2021/05/19 12:30:49 [alert] 1042#1042: worker process 18288 exited on signal 11
2021/05/19 12:30:49 [alert] 1042#1042: shared memory zone "nwaf_shared" was locked by 18288

A restart of nginx fixed it. But why this error happened and could it be prevented?
 
Hello.
What versions of nginx and nwaf-dyn are you using on this server?
We need 20-30 lines of nginx error log file before this message if you are using single error.log file for whole server.
The request that caused the segfault will also help us. From the access log or if you send it yourself.
 
Hi,
It was running the version prior to latest. I have just updated to the latest version of nwaf module and will update this post with more details , If I find any more issues
 
I updated Nginx and nwaf to latest version

#####################################################################################
# rpm -qa|grep waf
nwaf-dyn-1.19-4.3-401.x86_64

# nginx -V
nginx version: nginx/1.19.10
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.1.1k 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/etc/nginx/modules --with-pcre=./pcre-8.44 --with-pcre-jit --with-zlib=./zlib-1.2.11 --with-openssl=./openssl-1.1.1k --with-openssl-opt=enable-tls1_3 --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log --http-log-path=/var/log/nginx/access_log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/dev/shm/client_temp --http-proxy-temp-path=/dev/shm/proxy_temp --http-fastcgi-temp-path=/dev/shm/fastcgi_temp --http-uwsgi-temp-path=/dev/shm/uwsgi_temp --http-scgi-temp-path=/dev/shm/scgi_temp --user=nobody --group=nobody --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-file-aio --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-compat --with-http_v2_module --add-dynamic-module=/usr/local/rvm/gems/ruby-2.6.6/gems/passenger-6.0.7/src/nginx_module --add-dynamic-module=echo-nginx-module-0.61 --add-dynamic-module=headers-more-nginx-module-0.32 --add-dynamic-module=ngx_http_redis-0.3.8 --add-dynamic-module=redis2-nginx-module --add-dynamic-module=srcache-nginx-module-0.31 --add-dynamic-module=ngx_devel_kit-0.3.0 --add-dynamic-module=set-misc-nginx-module-0.31 --add-dynamic-module=ngx_http_geoip2_module --add-dynamic-module=testcookie-nginx-module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --with-ld-opt=-Wl,-E
##################################################################################

Started the error_log from scratch after the upgrade and these are the only few lines in the log

worker process 12084 exited on signal 11 -- you can see it is segfaulting

The error logs are enabled on the individual server {} blocks and there is a common /var/log/nginx/error_log also defined
Interestingly this segfault is coming in the main error_log in the main context and not in the individual server {} context error_log files


2021/05/20 19:12:18 [warn] 11711#11711: conflicting server name "www" on X.X.X.208:80, ignored
2021/05/20 19:12:18 [warn] 11711#11711: conflicting server name "www" on X.X.X.208:443, ignored
2021/05/20 19:12:18 [notice] 11711#11711: signal process started
2021/05/20 19:12:53 [warn] 11953#11953: conflicting server name "www" on X.X.X.208:80, ignored
2021/05/20 19:12:53 [warn] 11953#11953: conflicting server name "www" on X.X.X.208:443, ignored
2021/05/20 19:12:53 [notice] 11953#11953: signal process started
2021/05/20 23:57:49 [alert] 7977#7977: worker process 12084 exited on signal 11
2021/05/21 00:32:29 [alert] 7977#7977: worker process 18390 exited on signal 11
2021/05/21 04:27:26 [error] 12083#12083: Nemesida WAF: IP 92.63.196.29 banned due to exceeding the blocking limit in "nwaf_limit" parameter
2021/05/21 04:27:26 [error] 12083#12083: Nemesida WAF: IP 92.63.196.29 has already been banned due to exceeding the blocking limit in "nwaf_limit" parameter
2021/05/21 04:27:27 [error] 12083#12083: Nemesida WAF: IP 92.63.196.29 has already been banned due to exceeding the blocking limit in "nwaf_limit" parameter

I am pretty sure the segfault is happening in nwaf as I manage a lot of servers with same nginx version and this server, in particular, have nwaf enabled and the segfault is happening here only
 
I updated Nginx and nwaf to latest version

#####################################################################################
# rpm -qa|grep waf
nwaf-dyn-1.19-4.3-401.x86_64

# nginx -V
nginx version: nginx/1.19.10
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
built with OpenSSL 1.1.1k 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/etc/nginx/modules --with-pcre=./pcre-8.44 --with-pcre-jit --with-zlib=./zlib-1.2.11 --with-openssl=./openssl-1.1.1k --with-openssl-opt=enable-tls1_3 --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error_log --http-log-path=/var/log/nginx/access_log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/dev/shm/client_temp --http-proxy-temp-path=/dev/shm/proxy_temp --http-fastcgi-temp-path=/dev/shm/fastcgi_temp --http-uwsgi-temp-path=/dev/shm/uwsgi_temp --http-scgi-temp-path=/dev/shm/scgi_temp --user=nobody --group=nobody --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-file-aio --with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-compat --with-http_v2_module --add-dynamic-module=/usr/local/rvm/gems/ruby-2.6.6/gems/passenger-6.0.7/src/nginx_module --add-dynamic-module=echo-nginx-module-0.61 --add-dynamic-module=headers-more-nginx-module-0.32 --add-dynamic-module=ngx_http_redis-0.3.8 --add-dynamic-module=redis2-nginx-module --add-dynamic-module=srcache-nginx-module-0.31 --add-dynamic-module=ngx_devel_kit-0.3.0 --add-dynamic-module=set-misc-nginx-module-0.31 --add-dynamic-module=ngx_http_geoip2_module --add-dynamic-module=testcookie-nginx-module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --with-ld-opt=-Wl,-E
##################################################################################

Started the error_log from scratch after the upgrade and these are the only few lines in the log

worker process 12084 exited on signal 11 -- you can see it is segfaulting

The error logs are enabled on the individual server {} blocks and there is a common /var/log/nginx/error_log also defined
Interestingly this segfault is coming in the main error_log in the main context and not in the individual server {} context error_log files


2021/05/20 19:12:18 [warn] 11711#11711: conflicting server name "www" on X.X.X.208:80, ignored
2021/05/20 19:12:18 [warn] 11711#11711: conflicting server name "www" on X.X.X.208:443, ignored
2021/05/20 19:12:18 [notice] 11711#11711: signal process started
2021/05/20 19:12:53 [warn] 11953#11953: conflicting server name "www" on X.X.X.208:80, ignored
2021/05/20 19:12:53 [warn] 11953#11953: conflicting server name "www" on X.X.X.208:443, ignored
2021/05/20 19:12:53 [notice] 11953#11953: signal process started
2021/05/20 23:57:49 [alert] 7977#7977: worker process 12084 exited on signal 11
2021/05/21 00:32:29 [alert] 7977#7977: worker process 18390 exited on signal 11
2021/05/21 04:27:26 [error] 12083#12083: Nemesida WAF: IP 92.63.196.29 banned due to exceeding the blocking limit in "nwaf_limit" parameter
2021/05/21 04:27:26 [error] 12083#12083: Nemesida WAF: IP 92.63.196.29 has already been banned due to exceeding the blocking limit in "nwaf_limit" parameter
2021/05/21 04:27:27 [error] 12083#12083: Nemesida WAF: IP 92.63.196.29 has already been banned due to exceeding the blocking limit in "nwaf_limit" parameter

I am pretty sure the segfault is happening in nwaf as I manage a lot of servers with same nginx version and this server, in particular, have nwaf enabled and the segfault is happening here only
Hello.
We need more information.
Nwaf error messages from the individual error log files at the time segfault occurs. They starts with "Nemesida WAF: ..." and will be from the worker process that created segfault.
 
Back
Top