Max processes 1024 limits.conf

We where having an issue with replica sets on our MongoDB databases changing primaries. The logs shows it was hitting resource limits creating new processes.

[initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
[initandlisten] can’t create new thread, closing connection

There is a bug fix in the init.d available per https://jira.mongodb.org/browse/SERVER-6443 to increase the ulimit -u however while working through increasing the available max processes on RHEL I was noting the soft limit would not increase past 1024 when checking /proc/pid/limits even when being increased in the startup script.

Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1024 256662 processes
Max open files 8192 8192 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 256662 256662 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

It turns out the pam package provides a file /etc/security/limits.d/90-nproc.conf which explicitly overrides the settings in security.conf for the number of processes,

# Default limit for number of user’s processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

Changing this allowed the number of processes to be increased.