On one of our RHEL systems we had an issue with high load. In some cases our java application crashed.
The following command helped us to find the reason (in our case it was the limit of processes per user).
vmstat -SM -t 10 10 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp--- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 12448 562 36934 0 0 0 3 3 2 0 0 100 0 0 2017-04-11 17:20:38 CEST 0 0 0 12448 562 36934 0 0 0 4 1059 1975 0 0 100 0 0 2017-04-11 17:20:48 CEST 0 0 0 12448 562 36934 0 0 0 3 1158 2038 0 0 100 0 0 2017-04-11 17:20:58 CEST
with ulimit we could see, that the „max user processes“ was set to a very low level. That was the reason why our application crashed (java could not open addidtional threads).
ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 257569 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1000000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1200 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
# show load, currently running threas/process and ? cat /proc/loadavg 0.00 0.00 0.00 1/1570 14415
# with this command you can count the number of processes (limited by "max user process" of ulimits command) ps -elfT | grep philipp | wc -l