Apr 112017
 

On one of our RHEL systems we had an issue with high load. In some cases our java application crashed.

The following command helped us to find the reason (in our case it was the limit of processes per user).

vmstat -SM -t 10 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0  12448    562  36934    0    0     0     3    3    2  0  0 100  0  0       2017-04-11 17:20:38 CEST
 0  0      0  12448    562  36934    0    0     0     4 1059 1975  0  0 100  0  0       2017-04-11 17:20:48 CEST
 0  0      0  12448    562  36934    0    0     0     3 1158 2038  0  0 100  0  0       2017-04-11 17:20:58 CEST

with ulimit we could see, that the „max user processes“ was set to a very low level. That was the reason why our application crashed (java could not open addidtional threads).

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 257569
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1000000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1200
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

 

# show load, currently running threas/process and ?
cat /proc/loadavg
0.00 0.00 0.00 1/1570 14415

 

# with this command you can count the number of processes (limited by "max user process" of ulimits command)
ps -elfT | grep philipp | wc -l

Sorry, the comment form is closed at this time.