Linux performance problems and some helpful commands

 IT, Linux  Kommentare deaktiviert für Linux performance problems and some helpful commands
Apr 112017
 

On one of our RHEL systems we had an issue with high load. In some cases our java application crashed.

The following command helped us to find the reason (in our case it was the limit of processes per user).

vmstat -SM -t 10 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ ---timestamp---
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0  12448    562  36934    0    0     0     3    3    2  0  0 100  0  0       2017-04-11 17:20:38 CEST
 0  0      0  12448    562  36934    0    0     0     4 1059 1975  0  0 100  0  0       2017-04-11 17:20:48 CEST
 0  0      0  12448    562  36934    0    0     0     3 1158 2038  0  0 100  0  0       2017-04-11 17:20:58 CEST

with ulimit we could see, that the „max user processes“ was set to a very low level. That was the reason why our application crashed (java could not open addidtional threads).

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 257569
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1000000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1200
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

 

# show load, currently running threas/process and ?
cat /proc/loadavg
0.00 0.00 0.00 1/1570 14415

 

# with this command you can count the number of processes (limited by "max user process" of ulimits command)
ps -elfT | grep philipp | wc -l

Remove fingerprint for RSA Key for SSH connections (remove single line from file via terminal)

 Linux  Kommentare deaktiviert für Remove fingerprint for RSA Key for SSH connections (remove single line from file via terminal)
Jan 252017
 

Sometimes host key verification fails because the host was replaced, IP has changed or something like this (also it could be a man in the middle attack).

In my case I work with high availability clusters and we often have the case that there is switch of the host. To avoid manual editing of the .ssh/known_hosts file I have written a simple bashrc alias / function.

#simple function to remove a line from .ssh/known_hosts
cleanup_known_hosts(){
    if [ x$1 == x ]; then
        echo "Syntax : cleanup_known_hosts Linenumber"
    else
        \cp ~/.ssh/known_hosts ~/.ssh/known_hosts_backup.$$
        sed $1'd' ~/.ssh/known_hosts > /tmp/known_hosts_temp.$$ && \cp /tmp/known_hosts_temp.$$ ~/.ssh/known_hosts
        rm -f /tmp/known_hosts_temp.$$
        echo "Finished"
    fi
}
alias cleanup_known_hosts=cleanup_known_hosts

With small adoptions you can use this function for all situations where you have to remove a given line in a file.

Remove last lines from file with shell, bash, csh, zsh

 IT, Linux  Kommentare deaktiviert für Remove last lines from file with shell, bash, csh, zsh
Jan 212016
 

To remove the last X lines of a file, the easiest way is to use head combined with wc -l

# get the output of wc -l (which counts the lines of a file), parse them to a number 
# and extract 3 (the number of lines you want to remove)
philipp@Host% set CMD1="expr `cat longFile.csv | wc -l` - 3"

# now remove last 3 lines by using head and redirect them to your file
philipp@Host% head -n  `$CMD1` Segment5_Contexts.csv > shortFile.csv

Converting videos with avconv on Linux

 IT, Linux  Kommentare deaktiviert für Converting videos with avconv on Linux
Jul 022014
 

The videos produces by my action cam are stored as *.mov files. These files are very large, so I wanted them to be converted to a better format (using a better container/codec). Ubuntu earlier provided a package named ffmpeg. This package was replaced by libav. Libav uses the codebase of ffmpeg. The installation of libav provides some tools for audio and video streams.

One ff these tools is avconv which converts audio and video files. To convert a file from mov to mpg4 you can use:

avconv -i myfile.mov -vcodec mpeg4 -acodec copy myfile.mp4

This will convert the video to mpeg4 and copy the audio stream. For my purpose I used the h264 codec. This codec is still under development because of some problems when using it for audio streams (also for audio streams in videos). Because I would like to remove the audio stream (the sound of my action cam is very bad) this codec is fine for me.

To convert a MOV file to mp4 using h264 codec (and removing the audio stream) you can use the following command:

avconv -i myfile.mov -vcodec h264 -an myfile.mp4

The parameter „-an“ will remove the audio stream from the video.

The file size of the generated mp4 file is round about just 1/4 of the origin mov file, but the quality is nearly the same.

Linux Script for resizing images / photos

 IT, Linux  Kommentare deaktiviert für Linux Script for resizing images / photos
Apr 112013
 

Hi there,

here a script that wl resize images (to 25% of original size) and moves the resized images o a folder called „25Prozent“:

#!/bin/bash

cd $1
echo "Starting Script for resizing Images to 25% of the origin size"
echo "Creating folder -> 25Prozent <-"
mkdir 25Prozent
echo "Folder created"
echo "*** Start resizing images ***"
for i in *.JPG;do convert $i -resize 25% -quality 95 25Prozent/$i;echo $i;done
for i in *.jpg;do convert $i -resize 25% -quality 95 25Prozent/$i;echo $i;done
echo "*** Finished resizing images ***"
echo "exit and good bye..."

Instead of

for i in *.JPG;do ....
for i in *.jpg;do ...

You can also use:

for i in $(find . -iname "*.jpg")

Thx to Timo for this information 😉