Allow Me To Save You Some Time: Some Useful Shortcuts

We aren’t machines, we are not supposed to keep everything inside of our heads; that means we often forget things, that’s why we need to better organize the things we need.

In this article, I’ll share my collection of shortcuts I gathered in past few years in order to save time, and now … yours! I hope you’ll find it useful.

Backup all MySQL Databases to individual files

Code:

fordb in $(mysql -e ‘show databases’ -s –skip-column-names); do mysqldump $db | gzip> “/backups/mysqldump-$(hostname)-$db-$(date +%Y-%m-%d-%H.%M.%S).gz”; done

Copy a MySQL Database to a new Server via ssh with one command

Code:

mysqldump –add-drop-table –extended-insert –force –log-error=error.log -uUSER -pPASS OLD_DB_NAME | ssh -C user@newhost “mysql -uUSER -pPASS NEW_DB_NAME”

Run complex remote shell cmd over ssh, without escaping quotes

Code:

ssh host -l user $(<cmd.txt)

Check site ssl certificate date

Code:

echo | openssls_client-connect Google 2>/dev/null |openssl x509 -dates -noout

Synchronize date and time with a server over ssh

Code:

date –set=”$(sshuser@server date)”

Duplicate installed packages from one machine to the other (RPM-based systems)

Code:

sshroot@remote.host “rpm -qa” | xargs yum -y install

Find SUID & SGID files

Code:

# find / ( -perm -4000 -o -perm -2000 ) -print

Find open ports

Code:

# netstat –listen

To display open ports and established TCP connections

Code:

# netstat -vatn

To display only open UDP ports try the following command

Code:

# netstat -vaun

# netstat -tulpn

If you want to see FQDN (full dns hostname), try removing the -n flag

Code:

$ netstat -vat

To display all open IPv4 network files in use by the process whose PID is 9255

Code:

# lsof -i 4 -a -p 9255

Lsof command examples

Code:

lsof -i :portNumber

lsof -itcp:portNumber

lsof -iudp:portNumber

lsof -i :80

lsof -i :80 | grep LISTEN

Sample outputs:

Code:

apache2 1607 root 3u IPv4 6472 0t0 TCP *:www (LISTEN)

apache2 1616 www-data 3u IPv4 6472 0t0 TCP *:www (LISTEN)

apache2 1617 www-data 3u IPv4 6472 0t0 TCP *:www (LISTEN)

apache2 1618 www-data 3u IPv4 6472 0t0 TCP *:www (LISTEN)

apache2 1619 www-data 3u IPv4 6472 0t0 TCP *:www (LISTEN)

apache2 1620 www-data 3u IPv4 6472 0t0 TCP *:www (LISTEN)

Find out the processes PID that opened TCPport **22**

Code:

# fuser22/tcp

Comparison of CPU utilization

The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. The accounting system, based on the values in the count and interval parameters. For example, to display the comparison of CPU utilization, 2 seconds apart and 5 times, use:

Code:

# sar -u 2 5

Output

Code:

Linux 2.6.18-53.el5 (slv0451i) 01/25/2011

09:55:02 AM CPU %user %nice %system %iowait %steal %idle

09:55:04 AM all 16.75 0.00 83.25 0.00 0.00 0.00

09:55:06 AM all 20.45 0.00 79.55 0.00 0.00 0.00

09:55:08 AM all 24.31 0.00 75.69 0.00 0.00 0.00

09:55:10 AM all 19.95 0.00 80.05 0.00 0.00 0.00

09:55:12 AM all 21.00 0.00 79.00 0.00 0.00 0.00

Average: all 20.49 0.00 79.51 0.00 0.00 0.00

Where

-u 12 5: Report CPU utilization. The following values are displayed:

o%user: Percentage of CPU utilization that occurred while executing at the user level (application).

o %nice: Percentage of CPU utilization that occurred while executing at the user level with nice priority.

o%system: Percentage of CPU utilization that occurred while executing at the system level (kernel).

o%iowait: Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.

o%idle: Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

To get multiple samples and multiple reports, set an output file for the sar command. Run the sar command as a background process using the following code:

Code:

#sar -o output.file 12 8 >/dev/null 2>&1 &

You should alsouse the nohup command so that you can logout and check back report later on:

Code:

# nohupsar -o output.file 12 8 >/dev/null 2>&1 &

All the data is captured in binary form and saved to a file (data.file). The data can then be selectively displayed by using the sar command with the -f option.

Code:

# sar -f data.file

Find CPU usage with mpstat (package sysstat)

Code:

# mpstat

Output

Code:

Linux 2.6.18-53.el5 (slv0451i) 01/25/2011

09:49:21 AM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s

09:49:21 AM all 11.33 0.00 14.72 0.54 0.08 0.22 0.00 73.11 1184.08

The mpstat command display activities for each available processor; processor 0 being the first one. Global average activities among all processors are also reported.

The mpstat command can be used on both, SMP and UP machines, but in the latter, only global average activities will be printed:

Code:

# mpstat -P ALL

Output

Code:

Linux 2.6.18-53.el5 (slv0451i) 01/25/2011

09:52:14 AM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s

09:52:14 AM all 11.33 0.00 14.73 0.54 0.08 0.22 0.00 73.10 1184.08

09:52:14 AM 0 13.15 0.00 14.71 0.65 0.12 0.31 0.00 71.06 1136.45

09:52:14 AM 1 9.50 0.00 14.74 0.43 0.05 0.14 0.00 75.11 47.63

Create an ISO image

Code:

# dd if=/dev/cdrom of=/file.iso

Clone

Code:

$ find . -path ./mnt -prune -o -path ./lost+found -prune -o -path ./sys -prune -o -path ./proc -prune -o -print |

cpio -pumd /destination &&mkdir /destination/mnt/ &&mkdir /destination/proc&&mkdir /destination/sys

Clone the current directory into /destination verboselyC

Code:

$ find . | cpio -pumdv /destination

Bash function to decompress archives

Code:

extract () {

if [ -f $1 ] ; then

case $1 in

*.tar.bz2) tar xvjf $1 ;;

*.tar.gz) tar xvzf $1 ;;

*.bz2) bunzip2 $1 ;;

*.rar) unrar x $1 ;;

*.gz) gunzip $1 ;;

*.tar) tar xvf $1 ;;

*.tbz2) tar xvjf $1 ;;

*.tgz) tar xvzf $1 ;;

*.zip) unzip $1 ;;

*.Z) uncompress $1 ;;

*.7z) 7z x $1 ;;

*) echo “‘$1’ cannot be extracted via >extract

esac

else

echo “‘$1’ is not a valid file”

fi

}

Monitor open TCP connections

Code:

$ watch -n 1 “netstat -tpanl | grep ESTABLISHED”

OR

$ lsof -i -T -n

Transfer files with netcat

-On client machine

Code:

$ nc -lpvv port > file

-On server machine

Code:

$ nc -vvclientip port < file

Example

Code:

Client: $ nc -l -p 6868 > file.txt

Server: $ nc 31.41.59.26 6868 < file.txt

Moving large directories

Code:

$ tar -C /usr -cf – home | tar -C /storage/export -xvf –

Or, to copy to a remote machine

Code:

$ tar -C /usr -cf – home | sshuser@somemachine tar -C /storage/export -xvf –

Get an ordered list of subdirectory sizes

Code:

$ du -sk ./* | sort -n | awk ‘BEGIN{ pref[1]=”K”; pref[2]=”M”; pref[3]=”G”;}

{ total = total + $1; x = $1; y = 1; while( x > 1024 ) { x = (x + 1023)/1024; y++; }

printf(“%g%st%sn”,int(x*10)/10,pref[y],$2); } END { y = 1; while( total > 1024 )

{ total = (total + 1023)/1024; y++; } printf(“Total: %g%sn”,int(total*10)/10,pref[y]); }’

List files opened by a PID

Code:

$ lsof -p 15857

Viewing Top Processes according to cpu, mem, swap size, etc…

Code:

$ pswwo pid,user,group,vsize:8,size:8,sz:6,rss:6,pmem:7,pcpu:7,time:7,wchan,sched=,stat,flags,comm,args k -vsz -A|sed -u ‘/^ *PID/d;10q’

List all files opened by a particular command

Code:

$ lsof -c dhcpd

Run a file system check on your next boot

Code:

$ touch /forcefsck

Show system installation date

Code:

$ tune2fs -l $(df -P / | tail -n1 | cut -d’ ‘ -f1 ) | grep ‘Filesystem created:

Turn off auto hard disc boot scanning for ext3

Code:

$ tune2fs -c -1 -i 0 /dev/VG0/data

Don’t spam root; log your cronjob output to syslog

Code:

$ */5 * * * * root /usr/local/nagios/sbin/nsca_check_disk 2>&1 |/usr/bin/logger -t nsca_check_disk

Create random numbers within range for conjob usage

Code:

$ H=$(until ([ $i -le 6 -a $i -gt 0 -o $i -le 23 -a $i -gt 21 ] ); do i=$(date +%N |

cut -c8-10); done ; echo $i) ; M=$(until [ $i -le 59 ]; do i=$(date +%N |

cut -c8-10); done ; echo $i) ; echo $M $H * * * backup-rsync-push.sh

List the CPU model name

Code:

$ grep ‘model|MHz’ /proc/cpuinfo |tail -n 2

Turn off all services in specific runlevel

Code:

$ fori in $(chkconfig –list | grep “4:on” | awk {‘print $1’}); do chkconfig –level 4 “$i” off; done

Find the device when you only know the mount point

Code:

$ df /media/mountpoint |egrep -o ‘^[/a-z0-9]*’

OR

$ df | grep -w ‘/media/armadillo’ | cut -d ” ” -f 1

Display any UDO/TCP connections by process name or by process id

Code:

$ lsof -nP -c COMMAND | egrep -o ‘(TCP|UDP).*$’ | sort -u

$ lsof -c apache2 | egrep -o ‘TCP.*ESTABLISHED.*$’

-nP flags are optional and UDP is irrelevant for established connections

Similar but using the process id

Code:

$ lsof -nP -p PID | egrep -o ‘(TCP|UDP).*$’

Detect your computer’s harddisk read speed without disk cache speed

Code:

$ cat /dev/sda | pv -r > /dev/null

Check the reserved block percentage of an Ext2/3 filesystem

You are probably aware that some percent of disk space on an ext2/ext3 file system is reserved for root (typically 5%). As documented elsewhere, this can be reduced to 1% with the following:

Code:

$ dumpe2fs -h /dev/sdX

$ tune2fs -m 1 /dev/sdX (where X = drive/partition, like /dev/sda1)

$ dumpe2fs -h /dev/sdX

Rsync directory tree including only files that match a certain find result

‘-mtime -10’ syncs only files newer than 10 days (-mtime is just one example, use whatever find expressions you need)

printf %P: File’s name with the name of the command line argument under which it was found removed.

This way, you can use any src directory; no need to cd into your src directory first.

Using in printf and a corresponding –from0 in rsync ensures that even filenames with newline characters work (thanks syssyphus for #3808).

Both, #1481 and #3808 just work if you either copy the current directory (.), or the filesystem root (/); otherwise, the output from find and the source dir from rsync just don’t match. #7685 works with an arbitrary source directory.

Code:

$ find /src/dir/ -mtime -10 -printf %P|rsync –files-from=- –from0 /src/dir/ /dst/dir/

Find the biggest file in current folder

Code:

$ ls -S|head -1

Find files that contain foo, but not bar

Code:

$ grep -l foo *cl*.log | xargsgrep -lL bar

Run local bash script on a remote server

Code:

$ ssh -T user@server< script.sh

Create a copy of the bootload and partition table

Code:

$ dd if=/dev/sda of=/home/sam/MBR.imagebs=512 count=1

Find files in multiple TAR files

Code:

$ find . -type f -name “*.tar” -printf [%f]n -exec tar -tf {} ; | grep -iE “[[]|”

Create a tar of directory structure only

Code:

$ tar -cf ~/out.tar –no-recursion –files-from

Backup a file with a date-time stamp

Code:

$ buf() { cp -v $1 ${1/${1%%.*}/$f-$(date +”%Y%m%d_%H%M%S”)};}

Use them wisely. There will be more to come!