Bash

Aus Leupers.net-Wiki
Wechseln zu: Navigation, Suche

General Tutorials


Helpful tools for bash and bash scripts

Renaming files

rename

$ rename '<regex>' <files>

with regex = e.g. s/old/new/g

mmv

$ mmv '<srcPattern>' '<dstPattern>'

with */?/[...] in srcPattern and #1/#2/#3/... in dstPattern. Parameter -n for "no-execute"; i.e. test/show only, without renaming.

Examples:

  • Bank Statement of German Bank "Commerzbank":
    Input: Wertpapier_123456789_20150704_123456.pdf (= <Type>_<AccountNo>_<Date>_<SomeID>.pdf)
    Goal: Add prefix with date. => 20150704_Wertpapier_123456789_20150704_123456.pdf
    (Note on mmv <srcPattern>: The [^0-9]* is required to rename only files that do not have the date as prefix already; here all bank statement files start with text, not numbers, before renaming.)
$ mmv -v "[^0-9]*_*_*_*.pdf" "#3_#1#2_#3_#4.pdf"
  • Bank Statement of German Bank "Consorsbank":
    Input: WICHTIGE_MITTEILUNG_dat20150626_id123456789 (date is prefixed with dat)
    Goal: Add prefix with date. => 20150626_WICHTIGE_MITTEILUNG_dat20150626_id123456789
$ mmv -v '[^0-9]*_dat*_*.pdf' '#3_#1#2_dat#3_#4.pdf'
  • In order to pretty-print the date as YYYY-MM-DD instead of YYYYMMDD the following code can be used:
for FILE in $(find -name "[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]*.pdf") ; do
   DST=$(echo $FILE | sed -r 's/([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])/\1-\2-\3/')
   mv -v ${FILE} ${DST}
done

Miscellaneous

  • BASH History Suggest Box
    Install hh on (K)Ubuntu as follows, then enjoy the new hh functionality when pressing <Ctrl-R>:
sudo add-apt-repository ppa:ultradvorka/ppa
sudo apt-get update
sudo apt-get install hh
hh --show-configuration >> ~/.bashrc
$ mount | column -t 
$ column -t -s ',' < data.csv


Miscellaneous

Unofficial Bash Strict Mode

To make bash scripts safer and reduce debugging, they should start with the following:

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
LC_ALL=C

Regarding the set command and IFS variable see link in header.
Regarding LC_ALL=C see more details below.

Scripts and LANG

Some scripts behave different depending on which user does run them. :-(
The reason is often the language settings, which are even propagated to SSH sessions.

Normally it's best to run all scripts with language C, because some commands have problems with LANG settings like de_DE or de_DE.UTF-8 and similar.

Every script should start with export LANG=C or even better:

export LC_ALL=C

LC_ALL overrides the value of the LANG environment variable and the values of any other LC_* environment variables.

Links:

Bash For Loop Examples

#!/bin/bash
 
for i in 1 2 3 4 5 ; do
   echo "Welcome $i times"
done
 
# bash v3.0+ (built-in support for ranges):
for i in {1..5} ; do
   echo "Welcome $i times"
done
 
# bash v4.0+ (built-in support for {START..END..INCREMENT} syntax):
for i in {0..10..2} ; do
   echo "Welcome $i times"
done
 
# alternative, but deprecated(!) because external tool seq is used:
for i in $(seq 1 2 20) ; do
   echo "Welcome $i times"
done
 
# C-like syntax:
for (( i=1; i<=5; i++ )) ; do
   echo "Welcome $i times"
done
 
# C-like syntax of infinite for loop (alternative: "while true ; do ... done"):
for (( ; ; )) ; do
   echo "infinite loops [ hit CTRL+C to stop]"
done
 
# Looping over files (list created by wildcard):
for file in /etc/* ; do
   echo "- ${file}"
done

Loop over File Names with Spaces

#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for f in *
do
  echo "$f"
done
IFS=$SAVEIFS

Reading text file line by line

#!/usr/bin/bash
filename="$1"
while read -r line ; do
   echo "Content: '$line'"
done < "$filename"

Notes:

  • Content of line will be trimmed - i.e. leading and trailing whitespaces are removed! - If you don't want this, you may try the following:
while IFS=''; read -r line ; do ...
  • There might be a problem reading the last line of a file if it has no trailing line feed (\n). In this case you may try the following:
while -r read line || [[ -n $line ]] ; do ...

Associative Arrays - Basics

declare -A array
array[foo]=bar
array[bar]=foo
 
# Keys:  ${!array[@]}
# Values: ${array[@]}
 
for i in "${!array[@]}"
do
  echo "key  : $i"
  echo "value: ${array[$i]}"
done

More information: Bash associative array examples @ Andy Balaam's Blog

Associative Arrays - exists() function for key checking

#-------------------------------------------------------------------------------
# Check for the existence of a key in an array.
# Works with indexed and associative arrays.
# Example:
# if   exists $key in array; then echo "$key = ${array[$key]}"; fi 
# if ! exists $key in array; then echo "no array element $key"; fi 
#-------------------------------------------------------------------------------
function exists()
{
   if [ "$2" != in ]; then
      echo "ERROR: Incorrect usage."
      echo "Usage: exists {key} in {array}"
      return
   fi   
   eval '[ ${'$3'[$1]+isset} ]'  
}
 
echo "Indexed array:"
array[2]="two"
key=1 ; if ! exists $key in array; then echo "- no array element $key"; fi 
key=2 ; if ! exists $key in array; then echo "- no array element $key"; fi 
key=1 ; if   exists $key in array; then echo "- $key = ${array[$key]}"; fi 
key=2 ; if   exists $key in array; then echo "- $key = ${array[$key]}"; fi 
echo
echo "Associative array:"
declare -A array2
array2['two']=2
key='one' ; if ! exists $key in array2; then echo "- no array element $key"; fi 
key='two' ; if ! exists $key in array2; then echo "- no array element $key"; fi 
key='one' ; if   exists $key in array2; then echo "- $key = ${array2[$key]}"; fi 
key='two' ; if   exists $key in array2; then echo - "$key = ${array2[$key]}"; fi

Output:

Indexed array:
- no array element 1
- 2 = two

Associative array:
- no array element one
- two = 2

Constructing variable names from variables

Links:

Example for scalar variable: Option 1 (uses eval => potentially unsafe)

param="MYPARAM"
 
# Set inital value with constructed variable name and value string.
# IMPORTANT: You must put double "double quotes" (escape inner quotes)!
eval CONFIG_$param="\"one one\""
echo "CONFIG_MYPARAM = $CONFIG_MYPARAM"
 
# Set/overwrite with constructed variable name and value in variable.
# IMPORTANT: Don't forget the "\" (backslash) in front of "$newvalue"!
newvalue="two two"
eval CONFIG_$param=\$newvalue
echo "CONFIG_MYPARAM = $CONFIG_MYPARAM"
 
# Print value with constructed variable name.
var=CONFIG_$param
echo "CONFIG_MYPARAM = ${!var}"

Example for scalar variable: Option 2 (uses declare => safe)

param="MYPARAM"
 
# Declare variable and initialize with value string.
declare CONFIG_$param="one one"
echo "CONFIG_MYPARAM = $CONFIG_MYPARAM"
 
# Re-declare to set new value with value in variable.
# IMPORTANT: Don't forget the quotes (") around "$newvalue"!
newvalue="two two"
declare CONFIG_$param="$newvalue"
echo "CONFIG_MYPARAM = $CONFIG_MYPARAM"
 
# Print value with constructed variable name.
var=CONFIG_$param
echo "CONFIG_MYPARAM = ${!var}"

Output of both variants:

CONFIG_MYPARAM = one one
CONFIG_MYPARAM = two two
CONFIG_MYPARAM = two two

Example for associative array:

section="SECTION1"
 
# Declare array and set inital value with value string.
# IMPORTANT: You must put double "double quotes" (escape inner quotes)!
declare -A CONFIG_$section
eval CONFIG_$section['myparam']="\"one one\""
echo "CONFIG_SECTION1['myparam'] = ${CONFIG_SECTION1['myparam']}"
 
# Set/overwrite with constructed array name and value in variable.
# IMPORTANT: Don't forget the quotes (") around "$newvalue"!
newvalue="two two"
eval CONFIG_$section['myparam']=\$newvalue
echo "CONFIG_SECTION1['myparam'] = ${CONFIG_SECTION1['myparam']}"
 
# Print array value with constructed variable name does NOT seem to work;
# non of the following works, but results in "bad substitution error". :-(
#var=CONFIG_$section
#echo "CONFIG_SECTION1['myparam'] = ${!$var['myparam']}"
#echo "CONFIG_SECTION1['myparam'] = ${${!$var}['myparam']}"
CONFIG_SECTION1['myparam'] = one one
CONFIG_SECTION1['myparam'] = two two

The printf command

Print formated:

printf "Surname: %s\nName: %s\n" "$SURNAME" "$LASTNAME"

Print to variable (-v <var>):

printf -v NUMBER "%02d" "${INPUT}"
echo $NUMBER

Extract substring in bash

Syntax:

 extract=${string:offset:length}

Example:

 string="I am Stefan."
 extract=${string:5:6}
 echo "Name = '$extract'"

Ausgabe:

 Name = 'Stefan'

Dynamische Teile, kann man oft mit cut ganz gut extrahieren.

$ echo "<name>_<timestamp>_<number>" \   # Beispiel-String erzeugen
  | cut -d"_" -f2                        # Splitten an "_" und 2. Spalte extrahieren.

Ausgabe:

<timestamp>

Alternativ ginge es auch mit einer der folgenden Varianten:
awk:

awk -F "_" '{print $2}'
  1. Trennzeichen festlegen mit -F; Spalte 2 drucken.

sed:

sed "s/^[^_]*_\([^_]*\)_.*$/\1/"
  1. String von Anfang bis Ende ersetzen: ^...$
  2. In (escapten) runden Klammern (\( und \)) eingeschlossen Teilstring als \1 merken und am Ende ausgeben.
  3. Muster, dass passen muss:
    1. Beliebig viele Zeichen (*), die nicht (^) _ sind: [^_]*
    2. Ein Unterstrich.
    3. Dann der interessante String (runde Klammern); wieder [^_]* (wie vorne).
    4. Noch ein Unterstrich (_) und alles was danach kommt (.*$)

Generating random number in Bash Shell Script

rand=$(( $RANDOM % 10 ));     echo $rand     # range 0..9
rand=$(( $RANDOM % 10 + 1 )); echo $rand     # range 1..10

Regular Expressions

echo -n "Your input> " 
read REPLY 
if [[ $REPLY =~ ^[0-9]+$ ]]; then
   echo "Numeric"
else 
   echo "Non-numeric"
fi

Trimming bash variable

Code:

trim() {
   local var=$@
   var="${var#"${var%%[![:space:]]*}"}"   # remove leading whitespace characters
   var="${var%"${var##*[![:space:]]}"}"   # remove trailing whitespace characters
   echo -n "$var"
}
 
VAR="  abc   "
echo "VAR = '$VAR'"
VAR=$(trim $VAR)
echo "VAR = '$VAR'"

Output:

VAR = '  abc   '
VAR = 'abc'

Entfernen eines Linesfeeds am Zeilenende: [Quelle: How to trim whitespace from bash variable?]

$ <command> | tr -d '\n'

Checking number of command line parameters

if [ ! "$#" -eq 1 ] ; then
    echo "ERROR: Exactly one parameter is required."
    exit 1
fi
if [ ! "$#" -gt 2 ] ; then
    echo "ERROR: At least two parameters are required."
    exit 1
fi

Include other scripts

The portable way (should also work with scripts in $PATH):

INCDIR="${BASH_SOURCE%/*}"
if [[ ! -d "$INCDIR" ]]; then INCDIR="$PWD"; fi
. "$INCDIR/include.sh"

How to execute multiple commands after xargs -0?

Example:

$ find -print0 | \
  xargs -0 -I{} sh -c 'FILE={} ; echo -n "${FILE}:" ; cat "${FILE}" | wc -l'

How to change the output color of echo in Linux

Example:

To be done...

Trap/Signal-Handling

Häufig kommt es vor, dass man irgendwelche Aktionen startet (z. B. schreiben temporärer Dateien etc.), die auf jeden Fall vor Beendigung des Skript noch aufgeräumt werden sollten/müssen.

Bei komplexeren Skripten und in Fehlerfällen gibt es meist an verschiedensten Stellen im Skript Ausstiegspunkte; und nicht nur am Ende, so dass man an jeder einzelnen Stelle dran denken muss, vorher noch einen cleanup auszuführen. Das ist lästig und fehleranfällig, weil es leicht vergessen wird.

Einfacher uns sicherer geht das über trap-Handler (d.h. Signal-Handler).

Außerdem kann man damit auch Programmabbrüche durch den User (z. B. durch Drücken von "Ctrl-C" oder Ausführen von "kill") abfangen, zunächst noch alles sauber aufgeräumen und erst dann das Skript beenden.

Beides zeigt das folgende Beispiel:

#!/bin/bash
 
function exitHandler()
{
    echo "Cleaning temporary files..."
    # delete temporary files
    # write last message to log
    # ...
}
 
function signalHandler()
{
    signal=$1
    if [[ "$signal" =~ (SIGINT|SIGTERM) ]] ; then
        exitCode=1
        echo -e "\nSignal '$signal' received. => Aborting with exit code ${exitCode}."
        exit $exitCode
    else
        echo -e "\nSignal $signal received and ignored."
    fi
}
 
trap 'signalHandler SIGINT'  SIGINT
trap 'signalHandler SIGTERM' SIGTERM
trap 'signalHandler SIGHUP'  SIGHUP
 
# Run exitHandler to cleanup, when exit is called
trap 'exitHandler' 0
 
echo "Waiting... (press Ctrl-C to terminate)"
for ((i=0; i< 10; i++)) ; do sleep 1 ; done
 
exit 0

Output:

Press "Ctrl-C":

^C
Signal 'SIGINT' received. => Aborting with exit code 1.
Cleaning temporary files...

Run "kill -SIGTERM <PID>":

Signal 'SIGTERM' received. => Aborting with exit code 1.
Cleaning temporary files...

kill -SIGHUP <PID>

Signal SIGHUP received and ignored.
Cleaning temporary files...

Natürlich lassen sich Signale wie SIGHUP, SIGUSR1 oder SIGUSR2 o.ä. auch gut nutzen, um Aktionen wie ein Reread der Config oder das Wegschreiben von Statusinformationen zu triggern.

Da das Aufräumen am Ende eines Skripts zum guten Programmierstil gehört, sollte die Verwendung von Signal-Handlern für jedes nicht-triviale bash-Skript verpflichtend sein.

BTW: Das Gleiche erreicht man in python mit: atexit und signal.

Debugging

Links:

$LINENO

The variable $LINENO contains the line number in the script or shell function currently executing.

Example:

#!/bin/bash
echo "This is line $LINENO."

Output:

This is line 2.

-x option

Print every command before executing it.
Example (myscript.sh):

#!/bin/bash -x
echo "Hello World"

Output:

+ echo "Hello World" 
Hello World 

PS4

PS4 is the prompt printed before the command line is echoed when the -x option is set. The first character of PS4 is replicated multiple times, as necessary, to indicate multiple levels of indirection. The default is ‘+ ’.

PS4='[$LINENO] ' ; bash -x myscript.sh

Output:

[2] echo "Hello World" 
Hello World

Of course it can be included into the script directly:

#!/bin/bash -x
PS4='[$LINENO] '
echo "Hello World"

Output:

+ PS4='[$LINENO] '
[3] echo "Hello World" 
Hello World

Note:

  • The "+" in front of "PS4" is printed, because the default prefix is used until PS4 is redefined.

=> PS4 variable should be redefined as early as possible in the script.

Emails per Skript über SMTP versenden

Vorbereitungen

Zunächst müssen die notwendigen Pakete installiert sein. Unter (K)Ubuntu (getestet mit 12.04.2 und 13.04) geht das ganz einfach per folgendem Befehl:

$ sudo apt-get install sendEmail libio-socket-ssl-perl libnet-ssleay-perl perl

Email-Versand

Ich verschicke über meine 1&1-Domain und möchte, dass die Übertragung per TLS verschlüsselt wird:

$ sendEmail -s smtp.1und1.de:587 -o tls=yes -xu smtpuser -xp smtppasswort -f ich@mydomain.de -t empfaenger@irgendwo.de \
  -u "Betreff: Test" -m "Hallo, dies ist ein Test." -v

Die Optionen im Einzelnen:

-v   Verbose, d. h. Ausgabe von Detailinfos; vor allem zur Fehlersuche sinnvoll.
-s   SMTP-Server-Adresse und -Port.
-o   Optinen:
     - tls=yes    Aktiviert die verschlüsselte Übertragung per TLS.
-xu  SMTP Username oder die eigene Emailadresse, mit der man sich auch einloggen kann.
-xp  SMTP Password; d. h. bei 1&1 das zum Mailaccount passende Passwort.
-f   From: Email-Adresse, d. h. die Emailadresse, die von der versandt werden soll.
-t   To: Email-Adresse, d. h. die Email-Adresse des oder der primären Empfänger(s).
-cc  CC: Email-Adresse, d. h. diese Empfänger erhalten eine Kopie der Mail;
                        sichtbar für alle. (Carbon Copy)
-bcc Bcc: Email-Adresse, d. h. diese Empfänger erhalten ebenfalls eine Kopie der Mail;
                         sind jedoch für andere unsichtbar.  (Blind Carbon Copy)
-u   Subject, d. h. die Betreff-Zeile.
-m   Message, d. h. der Inhalt der Email (Mailbody).
     Kann Text sein und z. B. "\n" für Zeilenumbrüche enthalten oder 
     kann HTML sein, muss dann aber mit "<html>" beginnen.
-a   Attachment; gibt den Pfad zu einer als Anhang mitzusendenden Datei an.

Anwendung in komplettem Skript: DynDNS - Freien Dyn.com Account verlängern

Troubleshooting

In case you get the following error when executing sendemail:

invalid SSL_version specified at /usr/local/share/perl/5.14.2/IO/Socket/SSL.pm line 332

the following workaround should help you:

# sudo nano usr/share/perl5/IO/Socket/SSL.pm

Change

     m{^(!?)(?:(SSL(?:v2|v3|v23|v2/3))|(TLSv1[12]?))$}i 
to:  m{^(!?)(?:(SSL(?:v2|v3|v23|v2/3))|(TLSv1[12]?))}i

[Source: http://raspberrypi.stackexchange.com/questions/2118/sendemail-failure]

Links