A Life Spent Making Mistakes

Couple other bash tips to help with more robust code.  The main improvement I learned from the previous link is the trap function.  This function lets you cleanup when specific system signals get sent from the OS, like INT (what gets sent to a program when Ctrl+c is typed) and the TERM signal.  A great example is:

trap “rm -f $lockfile; exit” INT TERM EXIT

In this case a lockfile is being removed just before closing a bash script.   You can get a full list of all the different system signals with the kill -l command.

The other major bash tool that I have used without ever really understanding what it did is the eval expression.  If you have ever written a sysinit configuration script, you know that you use eval to basically load/set variables from other subscripts or external files.  The reason eval does this is explained here.  The quick explanation is that eval forces bash to evaluate a second time any code reference passed to it.  So setting bash variables in-line is as easy as:

eval $(LANG=C grep -F “DEVICE=” ifcfg-$i)

One last bash builtin that may come in handy for some, the shopt function sets and displays bash built-in extended capabilities.  See the link for the man page with some examples.

Overall, I am consistently amazed at the power and flexibility of the Linux command line.

If the line is good

Found a couple links to articles that discuss Linux usage across a number of different markets.  Just citing the authors:

All of the above noted, Linux desktop usage still hovers around 1-3%

Trust the Engineer

(Update 11/20/2017: Additional link for AWS & new versions of Fedora.  Still a useful post.)

A client project along with my “Hacking & Countermeasures” class has recently necessitated a need for my own VPN for use in wireless applications. I needed to connect the VPN to my server rack and the system needed to be an “in-house” system I could turn up myself (sorry Cisco, no ASA for me.)  Finally, it needed to be an SSL based VPN solution as I have had entirely too many issues with locations filtering nonstandard Internet traffic effectively blocking IPSec VPN access on their networks.

I use Rackspace for my server infrastructure, so it only took me about 15 minutes to get the physical (errr… cloud… damn… whatever) Linux machine (Fedora 17 x64) up and running but actually setting up OpenVPN was significantly more challenging that I originally had considered.  The problem wasn’t the lack of documentation (actually the opposite was generally true.)  The problem is that VPN connectivity is so inherently picky, and there are SO many options, that getting a specific configuration running for a specific distribution can be a little overwhelming.

So, for my own personal benefit, here is some of the information I needed to get OpenVPN working on a Fedora 17 server routing http traffic as well as direct traffic to my private subnet.  OpenVPN will be configured to use port 443 (the standard web SSL port) using the TCP protocol.)  As OpenVPN uses SSL, and we will be using TCP on the HTTPS port, all the traffic will look like standard secure web traffic to the network, effectively keeping it from being filtered.

On the Server (as root):

  • Start by install openvpn and other support packages:
    • yum install openvpn pkcs11-tools pkc11-dump
  • We will use the easy-rsa script toolkit to create our shared keys.  So start by coping the example easy-rsa files into your home directory:
    • cp -ai /usr/share/openvpn/easy-rsa/2.0 ~/easy-rsa
    • cd ~/easy-rsa
  • Next you will need to edit the vars file.  Basically it is ID information for your server certificate.  The values other than the PKCS11_MODULE_PATH (which will be set to /usr/lib64/ on x64 machines) are not particularly critical but don’t leave them blank!  Mine looked something like this:

export KEY_CITY=”Norman”
export KEY_ORG=”Rockerssoft”
export KEY_EMAIL=”name@emailaddress.com”
export KEY_EMAIL=name@emailaddress.com
export KEY_CN=rockerssoft-vpn
export KEY_NAME=rockerssoft-vpn-key
export KEY_OU=rockerssoft-vpn
export PKCS11_MODULE_PATH=/usr/lib64/

  • Now we generate our server keys and setup our openvpn service directories:
    • . vars
    • ./clean-all
    • ./build-ca
    • ./build-inter $( hostname | cut -d. -f1 )
    • ./build-dh
    • mkdir /etc/openvpn/keys
  • Now with our keys built, we need to copy all of them (along with our certificates and template configuration information) into our service directory.
    • cp -ai keys/$( hostname | cut -d. -f1 ).{crt,key} keys/ca.crt keys/dh*.pem /etc/openvpn/keys/
    • cp -ai /usr/share/doc/openvpn-*/sample-config-files/roadwarrior-server.conf /etc/openvpn/server.conf
  • The config file we just copied to /etc/openvpn/server.conf will need to be edited for your specific server configuration.  If you have problems connecting later on it is most like an issue with either the server configuration file or the client configuration file not matching.  As we want the system to be a full VPN proxy for all internet traffic start by adding the following to the BOTTOM of your config file:
    • comp-lzo yes
    • push "redirect-gateway def1"
  • In /etc/openvpn/server.conf, edit the port number and add a line to have openvpn use tcp instead of udp for port 443.  This should be somewhere between line 9 and 12 and should look something like this when you are done.

port 443
proto tcp-server

  • In /etc/openvpn/server.conf, edit the cert and key file location names somewhere between line 17 and 20.  Add the full path to your key/cert files we moved two steps previous.  They should look something like this (notice the /etc/openvpn/keys preceding each entry:)

ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/bob-vpn-1.crt
key /etc/openvpn/keys/bob-vpn-1.key
dh /etc/openvpn/keys/dh1024.pem

  • After you have modified your server configuration files, you will need to tell the Linux Security subsystem (aka SELinux) to recognize the to file layout.  To do this type the following command:
    • restorecon -Rv /etc/openvpn
  • If you need to test your server settings you can run openvpn directly, say to debug your config file,  this way (press Ctrl+c to stop it):
    • openvpn /etc/openvpn/server.conf
  • Finally, you can turn the openvpn server on and enable it so that it starts during future reboots as well.
    • systemctl enable openvpn@server.service
    • systemctl start openvpn@server.service
  • Now that the server is running you will need to configure the firewall to allow vpn traffic connections AND route all your traffic through the system (via Network Address Translation.)  Start by backing up your old iptables configuration and enabling NAT forwarding in the Linux kernel:
    • mv /etc/sysconfig/iptables /etc/sysconfig/iptables.old
    • sysctl -w net.ipv4.ip_forward=1
  • Open up your favorite text editor and copy the following iptable rules into the file.  You will need to save the file as /etc/sysconfig/iptables.  This configuration assumes that eth0 is your public IP address and eth1 is your private.  If this is backwards just change eth0 to eth1 and vice versa.  Also it keeps port 22 open for ssh connectivity.

# Modified from iptables-saved by Bob Rockers
:OUTPUT ACCEPT [118860:18883888]
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 22 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp –dport 443 -j ACCEPT
-A INPUT -i tun+ -j ACCEPT
-A INPUT -j REJECT –reject-with icmp-host-prohibited
-A FORWARD -i tun+ -j ACCEPT
-A FORWARD -i eth1 -o tun+ -j ACCEPT
-A FORWARD -i eth0 -o tun+ -m state –state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -j REJECT –reject-with icmp-host-prohibited

  • To make NAT work across reboots you will need to modify the /etc/sysctl.conf file and change the line net.ipv4.ip_forward = 0to the following:
    • net.ipv4.ip_forward = 1
  • To make everything permanent type the following:
    • sysctl -p /etc/sysctl.conf
  • Now restart your firewall configuration:
    • systemctl restart iptables.service

That should take care of our server configuration. I will follow this post up with client configurations for Windows and Fedora 17 KDE installs. Please feel free to email any fixes/updates to the above configuration if you see something.  Below are a couple of the links I used to get this configuration working:

Finally, the above solution is susceptible to a man-in-the-middle attack from another client impersonating the server (not a problem for my setup as I personally know everyone who I have issued client certificates to.)  The solution is sign the server certificate with a tls-server only key and force clients to check this status on connection.  There is more documentation for this setup here and specifics about the easy-rsa setup here.  At some point I will update this tutorial to fix that issue but, for now, this has been a long enough post.

But what’s my motivation?

Scripts in generally and bash in particular fill an enormous amount of my time.  The ability to create scripts that can handle a number of diverse inputs is directly related to how flexible and robust the code-base is.  The most common problem when handling files in Bash is the problem with spaces.  Linux is both case sensitive and handles spaces with less… grace… than some OSes.  Bash suffers from these same issues.  The easiest way to handle this is with the IFS system variable.  IFS is simply the field delimiter for Linux (i.e. white space) and, because it is a modifiable system variable, you set it to something that you will not run into.  For example:

IFS=$(echo -en “\n\b”)
for var in *
# Do something with each line of a file
echo “$var”

That will solve the problem dealing with spaces when developing simple/basic scripts that are written for quick and dirty system management.  That said, when you are building scripts to use regularly you will need to do be a more comprehensive when testing your script.

A good place to start is by setting -u. Whenever testing new scripts, try running them without any arguments but WITH -u. If you fail to correctly initialize your variables running them with -u will warn you that there is a problem. For example:

$bash -u /tmp/mynewtestscript.sh
/tmp/mynewtestscript.sh: line 34: $DIRNAME: unbound variable

We can then verify that we have (at the very least) correctly initialized any variables that we will use and reduce the probability of side-effects.

A problem I ran into a lot with my early script creation was that I often needed standard output from one command to be sent to another command as command line input (as opposed to standard input.) The best way to solve this problem with using the bash built-in command execution form, for example:

echo $(ls)

But this isn’t always very elegant to implement directly, so another options is the wonderful xargs command.  xargs breaks the output of one command into individual arguments that it feeds to another command.  This allows you to use standard piping between otherwise un-pipeable commands.  For example:

ls | xargs echo

Sometime joining two vars can be complicated when those var names need characters between them.  To solve this you can you can use parameter substitution.  What this means, effectively, is that the var $tempvar and ${tempvar} are the same thing.  This allows you to combine variables with in-between characters without concern.


Another useful tip (found via this article from hacktux) is the mktemp executables for temporary file creation.  Needing a temp file to store intermediate data try the following:

tempdir=$(/bin/mktemp -d)

Another common problem for bash scripts being used for administration is that they need to be run as root (or sudo root on Ubuntu systems.)  The way to solve this is to check the EUID environmental variable.  Root will always be 0 for EUID and you can put a simple check at the beginning of your script with the following:

if [[ $EUID -ne 0 ]]; then
echo “This script must be run as root” 1>&2
exit 1

Need a random number of characters for your bash script?  Use dd and /dev/random to get a variable number of characters.  For example:

random=”$(dd if=/dev/urandom bs=3 count=1)”

Will give you three random characters (stored in $random) out of urandom current entropy pool.  Unfortunately the character  set is likely to be UTF-32 giving you a bunch of ?? symbols.  To convert those to base64 encoding just pipe the output through base64 (the conversion process may likely give you more than 3 characters to be sure to regexp to the number of characters you need):

random=”$(dd if=/dev/urandom bs=3 count=1 | base64)”