SSH Tunnel Maintenance and SELinux on RHEL/CentOS 6

I like FreeBSD, and if I got a choice, then I deploy servers with FreeBSD and suggest Apple Mac’s as clients. I was contracted to migrate a Web Application (Apache/PHP/MySQL) to RHEL 6.5 systems, serving Windows Clients, huh, and huhhh… Anyway, there is a distributed data store system which is either synchronized or replicated over ssh tunnels.

When setting up the ssh tunnels, SELinux came into my way at several stages, and I was close to switching it off several times, but eventually, I managed to keep it running, and so my client cannot blame me, to leaving his system unsecured.

IMHO, SELinux and its interactions with the various levels of a *nix-system is very poorly documented, and you need R&D strategies to get your stuff working with it. And here, I will disclose my findings and how I managed to satisfy the SELinux way of MAC.

The Tunnel User

I am used to setup a non-privileged tunnel user at the destination machine outside of the standard home directory. SELinux is used to not allow users to reside in anything else than the standard home directory, the documentation does not tell this to you, you need to find it out the hard way, i.e. the so called Research part of R&D. Once you got the knowledge you can do the Development:

Create the Tunnel User and its home directory:

yum install policycoreutils-python
useradd -c 'SSH Tunnel User' -d /var/tunnel \
   --home /var/tunnel -m -r -u 201 -s /sbin/nologin -Z unconfined_u tunnel

Set up the ssh directory within the tunnels home:

mkdir -m 0700 /var/tunnel/.ssh
touch /var/tunnel/.ssh/authorized_keys
chmod 0600 /var/tunnel/.ssh/authorized_keys
chown tunnel:tunnel -R /var/tunnel
chcon -v -u system_u -r object_r -t user_home_t /var/tunnel
chcon -v -u system_u -r object_r -t ssh_home_t /var/tunnel/.ssh
chcon -v -u system_u -r object_r -t ssh_home_t /var/tunnel/.ssh/authorized_keys

Allow ssh connections for the users root and tunnel only:

echo "AllowUsers root tunnel" >> /etc/ssh/sshd_config
service sshd restart

At the peer machine (say host A) create the ssh RSA key pair, and copy the public key to the pre-generated but still empty authorized_keys on the target host B:

ssh-keygen -C "SSH Tunnel from host A to host B"
scp ~/.ssh/

Note, when copying stuff onto an object confined by SELinux, the substitute will inherit all the rights and attributes from the replaced object. Under SELinux don’t use mv, but always use the cp/rm combo.

Tunnel Testing

Test the tunnel by issuing the following command on host A:

ssh -NC -L 13306:

This command should not return immediately. It should neither ask for a password nor give other responses. Keep it like this, open another terminal window, and assuming the MySQL server is running on host B, you can provoke it to respond by the following:

mysql --user=USER --password=PASSWORD --host= --port=13306 DATABASE
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 2
Server version: 5.1.71 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.


A Watchdog for Tunnel Maintenance

Now, we are able to establish a tunnel from host A to host B, but how to maintain it? Mac OS X got launchd which does a pretty good job. FreeBSD has some ports dealing with this kind of task. And RHEL? I don’t know, I don’t want to know, and I am angry with anyone who knows. I was tired doing more research, only to find out that SELinux doesn’t like stuff that launches other stuff, and then to find yet another solution for this.

So, I created a small daemon in C, called tguard, which compiles fine on Mac OS X, FreeBSD, RHEL, CentOS, and probably other Linux systems:

//  tguard.c
//  Created by Dr. Rolf Jansen on 2012-10-13.
//  Copyright 2012 InstantWare - Dr. Rolf Jansen. All rights reserved.
// gcc tguard.c -std=gnu99 -Os -g0 -o /usr/local/bin/tguard
//   or
// clang tguard.c -Wno-empty-body -Os -g0 -o /usr/local/bin/tguard
// strip /usr/local/bin/tguard

#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <signal.h>
#include <string.h>
#include <syslog.h>
#include <time.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/wait.h>

#define DAEMON_NAME    "tguard"

const char *pidfname = "/var/run/";

void usage(const char *executable)
   const char *r = executable + strlen(executable);
   while (--r >= executable && *r != '/'); r++;
   printf("nusage: %s [-p file] [-f] [-n] [-h] -x commandn", r);
   printf(" -p file the path to the pid file [default: /var/run/]n");
   printf(" -f      foreground mode, don't fork off as a daemon.n");
   printf(" -n      no console, don't fork off as a daemon - started/managed by launchd.n");
   printf(" -h      shows these usage instructions.");
   printf(" -x cmd  the actual command to be guarded.nn");

static void signals(int sig)
   switch (sig)
      case SIGHUP:
         syslog(LOG_ERR, "Received SIGHUP signal.");
         kill(0, SIGHUP);

      case SIGINT:
         syslog(LOG_ERR, "Received SIGINT signal.");
         kill(0, SIGINT);

      case SIGQUIT:
         syslog(LOG_ERR, "Received SIGQUIT signal.");
         kill(0, SIGQUIT);

      case SIGTERM:
         syslog(LOG_ERR, "Received SIGTERM signal.");
         kill(0, SIGTERM);

         syslog(LOG_ERR, "Unhandled signal (%d) %s", sig, strsignal(sig));

typedef enum
} DaemonKind;

void daemonize(DaemonKind kind)
   switch (kind)
      case noDaemon:

      case launchdDaemon:
         signal(SIGTERM, signals);

      case discreteDaemon:
         // fork off the parent process
         pid_t pid = fork();

         if (pid < 0)

         // if we got a good PID, then we can exit the parent process.
         if (pid > 0)

         // The child process continues here.
         // first close all open descriptors
         for (int i = getdtablesize(); i >= 0; --i)

         // re-open stdin, stdout, stderr connected to /dev/null
         int inouterr = open("/dev/null", O_RDWR);    // stdin
         dup(inouterr);                               // stdout
         dup(inouterr);                               // stderr

         // Change the file mode mask, 027 = complement of 750

         pid_t sid = setsid();
         if (sid < 0)
            exit(EXIT_FAILURE);     // should log the failure before exiting?

         // Check and write our pid lock file
         // and mutually exclude other instances from running
         int pidfile = open(pidfname, O_RDWR|O_CREAT, 0640);
         if (pidfile < 0)
            exit(1);                // can not open our pid file

         if (lockf(pidfile, F_TLOCK, 0) < 0)
            exit(0);                // can not lock our pid file -- was locked already

         // only first instance continues beyound this
         char s[256];
         int  l = snprintf(s, 256, "%dn", getpid());
         write(pidfile, s, l);      // record pid to our pid file

         signal(SIGHUP,  signals);
         signal(SIGINT,  signals);
         signal(SIGQUIT, signals);
         signal(SIGTERM, signals);
         signal(SIGCHLD, SIG_IGN);  // ignore child
         signal(SIGTSTP, SIG_IGN);  // ignore tty signals
         signal(SIGTTOU, SIG_IGN);
         signal(SIGTTIN, SIG_IGN);


int main(int argc, char *argv[])
   char        ch;
   const char *cmd   = argv[0];
   DaemonKind  dKind = discreteDaemon;

   while ((ch = getopt(argc, argv, "p:fnhx")) != -1)
      switch (ch)
         case 'p':
            pidfname = optarg;

         case 'f':
            dKind = noDaemon;

         case 'n':
            dKind = launchdDaemon;

         case 'x':
            goto execute;

         case 'h':

   argc -= optind;
   argv += optind;


   pid_t pid;
   int   statloc;
   long  sleep_usecs = 1000000;

      if ((pid = fork()) == 0)
         goto launch_child;

      if (pid < 0)
         return 1;

      if (pid > 0)
       	 clock_t t0 = clock();


         if (((double)(clock() - t0))/CLOCKS_PER_SEC > 180.0)   // The client died, however if it was running for more than 180 seconds
            sleep_usecs = 1000000;                              //   then reset the sleeping time to 1 second
         else                                                   //   otherwise
            sleep_usecs *= 2;                                   //   stepwise increase the sleeping time
            if (sleep_usecs > 256000000)                        //   but do not let it go above 256 seconds = ca. 4 min.
               sleep_usecs  = 256000000;

            syslog(LOG_WARNING, "Child [%zd] aborted! Re-spawning '%s' after %ld seconds.n", pid, argv[0], sleep_usecs/1000000);

         usleep(sleep_usecs);                                   // Sleep for a reasonable amount of time, before re-spawning the child.
   while (1);

   execvp(argv[0], &argv[0]);
   return 0;

On Linux compile it using:
  gcc tguard.c -std=gnu99 -Os -g0 -s -o /usr/local/bin/tguard

On FreeBSD or Mac OS X compile it using:
  clang tguard.c -std=gnu99 -Os -g0 -s -o /usr/local/bin/tguard

On the RHEL Host A add the following to /etc/rc.d/rc.local, i.e. one entry for each tunnel that needs to be maintained:

/usr/local/bin/tguard -p /var/run/ \
   -x /usr/bin/ssh -NC -L 7306:
/usr/local/bin/tguard -p /var/run/ \
   -x /usr/bin/ssh -NC -L 8306:
/usr/local/bin/tguard -p /var/run/ \
   -x /usr/bin/ssh -NC -L 9306:

Then restart Host A, and verify it coming up with working tunnels by issuing the command ps axj.

The End

Copyright © Dr. Rolf Jansen - 2013-12-19 20:43:19

Discussion on Twitter: 1082795464448200704