Tag Archives: Linux

Use SSH Tunnels for Remote Administration


Over the last week, I spent a lot of time redesigning the infrastructure for my server environment in Amazon Web Services. Part of the architecture included building a management backbone for the servers. The design looks similar to the following diagram:

Network Diagram – Click to See Full Size

There were a few important concepts to this design:

  • The Management Server would only be turned on when remote administration work needed to be accomplished. This is done through the AWS Console.
  • Having the Management Server available keeps from having to open remote administration services (i.e. SSH) on production servers that could be exploited.
  • The Management Server would not have a static public IP address. It will be dynamically assigned on each startup.

Internet Solution

All of the servers are Linux servers, so my remote administration is done through SSH. I started to research how others work with this design and here is what I found.

When you setup Linux servers in AWS, they are configured to use certificates to authenticate to the server. With this concept, here is how you connect in and remote to the Web Server:

Step 1 – Use your certificate file to SSH in to the Management Server.

Step 2 – Make sure you have a copy of your certificate file on the Management Server so that you can then SSH from the Management Server in to the Web Server.

Here’s the problem . . . doing this requires you to store your private certificate files on a server out in the cloud. If that server should ever get compromised, then the hacker has access to the keys and therefore has access to all of your servers.

My Solution

A while back, I had explored the concept of using SSH tunnels. It has been a while since I have used them, but a little searching on the Internet and I was quickly reminded of the concepts. Here is the basic process:

Step 1 – Setup an SSH tunnel from your computer to the Management Server using your certificate file. This tunnel creates an endpoint on your local machine that acts like an interface on the Management Server.

Step 2 – Use that local interface to tunnel through your connection and access the remote server, still using the certificate file on your local machine.

With this process, the certificate keys never leave your computer. They are never stored out on the cloud.

Specific Steps

I work with Mac computers, so I use the Terminal program and the shell based SSH utility for my administration. The following steps should work for any Mac or Linux/Unix based environment.

Step 1

The first step is to establish a tunnel from your computer to the Management Server. Using the diagram from above, here is how that would look.

ssh -i MySSHCertificate.pem -L 9500: ubuntu@

Let’s breakdown the various parts of the command:

-i MySSHCertificate.pem

This should look familiar if you have done any administration with Linux servers in AWS. This simply directs the SSH client to use the MySSHCertificate.pem certificate file when authenticating to the Management Server.

-L 9500:

This sequence does a lot of things. First, it establishes a local tunnel connection at port 9500 on your local computer. This is an arbitrary number and you have a lot of flexibility to pick any valid TCP port that you wish. Next, we are going to setup the tunnel from port 9500 to go to which is the internal management IP address of the Web Server that we want to manage. Finally, to manage the Web Server, we will be connecting to port 22 which is the SSH port for the Web Server.


Finally, we are connecting to our Management Server at it’s dynamic public IP address of and we are connecting as the ubuntu user.

If you are successful, you will see a normal SSH prompt to the Management Server. For now, minimize this terminal session and let it run in the background.

Step 2

Now we need to connect to the Web Server. We are going to tunnel through the locally established TCP port of 9500 to get to the remote server. To do this, we will start up a new terminal session and use the following command:

ssh -i MySSHCertificate.pem -p 9500 ubuntu@localhost

Let’s breakdown this command into more detail:

-i MySSHCertificate.pem

Again, we are using our locally stored certificate file to authenticate to the remote server. In this example, we are using one certificate for both servers. In a more secure environment, you might have a different certificate file for each server you connect to. Either scenario will work with this technique.

-p 9500

This -p option tells SSH which TCP port to use when setting up the connection. In our example, we setup a local port at port number 9500, so we need to tell SSH to use that port.


This part may seem a little strange. Remember that we are using a tunnel that has been established at a port on our local computer. If you use a tool like netstat, you would see that there is a listening port on your computer at 9500. We use that port as our tunnel to the remote server. So when setting up this SSH connection, we are basically connecting to our local computer, but using the 9500 port to get to the remote system.


Again, if everything worked as planned, you should now have an SSH connection to the remote Web Server. You should be able to do everything that you need.

When you are finished, simply exit out of your SSH session to the Web Server. Then go back to the session that you minimized from step 1 and exit out of that session. That will shutdown your tunnel to the environment. Lastly, don’t forget to go into the AWS Console and shutdown your Management Server when it is not in use.

Need SFTP?

At some point, you will probably need to use SFTP to move files back and forth to the Web Server. Not a problem . . . the technique works basically the same way with one minor change.

Step 1 – Setup you tunnel the same way we did above.

Step 2 – Use the following command to setup the SFTP to the Web Server:

sftp -i MySSHCertificate.pem -P 9500 ubuntu@localhost

I realized in my testing that for some reason, the sftp client uses a capital “P” instead of a lower case one to define the port to connect to.

Finding High CPU Sources in Linux

The Problem

I recently came across a situation with one of my Linux servers. A web application on the server was getting very slow to respond. Web pages were taking 25-30 seconds to load when they usually load in less than 4 seconds. Something was obviously wrong.

Looking to Splunk for Answers

I turned to my trusty logging software, Splunk. The more I use this software, the more I love the insight it gives me into my systems. I started off with my Operational Monitoring dashboard that I have built over time. This dashboard gives me a number of graphical views into the key server management indicators. I scrolled down the page to my CPU utilization view and I saw the following:

High CPU Usage Graph

My web server, shown here as the red line, was clearly in a higher than usual CPU utilization and that explained a lot of things. But it didn’t yet tell me why this was occurring.

I started a new search in Splunk for the period around when the CPU utilization started increasing. My Splunk environment captures a lot of data, so at a first glance, there was a lot to look at.

As I was sifting through the data, I remembered that through the Splunk Add-On for *nix, the software is periodically capturing the output of the Linux ps command. This command line utility reports on what applications are currently running at that moment on the server. In particular, it also shows the amount of CPU utilization that is being used by that program. Knowing this, I crafted the following search and visualization:

host=HOSTNAME source=ps | timechart span=1h sum(pctCPU) by COMMAND

Increasing Application CPU Usage

This search gave me a graphical view of each program that was running and sums up the pctCPU value for each record in that hour. I chose to use the SUM function because any subtle increase in CPU usage would be compounded with a SUM instead of an average.

The green line in this graph clearly showed me the program that was beginning to utilize more and more CPU at the same time that the graph above showed the higher overall utilization.

Wrapping Up

Now that I knew which application was causing the problem, I started doing some research. I did find other users reporting similar CPU utilization issues with the application. I felt better knowing it wasn’t an issue specific to my environment. I am still continuing to learn about this issue and I’m sure an overall fix will soon follow.