Over the last week, I spent a lot of time redesigning the infrastructure for my server environment in Amazon Web Services. Part of the architecture included building a management backbone for the servers. The design looks similar to the following diagram:
There were a few important concepts to this design:
- The Management Server would only be turned on when remote administration work needed to be accomplished. This is done through the AWS Console.
- Having the Management Server available keeps from having to open remote administration services (i.e. SSH) on production servers that could be exploited.
- The Management Server would not have a static public IP address. It will be dynamically assigned on each startup.
All of the servers are Linux servers, so my remote administration is done through SSH. I started to research how others work with this design and here is what I found.
When you setup Linux servers in AWS, they are configured to use certificates to authenticate to the server. With this concept, here is how you connect in and remote to the Web Server:
Step 1 – Use your certificate file to SSH in to the Management Server.
Step 2 – Make sure you have a copy of your certificate file on the Management Server so that you can then SSH from the Management Server in to the Web Server.
Here’s the problem . . . doing this requires you to store your private certificate files on a server out in the cloud. If that server should ever get compromised, then the hacker has access to the keys and therefore has access to all of your servers.
A while back, I had explored the concept of using SSH tunnels. It has been a while since I have used them, but a little searching on the Internet and I was quickly reminded of the concepts. Here is the basic process:
Step 1 – Setup an SSH tunnel from your computer to the Management Server using your certificate file. This tunnel creates an endpoint on your local machine that acts like an interface on the Management Server.
Step 2 – Use that local interface to tunnel through your connection and access the remote server, still using the certificate file on your local machine.
With this process, the certificate keys never leave your computer. They are never stored out on the cloud.
I work with Mac computers, so I use the Terminal program and the shell based SSH utility for my administration. The following steps should work for any Mac or Linux/Unix based environment.
The first step is to establish a tunnel from your computer to the Management Server. Using the diagram from above, here is how that would look.
ssh -i MySSHCertificate.pem -L 9500:10.1.0.15:22 firstname.lastname@example.org
Let’s breakdown the various parts of the command:
This should look familiar if you have done any administration with Linux servers in AWS. This simply directs the SSH client to use the MySSHCertificate.pem certificate file when authenticating to the Management Server.
This sequence does a lot of things. First, it establishes a local tunnel connection at port 9500 on your local computer. This is an arbitrary number and you have a lot of flexibility to pick any valid TCP port that you wish. Next, we are going to setup the tunnel from port 9500 to go to 10.1.0.15 which is the internal management IP address of the Web Server that we want to manage. Finally, to manage the Web Server, we will be connecting to port 22 which is the SSH port for the Web Server.
Finally, we are connecting to our Management Server at it’s dynamic public IP address of 22.214.171.124 and we are connecting as the ubuntu user.
If you are successful, you will see a normal SSH prompt to the Management Server. For now, minimize this terminal session and let it run in the background.
Now we need to connect to the Web Server. We are going to tunnel through the locally established TCP port of 9500 to get to the remote server. To do this, we will start up a new terminal session and use the following command:
ssh -i MySSHCertificate.pem -p 9500 ubuntu@localhost
Let’s breakdown this command into more detail:
Again, we are using our locally stored certificate file to authenticate to the remote server. In this example, we are using one certificate for both servers. In a more secure environment, you might have a different certificate file for each server you connect to. Either scenario will work with this technique.
This -p option tells SSH which TCP port to use when setting up the connection. In our example, we setup a local port at port number 9500, so we need to tell SSH to use that port.
This part may seem a little strange. Remember that we are using a tunnel that has been established at a port on our local computer. If you use a tool like netstat, you would see that there is a listening port on your computer at 9500. We use that port as our tunnel to the remote server. So when setting up this SSH connection, we are basically connecting to our local computer, but using the 9500 port to get to the remote system.
Again, if everything worked as planned, you should now have an SSH connection to the remote Web Server. You should be able to do everything that you need.
When you are finished, simply exit out of your SSH session to the Web Server. Then go back to the session that you minimized from step 1 and exit out of that session. That will shutdown your tunnel to the environment. Lastly, don’t forget to go into the AWS Console and shutdown your Management Server when it is not in use.
At some point, you will probably need to use SFTP to move files back and forth to the Web Server. Not a problem . . . the technique works basically the same way with one minor change.
Step 1 – Setup you tunnel the same way we did above.
Step 2 – Use the following command to setup the SFTP to the Web Server:
sftp -i MySSHCertificate.pem -P 9500 ubuntu@localhost
I realized in my testing that for some reason, the sftp client uses a capital “P” instead of a lower case one to define the port to connect to.