How to use SSH server on a remote device and overcome routing issues

Question

I am designing an embedded Linux device that acts as a gateway for sensor data. In some cases this device will use a back-haul that I don't control (e.g. customer supplied cell or cable modem). I assume this device will be behind some sort of NAT and that network parameters will be assigned via DHCP in most installations.

For typical communications, the device will open up sockets with a REST web service.

I would like to add some secure diagnostic, command, and control capabilities. What I want is for each gateway to be running something like an SSH server. That way I could have a very flexible interface that a human can use on demand to communicate with a misbehaving gateway and troubleshoot or fix what is going wrong. This presents some obvious security issues, but I am more interested in talking about the networking hurdles I need to overcome at this point:

1. My customers would need to open a port on their router and forward port that my device, or my device needs to be directly connected to the internet and not behind NAT.
2. If my device is behind NAT it will probably need a static IP
3. The customer will need a static IP for their network, or we'll need to capture dynamic IP's through our REST interface.

How can I hole punch SSH?

1. Can I make each gateway act like an SSH client and open a session with one of our servers, is there some way to transfer or takeover the SSH session?
2. Can I tunnel the data using an IP-in-IP connection or similar?

Answer 1

Don't try and solve this with trying to connect to the device the wrong way through NAT.

Approach this the other way round, have the device connect out to a known point (e.g. a cloud host) which can then be used to connect back to the device. e.g. Have a button/config option on the device that starts the VPN connection to a known point when you want to do the debugging, the user can then turn this off when no longer needed.

This also means that any attacker needs to both compromise the VPN endpoint and get the customer to enable "debug" mode before they could get in.

The device can still run a SSH server but only make it available over the VPN connection and access can be protected by only allowing access with a SSH key (with only the public part being on the device so it doesn't matter if it's compromised).

Answer 2

Rather than describing how to cover all your wishes, I'll act as if there's one precise question. (All your wishes may be covered in the article I recently found here )

The precise question I'll answer is: I have a linux gateway that gathers sensor data.
I have shell access to that gateway logged in as "gatewayuser"
I want to be able to access that gateway by ssh from "somewhere on the internet"
How do I do that?

OK, now we're working from some fixed criteria. You're right, SSH and tunneling are the answer. You probably can't or don't want to expose your sensor gateway as a fixed place on the internet. (At a minimum, your router will probably make it difficult.)

So, you want to use SSH reverse forwarding
ssh -R

If you want your sensor gateway to be able to continually try to establish this connection, you'll want the other end to be a static ip address or at least unchanging host name.

Say you use some cheap VPS or developer-tier (free) Amazon AWS (or other) instance. Say that this public host is assigned the IP address 111.222.33.44 and you created an account 'clouduser' Say this host has port 2222 available for use.

From your linux gateway, you would enter the command

sudo ssh -R 2222:localhost:22 [email protected]

Technically you will log into 111.222.33.44, and if you're going to run this in a crontab, that's not necessarily what you want.

Now, from that cloud server 111.222.33.44, assuming you've ssh'ed into it as clouduser, you can then run

ssh -p 2222 localhost -l gatewayuser

There are ways to secure this better (bind addresses come to mind) and make it so you don't have to log into the gateway first (crontab comes to mind). I found that blog article did a pretty good job, so I'm going to conclude my simple "do this one thing" reply and refer you off to those resources.