Running etcd over SSH
Using authorized_keys options to securely tunnel

Want to tunnel etcd traffic over SSH? Here’s a pitfall I ran into.

Mgmt supports tunnelling the etcd traffic over SSH. This is quite useful, because it means that for it to connect to etcd, it doesn’t need to open up any new ports other than 22.

You can run the agent like so:

mgmt run --ssh-url=etcd@etcdserver:22 --seeds=http://127.0.0.1:2379 --no-server --no-magic empty

and it just works!

It’s no fun giving out more access than is really necessary, so I added this line to the etcd user account in their ~/.ssh/authorized_keys file:

command="echo 'Port forwarding only'; exit",no-agent-forwarding,no-pty,no-user-rc,no-X11-forwarding,permitopen="localhost:2379" ssh-...

Note the actual key being used is ellipsized for brevity and customer privacy.

Things should be working, but they weren’t.

I initially suspect something amiss with the golang bindings. Perhaps they didn’t support the “only-port-forwarding” mode? While debugging the code, it would fail at this line when opening the internal tunnel:

import "golang.org/x/crypto/ssh"

sshConfig := &ssh.ClientConfig{
	User: "etcd",
	// ...
}
sshClient, err := ssh.Dial("tcp", "etcdserver:22", sshConfig)
// ...
tunnel, err := sshClient.Dial("tcp", "127.0.0.1:2379")
// ssh: rejected: administratively prohibited (open failed)

What’s going on? What does that error even mean.

I tried the venerable ssh utility and things worked fine:

ssh -N etcd@etcdserver -p 22 -L 2379:localhost:2379 -v

Give up? Amazingly, ssh treats localhost and 127.0.0.1 differently. This means you must be consistent everywhere, or you’ll get this error. Since localhost could resolve differently in certain rare situations, I’ve changed everything to use 127.0.0.1 and now things work as expected.

My updated authorized_keys now looks like:

command="echo 'Port forwarding only'; exit",no-agent-forwarding,no-pty,no-user-rc,no-X11-forwarding,permitopen="127.0.0.1:2379" ssh-...

If you like hacking on this part of the stack, get involved with mgmt and you’ll have a lot of fun!

Enterprise support, training, and consulting are all available.

Happy Hacking,

James

You can follow James on Mastodon for more frequent updates and other random thoughts.
You can follow James on Twitter for more frequent updates and other random thoughts.
You can support James on GitHub if you'd like to help sustain this kind of content.
You can support James on Patreon if you'd like to help sustain this kind of content.

May 14, 2025
379 words

• technical