So...
I've managed to get the AppleTV device logged in, but if you want to install an app, you need to have an iTunes store account.
To create an iTunes Store Account, it turns out that you need an iOS or Windows device on which you can install iTunes.
Guess what I don't have any of...