Thursday, December 30
Shared Directories Under LXD - The Easy Way
LXD is great. The problem is, it's deceptively complex - it's very easy to get up and running with containers, and it all works perfectly for basic tasks, but under the hood there's a ton of clever stuff going on and a huge set of features and options and when you try to do something complicated that was easy on older container systems like OpenVZ you can quickly run aground on restrictions imposed by the newer security model.
Case in point, shared directories.
By default it's probably not. If you get no result from that command, run these two commands:
Let's assume you already have a directory you want to share across a couple of containers. Let's call it /test, and it's owned by user test in group test:
And you have created containers called test1 and test2, which are each set up with a user test as well.
"testdisk" here can be any name you want; LXD will assign it to the virtual device it creates.
LXD is great. The problem is, it's deceptively complex - it's very easy to get up and running with containers, and it all works perfectly for basic tasks, but under the hood there's a ton of clever stuff going on and a huge set of features and options and when you try to do something complicated that was easy on older container systems like OpenVZ you can quickly run aground on restrictions imposed by the newer security model.
Case in point, shared directories.
Dead simple under OpenVZ. Create a directory, bind mount it into the container, it just works.
Under LXD it works too, sort of. The problem is that user IDs in LXD containers are different to user IDs on the host, and user IDs between any two containers are also different, even if you cloned a container, so permissions are a nightmare.
Under LXD it works too, sort of. The problem is that user IDs in LXD containers are different to user IDs on the host, and user IDs between any two containers are also different, even if you cloned a container, so permissions are a nightmare.
This is intentional. If there's a combination of (1) an RCE in your application code, (2) a local privilege escalation bug, and (3) a container escape bug, the hacker will be loose on your host node... As userid 9000000 with no privileges at all.
Anyway, there's an easy solution to this when it comes to shared directories and it's built right into LXD.
Anyway, there's an easy solution to this when it comes to shared directories and it's built right into LXD.
# lsmod | grep shiftfs
By default it's probably not. If you get no result from that command, run these two commands:
# snap set lxd shiftfs.enable=true# systemctl reload snap.lxd.daemon
Let's assume you already have a directory you want to share across a couple of containers. Let's call it /test, and it's owned by user test in group test:
drwxrwx--- 2 test test 4 Dec 29 17:38 test/
And you have created containers called test1 and test2, which are each set up with a user test as well.
# lxc config device add test1 testdisk disk path=/test source=/test shift=true# lxc config device add test2 testdisk disk path=/test source=/test shift=true
"testdisk" here can be any name you want; LXD will assign it to the virtual device it creates.
It's the
shift=true that does the magic. That uses shiftfs to map the users between the host and the containers for you automatically.Now your directory is shared and regular Unix permissions just work between the host and the containers.
But Wait There's More
The other neat thing here comes thanks to ZFS (and if you're using LXD without ZFS stop that and get ZFS set up right now).
That shared directory is part of your general ZFS pool, not your containers. This means:
That shared directory is part of your general ZFS pool, not your containers. This means:
- When you snapshot a small container with a ton of shared data, that snapshot isn't burdened with the shared data. Just what is actually running in the container itself.
- You can see the contents from the host without fussing around with namespaces.
- You can snapshot the contents with ZFS, and then you can see the contents of the snapshot directly in the filesystem. You can take a snapshot and then just rsync off a backup.
- How the hell did I get that working for the existing server without this? Because I know I did.
- Anyway.
- Let's say you have fifty apps running. You can have separate shared directories for all their databases under one ZFS pool and snapshot all of them in one go. Back them all up in one go.
- Let's say those fifty apps are using some non-Euclidean mix of MySQL, MariaDB, PostgreSQL, Redis, MongoDB, Cassandra, CouchDB, Neo4j, Elasticsearch, RabbitMQ, and who knows what else. When you take that snapshot, you snapshot them all together at a single point in time, so you have as close as possible to a consistent backup of the entire eldritch mess.
- You can set up a tree of ZFS datasets under a parent and adjust their individual configurations (e.g. recordsize and compression algorithm) to suit the speciifc databases and then use
zfs snapshot -rto snapshot the whole lot at once.
This was something holding up the server migration. It no longer is.
There's another way to do all this but right now it apparently (1) doesn't work on Ubuntu 20.04 and (2) doesn't support ZFS so for me a bit of a non-starter.
There's also an older and more difficult way to do this by manually setting up the user/group ID mappings but (1) I've lost that link and (2) you don't want to do that anyway.
Posted by: Pixy Misa at
04:10 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 750 words, total size 6 kb.
52kb generated in CPU 0.0182, elapsed 0.1352 seconds.
56 queries taking 0.1228 seconds, 338 records returned.
Powered by Minx 1.1.6c-pink.
56 queries taking 0.1228 seconds, 338 records returned.
Powered by Minx 1.1.6c-pink.
