Technically every time I have tried to push it into my production workflow I've hit some snags, mostly around networking and some volume stuff but those snags are getting chipped away each release. The last time I did a strong test flight was around 3.X, so probably about time to try again, 4.0 was a big release.
I like the integration with systemd and bringing "pods" out of k8. I like the "they're just processes" philosophical perspective and more "linux tech" focus of the team - i.e. cgroups v2 exists, lets use it. I would like to see some minor UI stuff such as compressing buildah stage output like docker buildx does but it's understandable why that isn't there (yet?).
I think my only remaining quibble is getting true remote addresses when using rootless networking.
You can use podman with buildkit, it's just kind of a pain because you have to do it manually. It is missing some nice to have feature integrations with buildkit. The instructions are fairly clear: https://github.com/moby/buildkit
- Faster network. Rootless Podman will run with native network speed. Normally rootless Podman runs with reduced network speed due to the performance penalty that comes from using slirp4netns.
- Improved security as you can disable the ability to establish outgoing connections with --network=none. The container can still communicate over the socket-activated socket with a client that has connected via the internet.
Docker does not support socket activation of containers. (Docker only supports socket activation of the Docker daemon)
Edit: A clarification about the network speed. The improved speed is about the communication
that passes over the socket-activated socket. This communication does not pass through slirp4netns so it has the same performance characteristics as the normal network on the host.
> - Faster network. Rootless Podman will run with native network speed. Normally rootless Podman runs with reduced network speed due to the performance penalty that comes from using slirp4netns.
Did they fix the IP propagation issue with Rootless networking ?
It makes it largely useless when the Proxy is also a container.
on Docker you can force it to use Slirp, but its slow and doesn't support IPv6.
Regarding the other idea:
I've now tested it and verified that it works. The remote address is available when running a socket-activated container with rootless Podman.
Docker can’t really support Socket Activation for containers because it is build around the docker daemon that fulfills similar tasks as systemd. They could only copy the concept to dockerd.
That's neat and quite a unixy solution but still fairly limiting. Are there any plans for a hybrid solution where most of podman runs as non-root via userns but invokes a suid helper to setup the network?
Does this work in systemd-less distros like the most common WSL2 ones? I tried to fake Podman to use in a devcontainer but it complained of a missing `/var/run/docker.sock`.
I was looking at this just last night and the story with Ubuntu is horrible. That's essentially going to be stuck on v3.4 for ten years because it's "a lot of work" to get into that distribution.
You'd think an entity the size of RedHat trying to take the reins from Docker would understand that this is an investment they have to make to make it a first-class replacement.
I also installed it on Windows to see how the WSL engine works but now it conflicts with my existing v3 Podman installation on Ubuntu 20.04 in WSLv2 so I guess I'm out of luck.
Also may be of interest to people here but Podman desktop had a release yesterday. It's pretty primitive and I couldn't get it to work to use my existing auth.json but it's there.
It was a pretty frustrating experience when all I wanted was to be able to "podman login" to a local repository so Jib would pull down base layers correctly.
If you install the podman package via Scoop, you'll get a `podman` client shim, and then if you run `podman machine init` it'll automatically set up a WSL instance running Fedora with Podman set up and relay the necessary sockets for you so that running `podman ps` or whatever on Windows Just Works™.
Then if you want to can run `wsl -d podman-machine-default` to log into the distro as normal. You can also copy the distro, import/export/register it as usual if you want a clone unaffiliated with the podman package per se.
What the official Windows Podman installer, also official Podman Desktop that installs Podman and scoop's Podman package should really do is offer to integrate with your existing WSL2 distro like Docker Desktop does.
The way that Docker Desktop integrates is just one extra step away from `podman machine init`, since Docker Desktop just does the same socket relaying stuff for your existing WSL distros as it does for Windows. The actual Docker daemon only runs on the special distro that the tool (Docker Desktop or `podman machine` or whatever) sets up.
I assume that Podman Desktop does that but idk because I don't use it. Rancher Desktop also works this same way.
In all cases, the integration with existing distros happens inside the GUI app. Maybe I'll check tomorrow whether Podman Desktop offers comparable integration.
But yeah, it's a good integration to have because the native Windows CLI experience is still so impractical and clunky that many developers end up setting up a pet distro in WSL and pretty much living in it as their default terminal session. Good integration with cmd.exe or (pwsh.exe running under Windows Terminal, for that matter) is cool but it doesn't mean much to someone who does all their work in an Ubuntu WSL VM or whatever.
Personally I followed the rootfs way of installing Fedora to WSL2. It was simple enough and worked fine (including podman). I found no reason to use external tools / scripts / modified distros.
Sadly, some anti-cheat tools in games still refuse to work with WSL2 (they hate Hyper-V, I guess it's been used as attack vector), so back to VMware Player on my personal workstation and using terminal to open Linux shell.
I don't think GP was suggesting any official affiliation between Red Hat and canonical. I think they were making a point that there's a lot of potential users on Ubuntu who might switch to podman if it were available. When trying to establish a project, user acquisition is a critically important part.
I agree with them. I think Red Hat should be making effort to get podman working well in Ubuntu (well, Debian but would benefit Ubuntu). Although it's very possible that Red Hat is trying and have met resistance. Canonical wants for a very different direction and it wouldn't surprise me at all if they were throwing road blocks in the way (or at least, doing nothing to remove the road blocks).
There's also Arch, or basically anything else besides Ubuntu. Podman isn't the only thing that is chronically out of date on it. Ubuntu has definitely lived long enough to become the villain.
I wonder why Fedora doesn't provide an official WSL package on the Microsoft Store as other distros do. My guess is that they feel that the WSL kernel and init diverge too far from the Fedora kernel and systemd. Can anyone from the Fedora project comment on this?
It's because Fedora strives to be open-source and free software only. WSL isn't completely libre so they can't support it officially and neither is the Windows Store. There's third party Fedora images for WSL you can install.
I used Arch and have access the pretty much the latest release of Podman anytime. Your qualms with Ubuntu packaging don't make a lot of sense. Any distro has requirements to become an official packager. Canonical and Ubuntu have been pushing LXC/LXD as their container solution. If they wanted the latest version of Podman then their packagers can build and package it, or someone else can and create a repository or PPA.
Also, I was able to get it to work on Windows fine. Maybe try removing your existing install and creating a new one.
> I was looking at this just last night and the story with Ubuntu is horrible. That's essentially going to be stuck on v3.4 for ten years because it's "a lot of work" to get into that distribution.
They used to provide relatively recent builds in their kubic repos. Unfortunately, for some reason, they decided to discontinue it[0]. They mentioned some CVEs or something in some issues raised around this, but to me that means pushing a new version/build and not discontinuing it.
Anyway, one of the members of the Containers org provides unstable kubic repos[1][2] for non RH systems. Unfortunately, this includes RCs, and non-stable versions, which is fine to get bleeding edge, but I'd rather just have the stable versions.
Due to the above, I've written some scripts to build deb packages for all the latest stable versions. So hopefully you can simply download the deb from GH releases[3] and then `dpkg -i *.deb && apt-get install -f`.
Ubuntu has nothing to do with Podman / Redhat. Ubuntu also has a terrible track record of not aligning where it makes sense with other distorts and not properly testing packages - it's just a bit of a flaky distro in general.
> . Ubuntu also has a terrible track record of not aligning where it makes sense with other distorts
You mean not aligning with Red Hat and what they're pushing on everyone else. Ubuntu is on a shorter release cycle compared to Debian so they're usually the first non-Red Hat distro with new stuff. Systemd vs Upstart, Unity vs GNOME (3?), etc.
They try to do new stuff, and there's nothing wrong with that. Not everyone should blindly follow RH's lead. Systemd was objectively shit at the beginning, run by a person who was actively hostile to any feedback he didn't like. There were multiple highly critical bugs whose patches weren't backported ('just update' as if it's that easy with the sprawling beast that is systemd).
> not properly testing packages
What do you mean? I only recall one popular instance of an issue with Ubuntu packages, and it's when they released a major upgrade to Samba because backporting a critical security fix to the previous major version, the one that came with the distro originally, was too hard (in their words), which ended up breaking Samba for a bunch of people.
Ubuntu isn't "flakey". It makes a different tradeoff compared to RHEL - slightly newer version of stuff for slightly less stability. For many orgs that's preferable to obsolete 10 year old versions of most software for amazing stability.
Talk about hyperboles. RHEL 8 is from 2018 and has had considerably more updates than Ubuntu 18.04. In fact some packages might even be newer than what is in 20.04.
Red Hat introduced modules with RHEL8, meaning they can easily make available more up-to-date versions of software if necessary. It's still not bleeding edge, but eg. PostgreSQL 13 was released on 2020-09-24 and that's available with support in RHEL 8 as a module. Similarly, you have eg. PHP 8, Ruby 3.0 and other software released after RHEL 8's initial release.
Modules are much nicer to use than the previous software collection system because they actually replace the "original" package, so it's just a straight version upgrade without having to worry about fixing configuration files etc. if it's compatible.
The trade off of modules, which it pays to be aware of, is that they have different support lifecycles to the distro they are in. They publish a list that is updates as new modules are released.[1] what this means in practice is that some appstream modules may only have a year or two of support, while other may have until distribution release EOL. For example, in RHEL 8 PHP 7.4 is supported until 2029, almost 9 years after released as a module, but earlier 7.x versions and 8.0 which are also modules have lifecycles that range from 18-24 months.
There's a lot of flexibility in this to support both those that need newer versions of things as well as older stable versions, just be aware and choose and plan accordingly.
I don't understand, Ubuntu releases new releases every 6 months? If you want the latest Docker/Linux kernel/whatever, just download the latest Ubuntu. They may be behind on one or two packages but that's it.
Normally you could also grab the releases from the source directly and let the upstream source figure out compatibility for you. However, it seems like the folks over at Podman have discontinued their external repository, so I guess they don't care about bringing new versions to Ubuntu either.
> You'd think an entity the size of RedHat trying to take the reins from Docker would understand that this is an investment they have to make to make it a first-class replacement.
Nah, red hat probaby cares very little about that.
Red hat probably cares about delivering the best it can for its users (red hat, centos and fedora users).
Podman probably has no explicit goal of replacing docker, it only has the goal of providing a workstation container management implementation. Which might happen to be an awesome substitute for docker.
I don't think Podman developers are even really strongly integrated into Red Hat OS development goals. They create something that can be packaged and works on most Linux distros. I do think that Podman does try to be a replacement for docker though, and that is why they have the podman-docker layer. They may not outright say it because of the Docker licensing fiasco, but with Podman Desktop it is clear that they are pushing for an alternative to Docker.
I can't comment on Podman specifically, but Red Hat's approach to the projects that they choose to focus on in the "Red Hat ecosystem" seems to be to just do development in upstream first and then whoever is working on the distros packages it for Fedora and/or RHEL. I don't see why they should put resources into packaging in their upstream projects; it doesn't seem to me that they're hostile to other distros packaging their software.
Packaging is a distro problem, not an upstream one, though upstreams should of course work with distro maintainers to make packaging frictionless.
They are frantically working to be the alternative to docker.
Just give them enough time and it will run in Ubuntu just fine.
Obviously it will happen after they get it feature complete on their own OS.
Given the amount of work need to achieve feature parity with docker (which I suppose Docker Inc tough was it's moat), they have no viable competition right now and so this strategy makes sense.
> I was looking at this just last night and the story with Ubuntu is horrible. That's essentially going to be stuck on v3.4 for ten years because it's "a lot of work" to get into that distribution.
Ah shit, thanks for the heads up, I'm in the process of upgrading (actually, on a test snapshot) a client's Ubuntu 14.04 LTS to 22.04 LTS and migrating everything to containers running on podman.
Their IT department is crap so instead of creating a new VPS on top of RHEL or something and switching the DNS entry, I have to stay on Ubuntu Server, which I hate.
Does anyone have experience “migrating” from docker to podman on a somewhat large scale? I am starting a new job soon in a DevOps role where the devs use both podman and docker and although the resulting images are the same, using two tools where one would suffice does not seem optimal.
For context, it is a medium-sized (~40 devs) commodities trading company running some hundred applications in azure k8s but considering moving to on-prem k8s.
If you want to do a straight migration you can start the socket unit and expose it via the DOCKER_HOST environment variable and use the standard docker tooling or use podman-docker to make it seamless. As long as you make no other changes everything should just work.
Where people frequently run into issues is when they want to utilize the rootless feature, and that definitely requires adjustments and work.
I never used it, but have seen separate "rootless" images many times. Seems like you have to build a separate image for rootless Docker? podman runs regular images under an unprivileged user just fine.
You don't need a separate image. You can run a script to install a rootless systemd service that starts the Docker daemon. You just point the Docker client to that rootless daemon's socket and do things as before.
You'll need some configuration if you want to expose system ports (below 1024).
Using DOCKER_HOST and the podman-docker package (Debian and Ubuntu) also makes it really easy to use docker-compose which is more mature and feature-complete than podman-compose.
I agree, rootless was also the main blocker when I attempted my last migration in June. My goal was to run all my Docker containers as rootless Podman containers through systemd. The first service I ported was single-container, which made it easier. Still quite some work (like manually adjusting the podman-generated unit file, tracking down some weird-looking error messages, etc), but doable. Once you get to multi-container setups, I ran into even more issues (IIRC related to the linking of containers through a network, while staying rootless), so I gave up for the moment and started a Docker daemon for the rest of the services...
If you’re on regular docker and would like to go podman for rootless, you can take the middle step of migrating to rootless docker. Essential step, honestly - it’s significant work to get rootless working (but definitely worth it)
Haven’t done so myself yet, but rootless docker to podman ought to be more or less a direct switch
We got burned hard last year around GPUs (ai) and compose being not ready for enterprise settings for technical and redhat politics reasons, which are top use cases in the wild. Networking also looked problematic. We hadn't tried k8s for podman, which likely has its own issues.
So defaulting to docker would be the less risky bet and relegate podman to opt-in by trailblazers.
Podman core development is impressive, so frustrating to see the broader ecosystem issues.
Things like storage or traffic routing become uncertain/hacky at best or at worst just straight up don’t work. I’ve literally seen many on-prem k8s deployments where there was no support for persistent volume claims because of storage - whether the enterprise storage solution wasn’t supported by k8s or because there was no way to guarantee storage (HDDs being a capital expense and therefore purchased ahead of time).
Examples of friction off the top of my head:
* storage - who knows if there’s actually enough space to provision and attach a volume
* exposing a service externally
* load balancers managed by a different team (such as networking) and their process or tech environment not setup to support k8s
* certificates need to be requested and signed by approved enterprise certificate authority
* network security restrictions - lengthy process for firewall change requests or similar to expose a service externally (also related to point above on certificates)
I’ve also seen silly deployments where k8s runs inside VMware virtual machines - why not just run k8s directly on hardware if on-prem? Virtual machines just add a ton of unnecessary and expensive overhead.
Personally, my preferred way of doing k8s:
1. GKE (Google Kubernetes Engine) - the easiest and cheapest managed option
2. kops
Hot take here but I personally think EKS is convoluted and complicated as hell. I’ve not used AKS but in the past I’ve seen limitations such as network configuration that became problematic - couldn’t route traffic properly because IP space overlapped with corporate IP ranges already used by VPN and/or existing vnets, though it’s been a couple years since I first saw this specific example.
Not a Podman or Docker-specific question/complaint, but is anyone working on a solution around UID/GID remapping between container and host? Not the "solution" offered by user namespaces, because those can only do a 1:1 remap, but an "anything any user writes inside the container is owned by the host user".
In other words, if host UID 1000 runs a container that has UID 100, 101, 102 and they all write files to a mounted volume, it would be great if all file writes were attributed to the host UID of 1000.
Instead, if user namespaces are configured correctly they would attribute to UID 1000, 1001, 1002 on host.
fuse-overlay has some support for this, but it is fairly experimental. Basically it allows mapping of all files to a single UID, and then stores container owners/permission in XAttrs. When fuse-overlayfs exposes the image to the container process it exposes the xattr user/group and permissions.
I love Podman so much. It's seamless integration with systemd is a killer feature, as well as being able to generte Kubefiles. My main problem used to be feature parity with docker networking, but that seems to be mostly resolved since 4.0.
Trying to convince my team to use podman instead of docker (even though I've been using it myself primarily for months without issues), that's another story.
I love this, but one challenge I've found is podman kube play had issues with named volumes and `keep-id`. Hopefully those are fixed in this release. I guess I'll find out shortly.
I really like the ideas behind podman but they were not able to get enough feature parity with docker fast enough for my org. We use rancher desktop now for local development with Docker underneath. It would be cool if they added podman support in Rancher Desktop.
Minikube supports rootless Podman. Both are Kubernetes projects, but I find that Minikube appears to be getting more maintenance and is easier to mimic a production cluster for testing.
Darren here - PM for GitLab Runner. At this time, we aren't considering adding Podman to GitLab SaaS Runners on Linux. I created this issue, so please add comments there so we can continue the discussion.
I wouldn't count on it unless you add some extra repo. RHEL releases are historically behind when it comes to software, which is claimed to provide more stability, but often older kernels in RHEL simply will not support the new underlying functionality. I am hopeful that if Podman ever has to make the decision to adapt at a modern pace with new software releases, or constantly try to backport fixes and code for old software, that they'll choose the modern approach.
RHEL 8.6 upgraded podman from v3.3 to v4.0.2, so v4.2 is not outside the realm of possibility in the next release. Since podman development is driven by Red Hat, I feel that podman upgrades get quite some leeway... Very impatient to test out the play kube functionality managed by systemd.
First, some software is updated frequently in RHEL, including podman. RHEL 8 was released in 2019 and it has podman 4.0.2 from earlier this year.
Second, even software that isn't updated often, or at all, in the base system might have newer releases available as modules. For example there are recent versions of Python in RHEL 8, with just the basic runtime so that you can use pip to install more packages.
Third, the RHEL kernel is updated much more than the corresponding LTS releases. The RHEL 8 kernel is closer to 5.15-ish than to the nominal 4.18 release from which it was forked. Pace for backports has slowed down a bit, but there's interest in keeping the kernel up to date because people are running RHEL 9 containers on the RHEL 8 kernel.
Given the module streams functionality in the package system in RHEL 8+, it's fairly plausible that it would be introduced using that path.
(Currently, there are 5 'container-tools' module streams listed, with podman versions including 1.0, 1.6, 3.0, 4.0 in stable streams, and 4.0.2 in the rolling stream.)
That is fair. I don't use Red Hat anymore really and their marketing and naming around modules was just confusing at the time. They took a common component of package management called repositories and made it confusing.
Big fan of Podman.
Technically every time I have tried to push it into my production workflow I've hit some snags, mostly around networking and some volume stuff but those snags are getting chipped away each release. The last time I did a strong test flight was around 3.X, so probably about time to try again, 4.0 was a big release.
I like the integration with systemd and bringing "pods" out of k8. I like the "they're just processes" philosophical perspective and more "linux tech" focus of the team - i.e. cgroups v2 exists, lets use it. I would like to see some minor UI stuff such as compressing buildah stage output like docker buildx does but it's understandable why that isn't there (yet?).
I think my only remaining quibble is getting true remote addresses when using rootless networking.
You can use podman with buildkit, it's just kind of a pain because you have to do it manually. It is missing some nice to have feature integrations with buildkit. The instructions are fairly clear: https://github.com/moby/buildkit
This really just means using rootless libcontainer (i.e. the Docker machinery) inside podman, it's not a native integration.
I haven't tried it out, but shouldn't you be able to detect the true remote address by using a socket that has been passed in via socket activation?
The remote address is available when running a socket-activated container with rootless Podman. I verified it in a test.
I like this Podman feature:
Socket activation of containers
Advantages:
- Faster network. Rootless Podman will run with native network speed. Normally rootless Podman runs with reduced network speed due to the performance penalty that comes from using slirp4netns.
- Improved security as you can disable the ability to establish outgoing connections with --network=none. The container can still communicate over the socket-activated socket with a client that has connected via the internet.
I contributed a Podman socket activation tutorial: https://github.com/containers/podman/blob/main/docs/tutorial...
and I wrote two blogs about the security advantages
https://www.redhat.com/sysadmin/socket-activation-podman
https://www.redhat.com/sysadmin/podman-systemd-limit-access
Docker does not support socket activation of containers. (Docker only supports socket activation of the Docker daemon)
Edit: A clarification about the network speed. The improved speed is about the communication that passes over the socket-activated socket. This communication does not pass through slirp4netns so it has the same performance characteristics as the normal network on the host.
> - Faster network. Rootless Podman will run with native network speed. Normally rootless Podman runs with reduced network speed due to the performance penalty that comes from using slirp4netns.
Did they fix the IP propagation issue with Rootless networking ? It makes it largely useless when the Proxy is also a container.
on Docker you can force it to use Slirp, but its slow and doesn't support IPv6.
You could detect the source IP address by using the command-line option:
--net=slirp4netns:port_handler=slirp4netns
See https://github.com/containers/podman/discussions/10472#discu...
Shouldn't it also be possible to detect the source IP address if you use socket activation? (I haven't tried it out, though).
But that's using Slirp, which has its own problems.
Ok, I understand your concern about Slirp.
Regarding the other idea: I've now tested it and verified that it works. The remote address is available when running a socket-activated container with rootless Podman.
Docker can’t really support Socket Activation for containers because it is build around the docker daemon that fulfills similar tasks as systemd. They could only copy the concept to dockerd.
Yeah, Docker the company is funky.
Docker the software is also funky.
That's neat and quite a unixy solution but still fairly limiting. Are there any plans for a hybrid solution where most of podman runs as non-root via userns but invokes a suid helper to setup the network?
Not exactly what you are asking for but there is a Systemd feature request to add Connect= setting to service unit files. https://github.com/systemd/systemd/issues/23067#issuecomment... (That could a be cool feature)
Also interesting would be to fix the security considerations of using bypass4netns:
"However, it is probably possible to connect to host loopback IPs by exploiting TOCTOU of struct sockaddr * pointers."
There seems to be an implementation idea for how the problem could be fixed:
https://github.com/rootless-containers/bypass4netns/issues/2...
Podman 4 supports netavark which to my understanding should solve a lot of the networking issues with slirp4netns
Not for rootless containers unless there’s a development I’m unaware of.
Yeah it works for rootless now too or at least that is my understanding.
Does this work in systemd-less distros like the most common WSL2 ones? I tried to fake Podman to use in a devcontainer but it complained of a missing `/var/run/docker.sock`.
I was looking at this just last night and the story with Ubuntu is horrible. That's essentially going to be stuck on v3.4 for ten years because it's "a lot of work" to get into that distribution.
You'd think an entity the size of RedHat trying to take the reins from Docker would understand that this is an investment they have to make to make it a first-class replacement.
I also installed it on Windows to see how the WSL engine works but now it conflicts with my existing v3 Podman installation on Ubuntu 20.04 in WSLv2 so I guess I'm out of luck.
Also may be of interest to people here but Podman desktop had a release yesterday. It's pretty primitive and I couldn't get it to work to use my existing auth.json but it's there.
It was a pretty frustrating experience when all I wanted was to be able to "podman login" to a local repository so Jib would pull down base layers correctly.
Red Hat has no official affiliation with Canonical who make Ubuntu.
If you want to test podman you 'll have better luck using an OS from the Fedora ecosystem where Red Hat has affiliations and is actively contributing.
Since you mentioned Windows I 'd suggest trying something like this [1] or this [2]
[1]: https://github.com/yosukes-dev/FedoraWSL [2]: https://github.com/WhitewaterFoundry/Fedora-Remix-for-WSL
Disclaimer. I am not using Windows to test above solutions anymore. More than a year ago I used [2] but from a casual look maybe [1] is better now.
If you install the podman package via Scoop, you'll get a `podman` client shim, and then if you run `podman machine init` it'll automatically set up a WSL instance running Fedora with Podman set up and relay the necessary sockets for you so that running `podman ps` or whatever on Windows Just Works™.
Then if you want to can run `wsl -d podman-machine-default` to log into the distro as normal. You can also copy the distro, import/export/register it as usual if you want a clone unaffiliated with the podman package per se.
What the official Windows Podman installer, also official Podman Desktop that installs Podman and scoop's Podman package should really do is offer to integrate with your existing WSL2 distro like Docker Desktop does.
The way that Docker Desktop integrates is just one extra step away from `podman machine init`, since Docker Desktop just does the same socket relaying stuff for your existing WSL distros as it does for Windows. The actual Docker daemon only runs on the special distro that the tool (Docker Desktop or `podman machine` or whatever) sets up.
I assume that Podman Desktop does that but idk because I don't use it. Rancher Desktop also works this same way.
In all cases, the integration with existing distros happens inside the GUI app. Maybe I'll check tomorrow whether Podman Desktop offers comparable integration.
But yeah, it's a good integration to have because the native Windows CLI experience is still so impractical and clunky that many developers end up setting up a pet distro in WSL and pretty much living in it as their default terminal session. Good integration with cmd.exe or (pwsh.exe running under Windows Terminal, for that matter) is cool but it doesn't mean much to someone who does all their work in an Ubuntu WSL VM or whatever.
Personally I followed the rootfs way of installing Fedora to WSL2. It was simple enough and worked fine (including podman). I found no reason to use external tools / scripts / modified distros.
Sadly, some anti-cheat tools in games still refuse to work with WSL2 (they hate Hyper-V, I guess it's been used as attack vector), so back to VMware Player on my personal workstation and using terminal to open Linux shell.
I don't think GP was suggesting any official affiliation between Red Hat and canonical. I think they were making a point that there's a lot of potential users on Ubuntu who might switch to podman if it were available. When trying to establish a project, user acquisition is a critically important part.
I agree with them. I think Red Hat should be making effort to get podman working well in Ubuntu (well, Debian but would benefit Ubuntu). Although it's very possible that Red Hat is trying and have met resistance. Canonical wants for a very different direction and it wouldn't surprise me at all if they were throwing road blocks in the way (or at least, doing nothing to remove the road blocks).
There's also Arch, or basically anything else besides Ubuntu. Podman isn't the only thing that is chronically out of date on it. Ubuntu has definitely lived long enough to become the villain.
I wonder why Fedora doesn't provide an official WSL package on the Microsoft Store as other distros do. My guess is that they feel that the WSL kernel and init diverge too far from the Fedora kernel and systemd. Can anyone from the Fedora project comment on this?
The reason is that Microsoft wanted indemnification agreement and Fedora refused to provide it.
More here: https://lists.fedoraproject.org/archives/list/legal@lists.fe...
No, it's not kernel related.
It's because Fedora strives to be open-source and free software only. WSL isn't completely libre so they can't support it officially and neither is the Windows Store. There's third party Fedora images for WSL you can install.
https://lists.fedoraproject.org/archives/list/legal@lists.fe...
What would be the benefit for Fedora, given that it seems like non-trivial effort?
Looks like Debian is doing that work, but only in experimental for now...
https://packages.debian.org/source/experimental/libpod
Some context/discussion and an experimental ppa for Ubuntu:
https://github.com/containers/podman/issues/14302#issuecomme...
I used Arch and have access the pretty much the latest release of Podman anytime. Your qualms with Ubuntu packaging don't make a lot of sense. Any distro has requirements to become an official packager. Canonical and Ubuntu have been pushing LXC/LXD as their container solution. If they wanted the latest version of Podman then their packagers can build and package it, or someone else can and create a repository or PPA.
Also, I was able to get it to work on Windows fine. Maybe try removing your existing install and creating a new one.
> I was looking at this just last night and the story with Ubuntu is horrible. That's essentially going to be stuck on v3.4 for ten years because it's "a lot of work" to get into that distribution.
They used to provide relatively recent builds in their kubic repos. Unfortunately, for some reason, they decided to discontinue it[0]. They mentioned some CVEs or something in some issues raised around this, but to me that means pushing a new version/build and not discontinuing it.
Anyway, one of the members of the Containers org provides unstable kubic repos[1][2] for non RH systems. Unfortunately, this includes RCs, and non-stable versions, which is fine to get bleeding edge, but I'd rather just have the stable versions.
Due to the above, I've written some scripts to build deb packages for all the latest stable versions. So hopefully you can simply download the deb from GH releases[3] and then `dpkg -i *.deb && apt-get install -f`.
[0] https://podman.io/blogs/2022/04/05/ubuntu-2204-lts-kubic.htm...
[1] https://github.com/containers/podman/issues/14302#issuecomme...
[2] https://build.opensuse.org/project/show/devel:kubic:libconta...
[3] https://github.com/hoshsadiq/podman-deb/releases
Ubuntu has nothing to do with Podman / Redhat. Ubuntu also has a terrible track record of not aligning where it makes sense with other distorts and not properly testing packages - it's just a bit of a flaky distro in general.
> . Ubuntu also has a terrible track record of not aligning where it makes sense with other distorts
You mean not aligning with Red Hat and what they're pushing on everyone else. Ubuntu is on a shorter release cycle compared to Debian so they're usually the first non-Red Hat distro with new stuff. Systemd vs Upstart, Unity vs GNOME (3?), etc.
They try to do new stuff, and there's nothing wrong with that. Not everyone should blindly follow RH's lead. Systemd was objectively shit at the beginning, run by a person who was actively hostile to any feedback he didn't like. There were multiple highly critical bugs whose patches weren't backported ('just update' as if it's that easy with the sprawling beast that is systemd).
> not properly testing packages
What do you mean? I only recall one popular instance of an issue with Ubuntu packages, and it's when they released a major upgrade to Samba because backporting a critical security fix to the previous major version, the one that came with the distro originally, was too hard (in their words), which ended up breaking Samba for a bunch of people.
Ubuntu isn't "flakey". It makes a different tradeoff compared to RHEL - slightly newer version of stuff for slightly less stability. For many orgs that's preferable to obsolete 10 year old versions of most software for amazing stability.
> obsolete 10 year old versions
Talk about hyperboles. RHEL 8 is from 2018 and has had considerably more updates than Ubuntu 18.04. In fact some packages might even be newer than what is in 20.04.
Red Hat introduced modules with RHEL8, meaning they can easily make available more up-to-date versions of software if necessary. It's still not bleeding edge, but eg. PostgreSQL 13 was released on 2020-09-24 and that's available with support in RHEL 8 as a module. Similarly, you have eg. PHP 8, Ruby 3.0 and other software released after RHEL 8's initial release.
Modules are much nicer to use than the previous software collection system because they actually replace the "original" package, so it's just a straight version upgrade without having to worry about fixing configuration files etc. if it's compatible.
The trade off of modules, which it pays to be aware of, is that they have different support lifecycles to the distro they are in. They publish a list that is updates as new modules are released.[1] what this means in practice is that some appstream modules may only have a year or two of support, while other may have until distribution release EOL. For example, in RHEL 8 PHP 7.4 is supported until 2029, almost 9 years after released as a module, but earlier 7.x versions and 8.0 which are also modules have lifecycles that range from 18-24 months.
There's a lot of flexibility in this to support both those that need newer versions of things as well as older stable versions, just be aware and choose and plan accordingly.
1: https://access.redhat.com/support/policy/updates/rhel-app-st...
But you have Ubuntu 22.04 if you want newer versions of everything.
You also have RHEL 9 then.
Maybe, but plenty of RHEL8 packages still package versions from 10+ years ago.
It was forked from Fedora 5 years ago and it has versions from 10+ years ago?
Can you name a few specifically?
I don't understand, Ubuntu releases new releases every 6 months? If you want the latest Docker/Linux kernel/whatever, just download the latest Ubuntu. They may be behind on one or two packages but that's it.
Normally you could also grab the releases from the source directly and let the upstream source figure out compatibility for you. However, it seems like the folks over at Podman have discontinued their external repository, so I guess they don't care about bringing new versions to Ubuntu either.
> You'd think an entity the size of RedHat trying to take the reins from Docker would understand that this is an investment they have to make to make it a first-class replacement.
Nah, red hat probaby cares very little about that.
Red hat probably cares about delivering the best it can for its users (red hat, centos and fedora users).
Podman probably has no explicit goal of replacing docker, it only has the goal of providing a workstation container management implementation. Which might happen to be an awesome substitute for docker.
I don't think Podman developers are even really strongly integrated into Red Hat OS development goals. They create something that can be packaged and works on most Linux distros. I do think that Podman does try to be a replacement for docker though, and that is why they have the podman-docker layer. They may not outright say it because of the Docker licensing fiasco, but with Podman Desktop it is clear that they are pushing for an alternative to Docker.
I can't comment on Podman specifically, but Red Hat's approach to the projects that they choose to focus on in the "Red Hat ecosystem" seems to be to just do development in upstream first and then whoever is working on the distros packages it for Fedora and/or RHEL. I don't see why they should put resources into packaging in their upstream projects; it doesn't seem to me that they're hostile to other distros packaging their software.
Packaging is a distro problem, not an upstream one, though upstreams should of course work with distro maintainers to make packaging frictionless.
They are frantically working to be the alternative to docker.
Just give them enough time and it will run in Ubuntu just fine.
Obviously it will happen after they get it feature complete on their own OS.
Given the amount of work need to achieve feature parity with docker (which I suppose Docker Inc tough was it's moat), they have no viable competition right now and so this strategy makes sense.
Could you file a bug to podman desktop https://github.com/containers/podman-desktop/issues about your auth.json file / "podman login" ?
Ubuntu just imports the package from Debian, there it's also still on 3.4 and will be removed by the end of the month
https://tracker.debian.org/pkg/libpod
> I was looking at this just last night and the story with Ubuntu is horrible. That's essentially going to be stuck on v3.4 for ten years because it's "a lot of work" to get into that distribution.
Ah shit, thanks for the heads up, I'm in the process of upgrading (actually, on a test snapshot) a client's Ubuntu 14.04 LTS to 22.04 LTS and migrating everything to containers running on podman.
Their IT department is crap so instead of creating a new VPS on top of RHEL or something and switching the DNS entry, I have to stay on Ubuntu Server, which I hate.
Yup that's pretty annoying. Docker offers official repos to always have the latest version, wish podman will offer something similar.
Let me guess, Podman desktop is an Electron app.
Does anyone have experience “migrating” from docker to podman on a somewhat large scale? I am starting a new job soon in a DevOps role where the devs use both podman and docker and although the resulting images are the same, using two tools where one would suffice does not seem optimal.
For context, it is a medium-sized (~40 devs) commodities trading company running some hundred applications in azure k8s but considering moving to on-prem k8s.
If you want to do a straight migration you can start the socket unit and expose it via the DOCKER_HOST environment variable and use the standard docker tooling or use podman-docker to make it seamless. As long as you make no other changes everything should just work.
Where people frequently run into issues is when they want to utilize the rootless feature, and that definitely requires adjustments and work.
Rootless is definitely one of the major selling points for using podman over docker, especially for local development.
Isn't there support for rootless docker these days as well?
I never used it, but have seen separate "rootless" images many times. Seems like you have to build a separate image for rootless Docker? podman runs regular images under an unprivileged user just fine.
You don't need a separate image. You can run a script to install a rootless systemd service that starts the Docker daemon. You just point the Docker client to that rootless daemon's socket and do things as before.
You'll need some configuration if you want to expose system ports (below 1024).
Using DOCKER_HOST and the podman-docker package (Debian and Ubuntu) also makes it really easy to use docker-compose which is more mature and feature-complete than podman-compose.
I agree, rootless was also the main blocker when I attempted my last migration in June. My goal was to run all my Docker containers as rootless Podman containers through systemd. The first service I ported was single-container, which made it easier. Still quite some work (like manually adjusting the podman-generated unit file, tracking down some weird-looking error messages, etc), but doable. Once you get to multi-container setups, I ran into even more issues (IIRC related to the linking of containers through a network, while staying rootless), so I gave up for the moment and started a Docker daemon for the rest of the services...
If you’re on regular docker and would like to go podman for rootless, you can take the middle step of migrating to rootless docker. Essential step, honestly - it’s significant work to get rootless working (but definitely worth it)
Haven’t done so myself yet, but rootless docker to podman ought to be more or less a direct switch
We got burned hard last year around GPUs (ai) and compose being not ready for enterprise settings for technical and redhat politics reasons, which are top use cases in the wild. Networking also looked problematic. We hadn't tried k8s for podman, which likely has its own issues.
So defaulting to docker would be the less risky bet and relegate podman to opt-in by trailblazers.
Podman core development is impressive, so frustrating to see the broader ecosystem issues.
On-prem k8s is almost always a bad idea.
Things like storage or traffic routing become uncertain/hacky at best or at worst just straight up don’t work. I’ve literally seen many on-prem k8s deployments where there was no support for persistent volume claims because of storage - whether the enterprise storage solution wasn’t supported by k8s or because there was no way to guarantee storage (HDDs being a capital expense and therefore purchased ahead of time).
Examples of friction off the top of my head:
* storage - who knows if there’s actually enough space to provision and attach a volume
* exposing a service externally
I’ve also seen silly deployments where k8s runs inside VMware virtual machines - why not just run k8s directly on hardware if on-prem? Virtual machines just add a ton of unnecessary and expensive overhead.Personally, my preferred way of doing k8s:
1. GKE (Google Kubernetes Engine) - the easiest and cheapest managed option
2. kops
Hot take here but I personally think EKS is convoluted and complicated as hell. I’ve not used AKS but in the past I’ve seen limitations such as network configuration that became problematic - couldn’t route traffic properly because IP space overlapped with corporate IP ranges already used by VPN and/or existing vnets, though it’s been a couple years since I first saw this specific example.
1. Just pay the vendor. Diy csi is also an option
2. MetallB. Not sure how cert thing applies here…
3. vmware based vms are to manage underlying node image so that you dont have to mess with ipmi/ipxe that often. They can also make networking easier
Not a Podman or Docker-specific question/complaint, but is anyone working on a solution around UID/GID remapping between container and host? Not the "solution" offered by user namespaces, because those can only do a 1:1 remap, but an "anything any user writes inside the container is owned by the host user".
In other words, if host UID 1000 runs a container that has UID 100, 101, 102 and they all write files to a mounted volume, it would be great if all file writes were attributed to the host UID of 1000.
Instead, if user namespaces are configured correctly they would attribute to UID 1000, 1001, 1002 on host.
fuse-overlay has some support for this, but it is fairly experimental. Basically it allows mapping of all files to a single UID, and then stores container owners/permission in XAttrs. When fuse-overlayfs exposes the image to the container process it exposes the xattr user/group and permissions.
You can use ACLs for this.
Anyone know a good resource for understanding the fundamentals of containers, in particular, how it persists state?
Also maybe how different products (kubernetes, podman, docker) fit into this space?
Thanks in advance!
Maybe articles like https://www.infoq.com/articles/build-a-container-golang/ could help? Basically learn how to build a simple one.
I love Podman so much. It's seamless integration with systemd is a killer feature, as well as being able to generte Kubefiles. My main problem used to be feature parity with docker networking, but that seems to be mostly resolved since 4.0.
Trying to convince my team to use podman instead of docker (even though I've been using it myself primarily for months without issues), that's another story.
The change log is endless ! I'm specifically interesting by the new support of Gitlab Runner !
This version of Podman is blazing!
This version contains the answer to everything.
I love this, but one challenge I've found is podman kube play had issues with named volumes and `keep-id`. Hopefully those are fixed in this release. I guess I'll find out shortly.
Very cool to see Podman Desktop. This seems a replacement for Docker Desktop that is now organizations with more than 250 employees or more than $10 million in revenue https://www.infoq.com/news/2021/09/docker-desktop-subscripti...
Also cool to see GitLab Runner support as the first feature mention.
It is, however, missing the 1-click Kubernetes feature of Docker Desktio.
I really like the ideas behind podman but they were not able to get enough feature parity with docker fast enough for my org. We use rancher desktop now for local development with Docker underneath. It would be cool if they added podman support in Rancher Desktop.
Read the release log for Funny Number jokes, came away disappointed
This looks great. Now if only KinD supported Podman!
Minikube supports rootless Podman. Both are Kubernetes projects, but I find that Minikube appears to be getting more maintenance and is easier to mimic a production cluster for testing.
kind works fine with podman: KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster https://kind.sigs.k8s.io/docs/user/rootless/
https://kind.sigs.k8s.io/docs/user/rootless/
I'm aware, wasn't able to make it work the last time. The support is still experimental.
Is anybody able to install it on MacOs with brew? Seems to be broken:
„Error: podman: Failed to download resource „podman_bottle_manifest“
I just updated from 4.1 to 4.2 on my M1 using Brew without an issue.
Now if Gitlab.com only supported leveraging Podman in their runners.
Darren here - PM for GitLab Runner. At this time, we aren't considering adding Podman to GitLab SaaS Runners on Linux. I created this issue, so please add comments there so we can continue the discussion.
https://gitlab.com/gitlab-org/gitlab/-/issues/370522
Podman is supported as a gitlab-runner now, as I understand it.
https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29108
Wasn't podman mostly C at some point? Did they rewrite it in Go?
I've did a bit of reverse searching. They've used Go all the time[1]
[1](https://github.com/containers/podman/tree/3d0100bb44834d079a...)
They were first to push crun[1], in place of runc, that is your C code in the stack :-)
[1] https://github.com/containers/crun
Will this eventually make it to RHEL8? Today it's at 4.0.2.
I wouldn't count on it unless you add some extra repo. RHEL releases are historically behind when it comes to software, which is claimed to provide more stability, but often older kernels in RHEL simply will not support the new underlying functionality. I am hopeful that if Podman ever has to make the decision to adapt at a modern pace with new software releases, or constantly try to backport fixes and code for old software, that they'll choose the modern approach.
RHEL 8.6 upgraded podman from v3.3 to v4.0.2, so v4.2 is not outside the realm of possibility in the next release. Since podman development is driven by Red Hat, I feel that podman upgrades get quite some leeway... Very impatient to test out the play kube functionality managed by systemd.
This is not really correct.
First, some software is updated frequently in RHEL, including podman. RHEL 8 was released in 2019 and it has podman 4.0.2 from earlier this year.
Second, even software that isn't updated often, or at all, in the base system might have newer releases available as modules. For example there are recent versions of Python in RHEL 8, with just the basic runtime so that you can use pip to install more packages.
Third, the RHEL kernel is updated much more than the corresponding LTS releases. The RHEL 8 kernel is closer to 5.15-ish than to the nominal 4.18 release from which it was forked. Pace for backports has slowed down a bit, but there's interest in keeping the kernel up to date because people are running RHEL 9 containers on the RHEL 8 kernel.
Given the module streams functionality in the package system in RHEL 8+, it's fairly plausible that it would be introduced using that path.
(Currently, there are 5 'container-tools' module streams listed, with podman versions including 1.0, 1.6, 3.0, 4.0 in stable streams, and 4.0.2 in the rolling stream.)
That is fair. I don't use Red Hat anymore really and their marketing and naming around modules was just confusing at the time. They took a common component of package management called repositories and made it confusing.
I'd personnally expect in RHEL 8.7, as 8.6 already been shipping podman v4.0
How does Podman Desktop compare to Rancher Desktop?
No Kubernetes cluster, which is kind of the whole point of RD.