Debian and our frenemies of containers and userland repos

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Debian and our frenemies of containers and userland repos

"Yao Wei (魏銘廷)"-2
Hi,

Following to Mo Zhou's thread of Conda and Debian, it reminds me that,
could Debian reduced into a "proof of concept" as an operating system
with collection of apps and things composed of completely free software,
as more and more software repositories are moving away from the free
software repositories like Debian, and userland repositories and app
containers becomes more prominent and easier to access.  I feel the fear
when I was in a flatpak session in DebConf17.

It can be a "solid base" of container images and barebone systems, but
the days are numbered as operating systems as free and focused on its
mission (like Google COOS, Yocto, Alpine etc.) is evolving steady.

Could it be a disaster for us?  And more importantly, do users care?

Best regards,
Yao Wei

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Kyle Edwards
On Thu, 2019-07-11 at 23:25 +0800, Yao Wei wrote:

> Hi,
>
> Following to Mo Zhou's thread of Conda and Debian, it reminds me
> that,
> could Debian reduced into a "proof of concept" as an operating system
> with collection of apps and things composed of completely free
> software,
> as more and more software repositories are moving away from the free
> software repositories like Debian, and userland repositories and app
> containers becomes more prominent and easier to access.  I feel the
> fear
> when I was in a flatpak session in DebConf17.
>
> It can be a "solid base" of container images and barebone systems,
> but
> the days are numbered as operating systems as free and focused on its
> mission (like Google COOS, Yocto, Alpine etc.) is evolving steady.
>
> Could it be a disaster for us?  And more importantly, do users care?
>
> Best regards,
> Yao Wei

I think that there will always be some demand for APT-style packages as
opposed to containers. I know that I personally am a lot more likely to
use software if I can get it from apt-get than if I have to download a
Docker image.

I have also seen this first-hand from other people. Since we launched
our Ubuntu CMake APT repository earlier this year, it has become
massively popular. It is regularly getting hundreds of downloads per
week, both from people inside our company and from external users. I
suspect it is only going to get bigger over time, especially with our
next release cycle. I get emails about it almost every day.

The demand is there. APT-style repositories aren't going away any time
soon.

Kyle

Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Enrico Weigelt, metux IT consult-3
In reply to this post by "Yao Wei (魏銘廷)"-2
On 11.07.19 17:25, Yao Wei wrote:

Hi,

> It can be a "solid base" of container images and barebone systems, but
> the days are numbered as operating systems as free and focused on its
> mission (like Google COOS, Yocto, Alpine etc.) is evolving steady.
>
> Could it be a disaster for us?  And more importantly, do users care?

I don't think so.

COOS:   just yet another special purpose distro, in that case for
         docker hosts. neither the first, nor the last one to come.
Yocto:  just yet another compile-yourself distro, focused on embeedded,
         that happens to be hyped by certain corporations.
         (for small/embedded devices, I'd really recommend ptxdist).
Alpine: yet another distro, optimized for running in small containers

BTW: the idea of building small payload/application-specific
containers/chroot's is anything but new. I've done it somewhere in
the 90th. But nowadays, these so-called "small" containers tend to be
bigger than whole machines of the 90th.

Containerization is a valid approach for some kind of workloads
(eg. specific inhouse applications) that can be easily isolated from
the rest. But it comes with the price of huge redundancies (depending
on how huge some application stacks are). And unless everybody wants
to go back of maintaining everything on his own, we still need distros.

If different applications need to deeply interact (eg. various plugin
stuff, applications calling each other, etc), containerization doesn't
help much. (eg: how can you have a pure texlive in one container and
extra things like fonts, document classes, etc, in separate ones ? :o)

The whole point about containerization isn't about packaging and
deployment of individual applications - instead it's about automatizing
the rollout of fully-configured installations.

One thing seems to be right: folks who always have been hostile towards
the whole concept of distros now have a better excuse.


--mtx

--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[hidden email] -- +49-151-27565287

Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Michael Kesper-5
Hi all,

On 22.07.19 12:38, Enrico Weigelt, metux IT consult wrote:
> COOS:   just yet another special purpose distro, in that case for
>         docker hosts. neither the first, nor the last one to come.
> Yocto:  just yet another compile-yourself distro, focused on embeedded,
>         that happens to be hyped by certain corporations.
>         (for small/embedded devices, I'd really recommend ptxdist).
> Alpine: yet another distro, optimized for running in small containers

Just a shame it seems the default for everyone and their cat to use it
as a base image.

Recent article re Python container images:
https://pythonspeed.com/articles/base-image-python-docker-images/

> Containerization is a valid approach for some kind of workloads
> (eg. specific inhouse applications) that can be easily isolated from
> the rest. But it comes with the price of huge redundancies (depending
> on how huge some application stacks are). And unless everybody wants
> to go back of maintaining everything on his own, we still need distros.
>
> If different applications need to deeply interact (eg. various plugin
> stuff, applications calling each other, etc), containerization doesn't
> help much. (eg: how can you have a pure texlive in one container and
> extra things like fonts, document classes, etc, in separate ones ? :o)
>
> The whole point about containerization isn't about packaging and
> deployment of individual applications - instead it's about automatizing
> the rollout of fully-configured installations.
Good points!

Best
Michael


signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Marc Haber-3
In reply to this post by Enrico Weigelt, metux IT consult-3
On Mon, 22 Jul 2019 12:38:36 +0200, "Enrico Weigelt, metux IT consult"
<[hidden email]> wrote:
>Containerization is a valid approach for some kind of workloads
>(eg. specific inhouse applications) that can be easily isolated from
>the rest. But it comes with the price of huge redundancies (depending
>on how huge some application stacks are). And unless everybody wants
>to go back of maintaining everything on his own, we still need distros.

Compared to a full VM, a container _is_ smaller. I am not sure whether
the difference is as huge in times where we have kernel same-page
merging though.

Do we have a build technology that uses containers instead of chroots
yet?

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Ansgar Burchardt-8
Marc Haber writes:
> Compared to a full VM, a container _is_ smaller. I am not sure whether
> the difference is as huge in times where we have kernel same-page
> merging though.
>
> Do we have a build technology that uses containers instead of chroots
> yet?

I haven't used it so far, but at least "whalebuilder" exists.

The gitlab-ci used on salsa.d.o also uses Docker containers; people also
build packages using this (mostly for testing though, see for example [1]).

Ansgar

  [1] https://salsa.debian.org/salsa-ci-team/pipeline

Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Johannes Schauer-3
In reply to this post by Marc Haber-3
Quoting Marc Haber (2019-07-24 08:17:19)
> Do we have a build technology that uses containers instead of chroots yet?

Either using any container-based autopkgtest backend (like lxc or lxd):

    $ sbuild --chroot-mode=autopkgtest --autopkgtest-virt-server=lxc

Or using the built-in "unshare" backend which uses linux user namespaces:

    $ sbuild --chroot-mode=unshare --chroot=debian-unstable.tar

The latter allows one to either directly specify a chroot tarball with the
--chroot argument or will look inside ~/.cache/sbuild for a fitting chroot
tarball.

If you also build your chroot tarballs using a tool that doesn't require
superuser privileges like mmdebstrap (or debootstrap with the patch from
#829134) then you can essentially build arbitrary packages inside arbitrary
chroots without ever becoming root or touching anything outside your home
directory, given that you at some point did "sysctl -w
kernel.unprivileged_userns_clone=1" until #898446 is fixed.

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Johannes Schauer-3
In reply to this post by Marc Haber-3
Quoting Marc Haber (2019-07-24 08:17:19)

> On Mon, 22 Jul 2019 12:38:36 +0200, "Enrico Weigelt, metux IT consult"
> <[hidden email]> wrote:
> >Containerization is a valid approach for some kind of workloads
> >(eg. specific inhouse applications) that can be easily isolated from
> >the rest. But it comes with the price of huge redundancies (depending
> >on how huge some application stacks are). And unless everybody wants
> >to go back of maintaining everything on his own, we still need distros.
>
> Compared to a full VM, a container _is_ smaller. I am not sure whether
> the difference is as huge in times where we have kernel same-page
> merging though.
You can create really small Debian chroots with mmdebstrap. In contrast to
debootstrap, you can create chroots with just all Essential:yes packages and
their dependencies (debootstrap cannot do less than Priority:required):

   $ mmdebstrap --variant=essential unstable debian-unstable.tar

and you can use the dpkg path-excluded feature to remove lots of stuff you
might not need inside a container:

   $ mmdebstrap --variant=essential \
       --dpkgopt='path-exclude=/usr/share/man/*' \
       --dpkgopt='path-include=/usr/share/man/man[1-9]/*' \
       --dpkgopt='path-exclude=/usr/share/locale/*' \
       --dpkgopt='path-include=/usr/share/locale/locale.alias' \
       --dpkgopt='path-exclude=/usr/share/doc/*' \
       --dpkgopt='path-include=/usr/share/doc/*/copyright' \
       --dpkgopt='path-include=/usr/share/doc/*/changelog.Debian.*' \
       unstable debian-unstable.tar

or even with less than the Essential:yes package set but busybox instead:

   $ mmdebstrap --variant=custom \
       --include=dpkg,busybox,libc-bin,base-files,base-passwd,debianutils \
       --setup-hook='mkdir -p "\$1/bin"' \
       --setup-hook='for p in awk cat chmod chown cp diff echo env grep less ln mkdir mount rm rmdir sed sh sleep sort touch uname; do ln -s busybox "\$1/bin/\$p"; done' \
       --setup-hook='echo root:x:0:0:root:/root:/bin/sh > "\$1/etc/passwd"' \
       --setup-hook='printf "root:x:0:\nmail:x:8:\nutmp:x:43:\n" > "\$1/etc/group"' \
       unstable debian-unstable.tar

Thanks!

cheers, josch


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Paul Wise via nm
In reply to this post by Johannes Schauer-3
On Wed, Jul 24, 2019 at 2:42 PM Johannes Schauer wrote:

> Or using the built-in "unshare" backend which uses linux user namespaces:
>
>     $ sbuild --chroot-mode=unshare --chroot=debian-unstable.tar

Do you think it is feasible to use this on some of or the majority of
Debian buildds (most run stretch right now)? There would need to be
exceptions but most packages should build correctly in this scenario.
The main exception would be the Debian installer, IIRC it needs
network to the local apt mirror during the build process.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Johannes Schauer-3
Hi,

Quoting Paul Wise (2019-07-25 05:38:49)

> On Wed, Jul 24, 2019 at 2:42 PM Johannes Schauer wrote:
> > Or using the built-in "unshare" backend which uses linux user namespaces:
> >
> >     $ sbuild --chroot-mode=unshare --chroot=debian-unstable.tar
>
> Do you think it is feasible to use this on some of or the majority of
> Debian buildds (most run stretch right now)? There would need to be
> exceptions but most packages should build correctly in this scenario.
> The main exception would be the Debian installer, IIRC it needs
> network to the local apt mirror during the build process.
it would certainly be possible.

But you'd have to ask somebody who is more knowledgable about the security
implications of such a change. There certainly is a reason why #898446 is still
open.

Furthermore, since buildds currently use the schroot backend, I guess that
buildd admins already took all necessary precautions to secure their systems
against arbitrary code running as part of the package build process. I do not
know what benefit the "unshare" backend would have for buildds.

Indeed I rather made the "unshare" backend for personal use. With it, I can
manage all my chroots in my home directory without needing sudo to create or
update them. In the light of #898446 this might give me a false sense of
security but since I'm only ever processing Debian chroots, that should be
fine. Using sudo always scares me a bit. When using the schroot backend (which
is suid root) I sometimes have funny effects like some mounts still being there
and not unmounted after I cancelled a build or some unpacked chroots still
lingering around somewhere. Or processes run as root still running somewhere. I
then have to use "sudo" to clean this up and I would like to avoid that. With
the "unshare" backend, there can be no lingering mounts or processes and I
rather "rm -r" in my home directory than somewhere in /. When using "unshare"
there will not be any lingering processes because everything is inside its own
process group and there will not be any lingering mounts. Or bugs like #833525
in principle should not happen anymore because the unshared user does not even
have access to my home directory and thus cannot delete anything that it didn't
create itself.

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Paul Wise via nm
On Thu, Jul 25, 2019 at 2:18 PM Johannes Schauer wrote:

> But you'd have to ask somebody who is more knowledgable about the security
> implications of such a change. There certainly is a reason why #898446 is still
> open.
>
> Furthermore, since buildds currently use the schroot backend, I guess that
> buildd admins already took all necessary precautions to secure their systems
> against arbitrary code running as part of the package build process. I do not
> know what benefit the "unshare" backend would have for buildds.

I think my mental model of what the "unshare" backend does was
incorrect. I didn't think it needed #898446 to be closed. I assumed it
was just like schroot except with the addition of moving all processes
run within the chroot into a separate network/process/mount/etc
namespace with no access to the host namespaces. The primary advantage
of this would be to isolate the build chroot from the network. Perhaps
schroot is the component that should be adding support for separate
network/process/mount/etc namespaces?

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Johannes Schauer-3
Hi,

Quoting Paul Wise (2019-07-26 04:31:29)

> > But you'd have to ask somebody who is more knowledgable about the security
> > implications of such a change. There certainly is a reason why #898446 is still
> > open.
> >
> > Furthermore, since buildds currently use the schroot backend, I guess that
> > buildd admins already took all necessary precautions to secure their systems
> > against arbitrary code running as part of the package build process. I do not
> > know what benefit the "unshare" backend would have for buildds.
> I think my mental model of what the "unshare" backend does was
> incorrect. I didn't think it needed #898446 to be closed. I assumed it
> was just like schroot except with the addition of moving all processes
> run within the chroot into a separate network/process/mount/etc
> namespace with no access to the host namespaces.
Your initial intuition was correct. It is like a very primitive schroot with
just enough functionality so that sbuild can build packages with it. It lacks
all the advanced features that schroot has like configuration file management
and session management and it is baked directly into sbuild so you cannot use
it without sbuild. But feel free to steal the code for your own project! Sadly
this functionality requires a horribly complicated fork/syscall dance [1] which
I also had to copy to mmdebstrap because no existing tool seemed to do it
already.

[1] https://sources.debian.org/src/mmdebstrap/0.4.1-6/mmdebstrap/#L292

> The primary advantage of this would be to isolate the build chroot from the
> network. Perhaps schroot is the component that should be adding support for
> separate network/process/mount/etc namespaces?

Yes, it should. There is this bug for it which I openend and I only wrote the
"unshare" backend for sbuild when it became clear that schroot would not add
this functionality any time soon:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=802849

But to unshare all the namespaces, even schroot would need #898446 to be fixed.

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Simon McVittie-7
On Fri, 26 Jul 2019 at 07:34:58 +0200, Johannes Schauer wrote:
> Sadly
> this functionality requires a horribly complicated fork/syscall dance [1] which
> I also had to copy to mmdebstrap because no existing tool seemed to do it
> already.

bubblewrap might do the same dance, or at least a compatible dance?
It won't be suitable for mmdebstrap and general full-system containers
(there's only one uid and one gid in the container, with everything
visible from the host system mapped to 'nobody') but it might be suitable
for more restricted uses like building a Debian package.

> But to unshare all the namespaces, even schroot would need #898446 to be fixed.

On systems that restrict user namespace creation (Debian, Arch Linux,
RHEL), bubblewrap gets round this by being setuid root. To avoid reopening
the same security vulnerabilities that would be available via unrestricted
user namespace creation, it severely limits what it is willing to do/allow
in user-supplied code in the container, in particular setting NO_NEW_PRIVS
so that the setuid bit no longer works.

schroot is also setuid root, and sbuild relies on this to set up the
build-dependencies anyway, so in principle schroot/sbuild ought to be
able to do something like this:

- preparation step (as real root, in the chroot, with networking):
    - install build-dependencies
- either run bubblewrap or reimplement it
    - build step (as ordinary user, entering the chroot as a container,
      with no networking):
        - dpkg-buildpackage
- cleanup step (as real root, in the chroot):
    - destroy session chroot, if used

Doing that internally in schroot would require it to be actively
developed, but maybe it would be feasible to have code in sbuild that
wrapped bwrap (or even the combination of unshare(1) and setpriv(1))
around (only) the actual build step?

With the Debian Policy requirements around not writing to directories other
than /tmp, /var/tmp and the build directory, this would look something like:

    bwrap \
    --unshare-all \
    --ro-bind / / \                    # or --ro-bind /var/.../my-chroot /
    --bind /tmp /tmp \                 # or --tmpfs /tmp
    --bind /var/tmp /var/tmp \         # or --tmpfs /var/tmp
    --bind /build/hello-2.10-2 /build/hello-2.10-2 \   # or wherever the build directory is kept
    --setenv TMPDIR /tmp \
    --dev-bind /dev /dev \             # or --dev /dev for a minimal version
    --proc /proc \
    --die-with-parent \
    --chdir /build/hello-2.10-2 \
    dpkg-buildpackage

This would break any package that relies on being able to run setuid
executables (such as bwrap itself), and get privileges that way, during
its build - but perhaps that's desirable, because buildd operators
probably don't want setuid to be allowed anyway, in case it can be used
to escape the chroot?

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: Debian and our frenemies of containers and userland repos

Johannes Schauer-3
Hi,

Quoting Simon McVittie (2019-07-26 09:51:24)

> schroot is also setuid root, and sbuild relies on this to set up the
> build-dependencies anyway, so in principle schroot/sbuild ought to be
> able to do something like this:
>
> - preparation step (as real root, in the chroot, with networking):
>     - install build-dependencies
> - either run bubblewrap or reimplement it
>     - build step (as ordinary user, entering the chroot as a container,
>       with no networking):
>         - dpkg-buildpackage
> - cleanup step (as real root, in the chroot):
>     - destroy session chroot, if used
>
> Doing that internally in schroot would require it to be actively
> developed, but maybe it would be feasible to have code in sbuild that
> wrapped bwrap (or even the combination of unshare(1) and setpriv(1))
> around (only) the actual build step?
>
> With the Debian Policy requirements around not writing to directories other
> than /tmp, /var/tmp and the build directory, this would look something like:
>
>     bwrap \
>     --unshare-all \
>     --ro-bind / / \                    # or --ro-bind /var/.../my-chroot /
>     --bind /tmp /tmp \                 # or --tmpfs /tmp
>     --bind /var/tmp /var/tmp \         # or --tmpfs /var/tmp
>     --bind /build/hello-2.10-2 /build/hello-2.10-2 \   # or wherever the build directory is kept
>     --setenv TMPDIR /tmp \
>     --dev-bind /dev /dev \             # or --dev /dev for a minimal version
>     --proc /proc \
>     --die-with-parent \
>     --chdir /build/hello-2.10-2 \
>     dpkg-buildpackage
>
> This would break any package that relies on being able to run setuid
> executables (such as bwrap itself), and get privileges that way, during
> its build - but perhaps that's desirable, because buildd operators
> probably don't want setuid to be allowed anyway, in case it can be used
> to escape the chroot?
this is all very interesting! Thanks for writing it up!

I will not be spending time on writing a backend using bubble wrap but I'll
accept patches if anybody would like to do that work. This could easily be done
by extending the current "sudo" chroot mode and wrapping the package build step
itself by bubblewrap. Indeed it could probably be already done today by setting
the BUILD_ENV_CMND config value to the bwrap line you posted above with the
sudo chroot mode.

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment