Control Debian image size

Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Control Debian image size

IAmANobody
Hi,

I have been searching build system and distro for my small Linux
embedded device for many weeks, the device has small resources, about
128 MB Flash and 64 MB RAM on imx6, it is a typical embedded system,
no GUI, no Desktop environment, no multiple user interfaces, the C++
applications are simply for data transfer and communication using
TCP/IP, the applications were previously running well on Debian 8
armhf, so Debian is my first choice, but the Debian image size could
be a major problem, it could be too large to blow out device resource
constraints, many people commended me to use OE or Yocto or OpenWrt
distro which image size seems much smaller than the Debian armhf. Is
it possible to control / reduce Debian armhf image size by Debian
build, or isar or some other build bools, to reach the same resource
requirement level in OE Yocto or OpenWrt?

Thank you.

Kind regards,

- jh

Reply | Threaded
Open this post in threaded view
|

Re: Control Debian image size

Brian Beattie
Check out buildroot.
http://buildroot.org


On Sun, 2019-03-03 at 10:17 +1100, JH wrote:

> Hi,
>
> I have been searching build system and distro for my small Linux
> embedded device for many weeks, the device has small resources, about
> 128 MB Flash and 64 MB RAM on imx6, it is a typical embedded system,
> no GUI, no Desktop environment, no multiple user interfaces, the C++
> applications are simply for data transfer and communication using
> TCP/IP, the applications were previously running well on Debian 8
> armhf, so Debian is my first choice, but the Debian image size could
> be a major problem, it could be too large to blow out device resource
> constraints, many people commended me to use OE or Yocto or OpenWrt
> distro which image size seems much smaller than the Debian armhf. Is
> it possible to control / reduce Debian armhf image size by Debian
> build, or isar or some other build bools, to reach the same resource
> requirement level in OE Yocto or OpenWrt?
>
> Thank you.
>
> Kind regards,
>
> - jh
>

Reply | Threaded
Open this post in threaded view
|

Re: Control Debian image size

Johannes Schauer-3
In reply to this post by IAmANobody
Hi,

Quoting JH (2019-03-03 00:17:06)

> I have been searching build system and distro for my small Linux
> embedded device for many weeks, the device has small resources, about
> 128 MB Flash and 64 MB RAM on imx6, it is a typical embedded system,
> no GUI, no Desktop environment, no multiple user interfaces, the C++
> applications are simply for data transfer and communication using
> TCP/IP, the applications were previously running well on Debian 8
> armhf, so Debian is my first choice, but the Debian image size could
> be a major problem, it could be too large to blow out device resource
> constraints, many people commended me to use OE or Yocto or OpenWrt
> distro which image size seems much smaller than the Debian armhf. Is
> it possible to control / reduce Debian armhf image size by Debian
> build, or isar or some other build bools, to reach the same resource
> requirement level in OE Yocto or OpenWrt?
it is possible to produce a Debian rootfs well below 128 MB in size if you
choose not to install locales, man pages and docs using the --path-exclude dpkg
option:

https://raphaelhertzog.com/2010/11/15/save-disk-space-by-excluding-useless-files-with-dpkg/

An easy way to create a chroot tarball with this option already set is by using
mmdebstrap (shameless plug: I'm the author) like so:

mmdebstrap --variant=apt \
    --dpkgopt='path-exclude=/usr/share/man/*' \
    --dpkgopt='path-include=/usr/share/man/man[1-9]/*' \
    --dpkgopt='path-exclude=/usr/share/locale/*' \
    --dpkgopt='path-include=/usr/share/locale/locale.alias' \
    --dpkgopt='path-exclude=/usr/share/doc/*' \
    --dpkgopt='path-include=/usr/share/doc/*/copyright' \
    --dpkgopt='path-include=/usr/share/doc/*/changelog.Debian.*' \
    unstable > debian-unstable.tar

The resulting tarball debian-unstable.tar is 69M in size. You can find a very
similar example in the mmdebstrap man page. You can go even smaller if you omit
the path-include parts of above command (64M) and even smaller by not
installing apt (49M). You can go even more extreme with the custom variant
which lets you pick exactly the package set you want, ignoring Essential:yes
packages.

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Control Debian image size

W. Martin Borgert
On 2019-03-03 08:38, Johannes Schauer wrote:
> Quoting JH (2019-03-03 00:17:06)
> > I have been searching build system and distro for my small Linux
> > embedded device for many weeks, the device has small resources, about
> > 128 MB Flash and 64 MB RAM on imx6, it is a typical embedded system,
> > no GUI, no Desktop environment, no multiple user interfaces, the C++
>
> it is possible to produce a Debian rootfs well below 128 MB in size if you
> choose not to install locales, man pages and docs using the --path-exclude dpkg
> option:

128 MB flash and 64 MB of RAM doesn't sound too small for Debian
to me.

In my company we are using armel and armhf systems similar to
yours. They have both more flash and RAM, but we throw a
Python 3 stack and Nginx on the poor things. We use
path-exclude, too, to avoid docs, manpages, and translations.
Also we run debootstrap with "--exclude=(some packages)".

If the system still does not fit, look whether there is any
dependency you do not need. Then
 1. file a bug against the package, that Debian should turn
    the dependency into a recommends
 2. until the bug is fixes, replace the dependency with a
    dummy package with "equivs".

If there is a small number of huge packages you can't avoid,
try to rebuild them with less features and/or less
dependencies.

Reply | Threaded
Open this post in threaded view
|

Re: Control Debian image size

Andrey Skvortsov
In reply to this post by IAmANobody
Hi,

the first question is do you need/want your system to be Debian?
Debian gives you stable base, reuse of broadly tested and deployed
binary packages, good support, security updates.


Johannes has very good described steps to shrink size of Debian system.


Here is information about shrinking size provided from ELBE:
Information source: https://elbe-rfs.org/images/20150322-clt-elbe.pdf

For example, default image size (from 2015)
231 MB
common finetuning rules
-44 MB (rm /var/lib/apt/lists/*debian*)
-40 MB (rm /var/cache/apt/*.bin)
-40 MB (rm /usr/share/locale/*)
-13 MB (rm /usr/share/doc)
-08 MB (rm /usr/share/man*)
new rfs size
86 MB

It brings you easy below 128M requirements.

You may look at ELBE (https://elbe-rfs.org). It doesn't only build
images (in format you need: ubifs, tar, MSDOS parttable),
but allows you to check for (security) updates, generate license
information, source CD for license compliance.


But you certainly can go smaller with buildroot or Open Embedded.

On 19-03-03 10:17, JH wrote:

> Hi,
>
> I have been searching build system and distro for my small Linux
> embedded device for many weeks, the device has small resources, about
> 128 MB Flash and 64 MB RAM on imx6, it is a typical embedded system,
> no GUI, no Desktop environment, no multiple user interfaces, the C++
> applications are simply for data transfer and communication using
> TCP/IP, the applications were previously running well on Debian 8
> armhf, so Debian is my first choice, but the Debian image size could
> be a major problem, it could be too large to blow out device resource
> constraints, many people commended me to use OE or Yocto or OpenWrt
> distro which image size seems much smaller than the Debian armhf. Is
> it possible to control / reduce Debian armhf image size by Debian
> build, or isar or some other build bools, to reach the same resource
> requirement level in OE Yocto or OpenWrt?
>
> Thank you.
>
> Kind regards,
>
> - jh
>
--
Best regards,
Andrey Skvortsov

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Control Debian image size

IAmANobody
In reply to this post by Johannes Schauer-3
Hi Johannes,

On 3/3/19, Johannes Schauer <[hidden email]> wrote:

> it is possible to produce a Debian rootfs well below 128 MB in size if you
> choose not to install locales, man pages and docs using the --path-exclude
> dpkg
> option:
>
> https://raphaelhertzog.com/2010/11/15/save-disk-space-by-excluding-useless-files-with-dpkg/
>
> An easy way to create a chroot tarball with this option already set is by
> using
> mmdebstrap (shameless plug: I'm the author) like so:
>
> mmdebstrap --variant=apt \
>     --dpkgopt='path-exclude=/usr/share/man/*' \
>     --dpkgopt='path-include=/usr/share/man/man[1-9]/*' \
>     --dpkgopt='path-exclude=/usr/share/locale/*' \
>     --dpkgopt='path-include=/usr/share/locale/locale.alias' \
>     --dpkgopt='path-exclude=/usr/share/doc/*' \
>     --dpkgopt='path-include=/usr/share/doc/*/copyright' \
>     --dpkgopt='path-include=/usr/share/doc/*/changelog.Debian.*' \
>     unstable > debian-unstable.tar
>
> The resulting tarball debian-unstable.tar is 69M in size. You can find a
> very
> similar example in the mmdebstrap man page. You can go even smaller if you
> omit
> the path-include parts of above command (64M) and even smaller by not
> installing apt (49M). You can go even more extreme with the custom variant
> which lets you pick exactly the package set you want, ignoring
> Essential:yes
> packages.
>

That is encouraging, I'll remove the /var/cache as well.

But the chroot rootfs in tar ball only works to install a device where
the kernel is already available, correct?

What can I do to build Debian image with kernel for imx6? Using Isar
or something?

Thank you very much.

- jh

Reply | Threaded
Open this post in threaded view
|

Re: Control Debian image size

IAmANobody
In reply to this post by Andrey Skvortsov
Thanks Andrey and Martin,

Our C++ applications were previously running on debian 8 armhf, it is
probably more secure if we can run it on debian 9 armhf, if the size
does not fit, I'll have to look at OE and Yocto.

The ELBE is the tool I am looking for building the debian image, I
think Isar is another tool similar to ELBE. Going to do some home
work.

Thank you all.

Kind regards.

- jh

Reply | Threaded
Open this post in threaded view
|

Re: Control Debian image size

Johannes Schauer-3
In reply to this post by IAmANobody
Quoting JH (2019-03-03 11:06:39)

> > An easy way to create a chroot tarball with this option already set is by
> > using mmdebstrap (shameless plug: I'm the author) like so:
> >
> > mmdebstrap --variant=apt \
> >     --dpkgopt='path-exclude=/usr/share/man/*' \
> >     --dpkgopt='path-include=/usr/share/man/man[1-9]/*' \
> >     --dpkgopt='path-exclude=/usr/share/locale/*' \
> >     --dpkgopt='path-include=/usr/share/locale/locale.alias' \
> >     --dpkgopt='path-exclude=/usr/share/doc/*' \
> >     --dpkgopt='path-include=/usr/share/doc/*/copyright' \
> >     --dpkgopt='path-include=/usr/share/doc/*/changelog.Debian.*' \
> >     unstable > debian-unstable.tar
> >
> > The resulting tarball debian-unstable.tar is 69M in size. You can find a
> > very similar example in the mmdebstrap man page. You can go even smaller if
> > you omit the path-include parts of above command (64M) and even smaller by
> > not installing apt (49M). You can go even more extreme with the custom
> > variant which lets you pick exactly the package set you want, ignoring
> > Essential:yes packages.
> That is encouraging, I'll remove the /var/cache as well.
>
> But the chroot rootfs in tar ball only works to install a device where
> the kernel is already available, correct?
Yes, this does not include a kernel but only Essential:yes packages, apt and
their dependencies.

> What can I do to build Debian image with kernel for imx6? Using Isar or
> something?

I'm not familiar with imx6 and which kernel image you need for that but I can
tell you how you can create a bootable qemu image for amd64 and maybe by
replacing linux-image-amd64 by the correct kernel this might work for you.
Though maybe you boot with uboot or the like and thus more work might be
needed.

    $ mmdebstrap --variant=apt --include=linux-image-amd64 unstable debian-unstable.tar
    $ cat << END > extlinux.conf
    > default linux
    > timeout 0
    >
    > label linux
    > kernel /vmlinuz
    > append initrd=/initrd.img root=/dev/sda1 rw console=ttyS0
    END
    $ guestfish -N debian-unstable.img=disk:2G -- \
        part-disk /dev/sda mbr : \
        part-set-bootable /dev/sda 1 true : \
        mkfs ext2 /dev/sda1 : mount /dev/sda1 / : \
        tar-in debian-unstable.tar / : \
        extlinux / : \
        copy-in extlinux.conf /
    $ qemu-img convert -O qcow2 debian-unstable.img debian-unstable.qcow2

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Debian 10 bloats like a balloon (was: Control Debian image size)

W. Martin Borgert
In reply to this post by W. Martin Borgert
On 2019-03-03 10:48, W. Martin Borgert wrote:
> If there is a small number of huge packages you can't avoid,
> try to rebuild them with less features and/or less
> dependencies.

I'm currently at moving an older embedded system from Debian 6
(squeeze) to Debian 10 (buster). Which eats a lot of the flash
and all of the RAM. Also, boot time doubled. Just some examples
how Debian changed in the last years:

Binary\Debian      6 (squeeze)  10 (buster)
init¹                      0.1          1.2
ModemManager               0.4          1.2
NetworkManager             0.6          2.3
Python²                    2.3          4.2
/usr/lib/ + /lib/         92.0        247.0

¹ upstart -> systemd
² Python 2 -> Python 3

Those are the executables, not packages sizes, which might be
even larger.

Also, new dependencies in the chain led to 300 instead of 200
depending packages for the same application.

One way out of this is compiling packages with different compile
options and less dependencies. E.g. libxml2 has a new
dependency, libicu63. Leaving it out frees 30 MiB! libicudata.so
is the single largest library on that system and not needed.

I'm happy about reading more ideas here, how to unbloat buster.

TIA & Cheers

Reply | Threaded
Open this post in threaded view
|

Re: Debian 10 bloats like a balloon (was: Control Debian image size)

Jonas Smedegaard-2
Quoting W. Martin Borgert (2019-05-22 10:52:31)

> On 2019-03-03 10:48, W. Martin Borgert wrote:
> > If there is a small number of huge packages you can't avoid,
> > try to rebuild them with less features and/or less
> > dependencies.
>
> I'm currently at moving an older embedded system from Debian 6
> (squeeze) to Debian 10 (buster). Which eats a lot of the flash
> and all of the RAM. Also, boot time doubled. Just some examples
> how Debian changed in the last years:
>
> Binary\Debian      6 (squeeze)  10 (buster)
> init¹                      0.1          1.2
> ModemManager               0.4          1.2
> NetworkManager             0.6          2.3
> Python²                    2.3          4.2
> /usr/lib/ + /lib/         92.0        247.0
>
> ¹ upstart -> systemd
> ² Python 2 -> Python 3
>
> Those are the executables, not packages sizes, which might be
> even larger.
>
> Also, new dependencies in the chain led to 300 instead of 200
> depending packages for the same application.
>
> One way out of this is compiling packages with different compile
> options and less dependencies. E.g. libxml2 has a new
> dependency, libicu63. Leaving it out frees 30 MiB! libicudata.so
> is the single largest library on that system and not needed.
>
> I'm happy about reading more ideas here, how to unbloat buster.
You can save some by avoiding network-manager.

Check if you use _both_ python2 and python3, and consider avoiding the
parts pulling in one of them.

I'd be happy to try analyze more closely, and perhaps codify as a boxer
profile, if your system setup is not a secret.


 - Jonas

--
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debian 10 bloats like a balloon (was: Control Debian image size)

W. Martin Borgert
On 2019-05-22 11:39, Jonas Smedegaard wrote:
> You can save some by avoiding network-manager.

Probably a good idea, but not so good for that specific system.
It uses a lot of NM (and MM) functionality. Too bad, that they
more than tripled in size.

I'll check, whether I can build NM and MM with some options to
decrease size and dependency tree.

Btw, MM is not as convenient to compile as before. In squeeze,
MM was cross-buildable easily. Now it makes use of GIR stuff,
which breaks cross-compilation. (One can remove it from the
Debian build, so MM is still cross-buildable, but not as easy as
before.) Note to myself: Avoid everything that uses GIR.

> Check if you use _both_ python2 and python3, and consider avoiding the
> parts pulling in one of them.

No, we moved from Python 2 to Python 3, see my table. The
Python 3 binary alone is *much* larger than the Python 2 binary.

> I'd be happy to try analyze more closely, and perhaps codify as a boxer
> profile, if your system setup is not a secret.

Unfortunately, some parts of it are completely non-free :-(
But I'm allowed to talk about it, there's no gag order.

Reply | Threaded
Open this post in threaded view
|

Re: Control Debian image size

Baurzhan Ismagulov
In reply to this post by IAmANobody
Hello JH,

On Sun, Mar 03, 2019 at 09:06:39PM +1100, JH wrote:
> What can I do to build Debian image with kernel for imx6? Using Isar
> or something?

Kernel modules also take considerable space. I've bitbaked
multiconfig:qemuarm-stretch:isar-image-base with Isar and removed the stock
linux-image-4.9.0-9-armmp. The disk usage dropped from 277 MiB to 125 MiB. So,
you want to build your own kernel enabling only the drivers you need.

Using Isar, you may build your own kernel and implement the others' suggestions
(I think Isar already covers many of them). If you have questions, please cc
me.

In the past, we used Debian on a 32-MiB system with JFFS compression and
busybox. We had to use perl-minimal instead of perl (required for postinst
scripts) to reduce the dependencies.

With kind regards,
Baurzhan.