parallel buildd instances in chroots on same host?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

parallel buildd instances in chroots on same host?

Ryan Tandy-4
I'm investigating why openldap failed its tests on powerpc:

https://buildd.debian.org/status/fetch.php?pkg=openldap&arch=powerpc&ver=2.4.48%2Bdfsg-1&stamp=1564255370&raw=0

The test001-slapadd failed because ldapsearch returned different data
than expected. The test script is simple, it just imports data into an
empty database (offline) and then starts slapd and tries to read back
the same data using ldapsearch (online).

It looks like the powerpc and ppc64 builds were running in parallel, on
kapitsa and kapitsa2 respectively. If these builds were running in
different chroots on the same host, then the test failure can be
explained by ldapsearch returning data from a slapd instance started by
the other build.

Am I correct about the two chroots on the single host, and is this a
common setup for Debian buildds? I had assumed builds were typically run
in isolated VMs.

The test suite supports customizing the port numbers (by an environment
variable), so I could randomize that in debian/rules. However the test
suite doesn't have a way to check whether the chosen port is already in
use, or whether slapd actually started successfully, so the risk of
collision can't be completely eliminated.

For now, could you please give back openldap/2.4.48+dfsg-1 on powerpc?

Thank you,
Ryan

Reply | Threaded
Open this post in threaded view
|

Re: parallel buildd instances in chroots on same host?

Aurelien Jarno
On 2019-08-11 10:58, Ryan Tandy wrote:

> I'm investigating why openldap failed its tests on powerpc:
>
> https://buildd.debian.org/status/fetch.php?pkg=openldap&arch=powerpc&ver=2.4.48%2Bdfsg-1&stamp=1564255370&raw=0
>
> The test001-slapadd failed because ldapsearch returned different data than
> expected. The test script is simple, it just imports data into an empty
> database (offline) and then starts slapd and tries to read back the same
> data using ldapsearch (online).
>
> It looks like the powerpc and ppc64 builds were running in parallel, on
> kapitsa and kapitsa2 respectively. If these builds were running in different
> chroots on the same host, then the test failure can be explained by
> ldapsearch returning data from a slapd instance started by the other build.
>
> Am I correct about the two chroots on the single host, and is this a common
> setup for Debian buildds? I had assumed builds were typically run in
> isolated VMs.

This is a correct assumption for official architectures. powerpc and
ppc64 are debian-ports architectures, so the setup might be different.
Adding [hidden email] and [hidden email] in Cc so
that the buildd maintainers can answer more about the setup.

--
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
[hidden email]                 http://www.aurel32.net

Reply | Threaded
Open this post in threaded view
|

Re: parallel buildd instances in chroots on same host?

John Paul Adrian Glaubitz
In reply to this post by Ryan Tandy-4
Hi Ryan!

On 8/11/19 7:58 PM, Ryan Tandy wrote:
> It looks like the powerpc and ppc64 builds were running in parallel, on kapitsa and kapitsa2 respectively. If these builds were running in different chroots on the same host, then the test failure can be explained by ldapsearch returning data from a slapd instance started by the other build.
>
> Am I correct about the two chroots on the single host, and is this a common setup for Debian buildds?

Yes, this machine is running two buildd instances on the same machine. The machine rather fast so running
just one buildd instance would be a waste of resources. We are doing the same for sparc64 as well. For
example, landau has so many CPUs and RAM that it can easily host 4 buildd instances in parallel while
still being fast. The machine has 128 vCPUs and 96 GB of RAM.

> I had assumed builds were typically run in isolated VMs.

Yes, this certainly applies for the release architectures but not necessarily for Debian Ports architectures
where the buildd instances are not maintained by the DSA team.

Isolated VMs have the disadvantage that CPU and memory are not dynamically shifted across build jobs which
can make build jobs less efficient when you have one VM building a small package while the other one is
building something like Rust or gcc.

> The test suite supports customizing the port numbers (by an environment variable), so I could randomize that in debian/rules. However the test suite doesn't have a way to check whether the chosen port is already in use, or whether slapd actually started successfully, so the risk of collision can't be completely eliminated.

Understood. We might change the setup in the future. We just recently received a new POWER server by
IBM for the powerpc and ppc64 ports but I haven't had the time yet to set it up as I'm busy with
SUSE dayjob work.

> For now, could you please give back openldap/2.4.48+dfsg-1 on powerpc?

Done and it built fine.

Adrian

--
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - [hidden email]
`. `'   Freie Universitaet Berlin - [hidden email]
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-openldap-devel] parallel buildd instances in chroots on same host?

Alister Winfield-2
Wouldn’t using something like containers (docker etc)  be the best fix here. Would ensure no unexpected interactions between the builds and yet have near zero overheads.

Just a thought.

Alister Winfield.

> On 14 Aug 2019, at 11:41, John Paul Adrian Glaubitz <[hidden email]> wrote:
>
> Hi Ryan!
>
> On 8/11/19 7:58 PM, Ryan Tandy wrote:
>> It looks like the powerpc and ppc64 builds were running in parallel, on kapitsa and kapitsa2 respectively. If these builds were running in different chroots on the same host, then the test failure can be explained by ldapsearch returning data from a slapd instance started by the other build.
>>
>> Am I correct about the two chroots on the single host, and is this a common setup for Debian buildds?
>
> Yes, this machine is running two buildd instances on the same machine. The machine rather fast so running
> just one buildd instance would be a waste of resources. We are doing the same for sparc64 as well. For
> example, landau has so many CPUs and RAM that it can easily host 4 buildd instances in parallel while
> still being fast. The machine has 128 vCPUs and 96 GB of RAM.
>
>> I had assumed builds were typically run in isolated VMs.
>
> Yes, this certainly applies for the release architectures but not necessarily for Debian Ports architectures
> where the buildd instances are not maintained by the DSA team.
>
> Isolated VMs have the disadvantage that CPU and memory are not dynamically shifted across build jobs which
> can make build jobs less efficient when you have one VM building a small package while the other one is
> building something like Rust or gcc.
>
>> The test suite supports customizing the port numbers (by an environment variable), so I could randomize that in debian/rules. However the test suite doesn't have a way to check whether the chosen port is already in use, or whether slapd actually started successfully, so the risk of collision can't be completely eliminated.
>
> Understood. We might change the setup in the future. We just recently received a new POWER server by
> IBM for the powerpc and ppc64 ports but I haven't had the time yet to set it up as I'm busy with
> SUSE dayjob work.
>
>> For now, could you please give back openldap/2.4.48+dfsg-1 on powerpc?
>
> Done and it built fine.
>
> Adrian
>
> --
> .''`.  John Paul Adrian Glaubitz
> : :' :  Debian Developer - [hidden email]
> `. `'   Freie Universitaet Berlin - [hidden email]
>  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
>
> _______________________________________________
> Pkg-openldap-devel mailing list
> [hidden email]
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-openldap-devel

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-openldap-devel] parallel buildd instances in chroots on same host?

Philipp Kern-2
On 8/14/2019 11:04 PM, Alister Winfield wrote:
> Wouldn’t using something like containers (docker etc)  be the best fix here. Would ensure no unexpected interactions between the builds and yet have near zero overheads.

Only if you build something that works properly on all architectures
Debian supports.

Kind regards
Philipp Kern