[Idea] Debian User Repository? (Not simply mimicing AUR)

classic Classic list List threaded Threaded
103 messages Options
1 ... 3456
Reply | Threaded
Open this post in threaded view
|

Re: [Idea] Debian User Repository? (Not simply mimicing AUR)

Enrico Weigelt, metux IT consult-3
On 10.04.19 03:53, Russ Allbery wrote:

Hi,

> Possibly my least favorite> thing about RPMs is the spec files, because by smashing everything>
together into the same file, the syntax of that file is absurd.  This
bit> is a shell script!  This bit is a configuration file!  This bit is>
human-readable text!  This bit is a patch list!  This bit is a file>
manifest!  This bit is a structured changelog!  This bit is a bunch of>
preprocessor definitions!  Aie.
Same for me.

Certainly, deb and debian methods also have their downsides:

#1: text-based patches inside debian/ make everything unncessarily
    complex, as soon as you're working w/ a decent VCS (eg. git).
    their historical purpose is long obsoleted (since over a decade)

#2: for many common usecases, full-blown makefile is too much
    complexity, and even w/ debhelper, knowing which variables have
    to be set in which exact way, isn't entirely trivial.
    some purely declarative rule file (eg. yaml) would make those very
    common usecases much easier.

#3: when you have to generate the control file on the fly, things easily
    get messy - i'm currently fighting here w/ packaging the kernel.
    the problem is that this file contains the source package name and
    source dependencies, which need to be known before the control file
    can be generated. circular dependency.

    I'm currently working around that by having a separate control file
    (debian/control.bootstrap) which is used by my build machinery
    (dck-buildpackage) in a separate preparation step, when control file
    is missing.


But still, IMHO, the debian packaging toolstack is much superior to
anything else i've every encountered.


--mtx

--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[hidden email] -- +49-151-27565287

Reply | Threaded
Open this post in threaded view
|

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

Enrico Weigelt, metux IT consult-3
In reply to this post by Helmut Grohne
On 10.04.19 16:56, Helmut Grohne wrote:

Hi,

> I looked into this. Your reasons are sound and you are scratching your> itch. This is great.
ACK. It's always good when people make their hands dirty and work on
solving actual problems. Even if the actual output (=code, etc) finally
doesn't get wide use or even thrown away completely, we still a lot
that way.

When I look back into my earlier years, I've written lots of things that
never have been actually used, but I've learned a lot that way.

> Your implementation goes straight from .durpkg -> .deb. I question this> decision: We already have lots of tools to go from .dsc to .deb. Your>
implementation replicates part of that and I think, this is bad as it>
makes it harder to collaborate.
I did a similar decision w/ dck-buildpackage, because I came to the
conclusion that intermediate steps via dsc are just unncessary
complexity and slowdown. But the motivation of dck-buildpackage was
getting rid of complicated and cumbersome things like pbuilder.

So, I can understand to his decision - he probably doesn't need anything
from the dsc-based tools, as he's operating in a completely different
scope.

> Let me propose a rather intrusive interface change to duprkit. What if> the result of building a .durpkg was a .dsc rather than a .deb? Then
you> could split duprkit into two tools:> >  * One tool to build source
packages from .durpkg files on demand.>  * One tool to build a specific
binary package from a given deb-src>    repository.
Let me propose an even more consequent approach: let it operate even one
step earlier in the pipeline by just generating a debianized source
tree. You could then use the tool of your choice to create dsc from that
and put in whatever kind of pipeline you prefer. My personal choice here
would be dck-buildpackage, and my infrastructure ontop of that.

By the way, this opens up another common topic: how do we get from an
upstream tree (in git repo) to a debianzed source tree w/ minimal manual
efforts ?

> Now in principle, the latter is simply sbuild or pbuilder, but there is> more to it:>  * Given the binary package name, figure out which source
package must>    be built.

Yet another tricky issue. The primary data source for that usually are
the control files. But they also somethimes are autogenerated.

Could we invent some metadata language for this, that also can handle
tricky cases like the kernel ?

>  * Given that source package's Build-Depends, figure out what other>    binary packages need to be built.>  * Recurse.>  * Build them in a
suitable order.

You're talking about building whole overlay repos ?
Then I might have something for you:

https://github.com/metux/deb-pkg

Note: it's still pretty hackish and needs some local per-project
customizations. Haven't had the time to make some general purpose
standalone package of it.

I'm just using it for building per private extra repos for my customers.

If anybody likes to join in and turn it into some general purpose
package, let's talk about that in a different thread. The first step
would be creating a decent command line interface (for now, the run-*
scripts are just project-specific dirty hacks to save me from typing
too much ;-)).

> (Some people will observe that this is the "bootstrap" problem. ;)

Not really bootstrap problem, but depenency problem. Easier to solve :p

> There is one major difficulty here (and duprkit doesn't presently solve> that either): If you figure that some binary package is missing, you>
have no way of knowing which .durpkg file to build to get the relevant>
source package.

Yes, he'd have to reinvent the dependency handling. This is one of the
points that let me question the whole approach and favour completely
different approaches like classic containers.

> Now let's assume that you do want to allow complex dependencies in this
> user repository. In this case, it would make sense to trade .durpkg
> files for plain "debian" directories with an additional debian/rules
> target to obtain the source. (We removed "get-orig-source" from policy a
> while ago, but maybe this is what you want here.)

Sounds a good idea.

Maybe we should put this to a broader discussion, along w/ the control
file generation problem. My desired outcome of that would be a generic
way for fully automatically building everything from a debianzed source
tree (eg. git repo) within a minimal container/jail, w/o any other extra
configuration outside that source tree - even for those cases where the
control file needs to be generated, which again needs some deps.


--mtx

--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[hidden email] -- +49-151-27565287

Reply | Threaded
Open this post in threaded view
|

Re: PPAs (Re: [Idea] Debian User Repository? (Not simply mimicing AUR))

Enrico Weigelt, metux IT consult-3
In reply to this post by Mo Zhou
On 11.04.19 09:44, Mo Zhou wrote:

> Different from that, duprkit's design don't hope to limit> the user with any pre-defined "sequence", but enable the users to>
selectively call the functions they need. In other words, the> user can
define how to deal with the prepared source+debian directories,>
afterall the .durpkg header is a shell script. That said, I think> some
more helper functions would be nice: [1].
I'm still struggling to understand why simply using old-fashioned
./debian/rules file (possibly w/o using debhelper+friends) isn't
sufficient here.

AFAIK, all that tools like buildd do for actual build is calling that
rule file w/ some specific target names (eg. 'binary'). You can put in
anything you like here - it's just a makefile. Theoretically, you could
also use shellscript instead.

If you drop the idea of having everything in a single file in favour
of debian trees (= something that has the 'debian' subdirectory with
a 'rules' file in it), the existing debian toolchains could be used.

> My blueprint includes a very light-weight/simple dependency tracking> mechanism. And I assume the project don't need to handle complex dep>
trees like apt. Because:> > 1. If a package is common and useful enough,
such that it has been>    adopted by many other projects, why don't I
package it for the>    official Debian release? So, I expect that most
packages that DUPR>    will deal with, are actually leaf or near-leaf
packages on the>    dependency tree.
Okay, that's a different topic. We have three options here:

a) put it into official debian repo. that would go the usual ways, but
   takes pretty long, until the next release is out and the desired
   audience actually uses it.

b) add it to backports repos. i'm not sure how the actual workflows
   and release timelines look here.

c) go the PPA route. here we'd need some repo-level dependency handling
   (not sure what tools exist here), and we'd have to coordinate between
   several PPAs

> 2. Some of my targeted upstreams do sourceless binary tarball release.>    They seldom get into the dependency trouble...

When I have to touch those stuff, I basically always run into trouble.
Many subtle breaks, that are extremly hard to resolve (even to track
down). Those stuff I'm only doing in containers. Binary-only stuff is
not trustworthy at all, so it really should be strictly confined.

Those vendors (eg. Microsoft/Skype) also like to mess w/ package manager
configuration, have implicit dependencies like silly Lennartware, etc.
I never ever run such crap outside a strictly confined container.

One of the worst things I've ever seen is coming from National
Instruments (which don't support Debian anyways, just ancient RHEL)
Traditionally they only provided ridiculous installer programs
(just like they're used to from the dillettantic Windows world)
that do lots of really weird things, even messing w/ the kernel
(yeah, they still insist on binary-only kernel modules, that's
always broken-by-design).

Somewhere in last summer they learned what package repos are for,
well, just partially learned. They now messed w/ the repo configs and
installed a globally trusted package source with explicitly disabled
authentication and plain http. Boom - 0day !

Due to their long history of hostility, total bullshit and censorship in
their own "community", I've posted that @full-disclosure (even goverment
institutions like BSI called by for interviews on that matter - their
products also run in critical infrastructure like power plants). Again
it took several month for the issue to be migitated by NI.

> 3. Inserting a DUPR package into the near-root part of the Debian>    dependency tree is, generally speaking, a terrible idea.>    Only
those who aware of what they are doing will do this.
ACK. Those stuff belongs into throwaway-containers.

> The `bin/duprCollector` will collect meta information from a collection> (and will form a dependency tree in the future). I have no plan to>
rethink about the "get-orig-source" target since there are ... lots> of
weird upstreams in my list...
Maybe we should talk about some of these cases, to get a better idea.

In general, we IMHO should rethink the workflows for creating the actual
buildable debian tree from upstream releases (in many packages that's
still pretty manual and hackish)


--mtx

--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
[hidden email] -- +49-151-27565287

1 ... 3456