Goal: a specialized inventory of installed packages

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Goal: a specialized inventory of installed packages

Richard Owlett-3
I moved from WindowsXP when Squeeze was the current release.
In the first year I did *many* installs from scratch to determine what I
wanted in a final system (made much use of preseeding).

I currently have a configuration of Stretch that meets most of my needs.
As the installation was performed using the default Debian installer,
there is a plethora of packages of no interest installed. As some
important packages are not installed by default, apt and Synaptic were
used to install them.

I'm setting out to do an _extremely_ custom *minimal* install of Buster.
The desired inventory shall list *ONLY* top level packages.
[ E.G. if gfortran was purposely installed, the ~dozen packages
installed because they were tagged as depends, recommends, or suggests
wold *NOT* be listed. ]

Is there a suitable tool? [Synaptic's History menu item is closest I've
found].

TIA










Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

The Wanderer
On 2020-04-05 at 11:30, Richard Owlett wrote:

> I moved from WindowsXP when Squeeze was the current release. In the
> first year I did *many* installs from scratch to determine what I
> wanted in a final system (made much use of preseeding).
>
> I currently have a configuration of Stretch that meets most of my
> needs. As the installation was performed using the default Debian
> installer, there is a plethora of packages of no interest installed.
> As some important packages are not installed by default, apt and
> Synaptic were used to install them.
>
> I'm setting out to do an _extremely_ custom *minimal* install of
> Buster. The desired inventory shall list *ONLY* top level packages. [
> E.G. if gfortran was purposely installed, the ~dozen packages
> installed because they were tagged as depends, recommends, or
> suggests wold *NOT* be listed. ]
This sounds like a job for apt's "manual" vs. "auto" status flag.

'apt-mark showmanual' will list all the packages which are flagged as
being manually installed.

Specifying a package explicitly to 'apt-get install' or similar flags it
as manually installed, if you actually go through and complete the
installation (cancelling the install also cancels the flag). Any
packages which it pulls in as dependencies - whether "Depends:"
"Recommends", or "Suggests:" - get flagged as "auto" instead.

From what I've seen, it looks as if debian-installer also flags some
packages as manually installed, during initial install of the Debian
system. I don't know which ones do and don't get that treatment.

I don't know if that's good enough for you, but it's very likely the
best you're going to get.

--
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

Richard Owlett-3
On 04/05/2020 10:52 AM, The Wanderer wrote:

> On 2020-04-05 at 11:30, Richard Owlett wrote:
>
>> I moved from WindowsXP when Squeeze was the current release. In the
>> first year I did *many* installs from scratch to determine what I
>> wanted in a final system (made much use of preseeding).
>>
>> I currently have a configuration of Stretch that meets most of my
>> needs. As the installation was performed using the default Debian
>> installer, there is a plethora of packages of no interest installed.
>> As some important packages are not installed by default, apt and
>> Synaptic were used to install them.
>>
>> I'm setting out to do an _extremely_ custom *minimal* install of
>> Buster. The desired inventory shall list *ONLY* top level packages. [
>> E.G. if gfortran was purposely installed, the ~dozen packages
>> installed because they were tagged as depends, recommends, or
>> suggests wold *NOT* be listed. ]
>
> This sounds like a job for apt's "manual" vs. "auto" status flag.
>
> 'apt-mark showmanual' will list all the packages which are flagged as
> being manually installed.

*YES* THANK YOU!

>
> Specifying a package explicitly to 'apt-get install' or similar flags it
> as manually installed, if you actually go through and complete the
> installation (cancelling the install also cancels the flag). Any
> packages which it pulls in as dependencies - whether "Depends:"
> "Recommends", or "Suggests:" - get flagged as "auto" instead.
>
>  From what I've seen, it looks as if debian-installer also flags some
> packages as manually installed, during initial install of the Debian
> system. I don't know which ones do and don't get that treatment.

I did a test run. I think I see the pattern of which packages it marks
as manual.
E.G. It shows systemd related items as "manual". But for *MY* purposes I
would class them as "auto". That will not be a problem in practice. I'll
just do what the installer thinks is an absolute minimal install and run
'apt-mark showmanual' against it. I'll save that list. Then create a
script to ignore those lines when 'apt-mark showmanual' is run against
my actual system.

>
> I don't know if that's good enough for you, but it's very likely the
> best you're going to get.
>

It looks like a good match so far. Thanks again.




Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

David Wright-3
In reply to this post by Richard Owlett-3
On Sun 05 Apr 2020 at 10:30:41 (-0500), Richard Owlett wrote:

> I currently have a configuration of Stretch that meets most of my needs.

> I'm setting out to do an _extremely_ custom *minimal* install of Buster.
> The desired inventory shall list *ONLY* top level packages.
> [ E.G. if gfortran was purposely installed, the ~dozen packages
> installed because they were tagged as depends, recommends, or suggests
> wold *NOT* be listed. ]
>
> Is there a suitable tool? [Synaptic's History menu item is closest
> I've found].

I assume that what you're after is a command like
  apt-get install a b c d …
where the list is the shortest that would install the said system.

In which case, I would presume the answer is no.

But it's not too onerous to write a script to parse the status file
(dpkg-query does the grunt work) and determine which packages are
not a [pre-]depends/recommends/suggests of any other. You may or
may not want to prune the resulting list, removing the names of any
packages that the d-i has no option but to install itself.

How kindly apt will take to be presented with such a list of packages
for installation at one sitting, I don't know.

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

Richard Owlett-3
On 04/05/2020 12:00 PM, David Wright wrote:

> On Sun 05 Apr 2020 at 10:30:41 (-0500), Richard Owlett wrote:
>
>> I currently have a configuration of Stretch that meets most of my needs.
>
>> I'm setting out to do an _extremely_ custom *minimal* install of Buster.
>> The desired inventory shall list *ONLY* top level packages.
>> [ E.G. if gfortran was purposely installed, the ~dozen packages
>> installed because they were tagged as depends, recommends, or suggests
>> wold *NOT* be listed. ]
>>
>> Is there a suitable tool? [Synaptic's History menu item is closest
>> I've found].
>
> I assume that what you're after is a command like
>    apt-get install a b c d …
> where the list is the shortest that would install the said system.
>
> In which case, I would presume the answer is no.
>
> But it's not too onerous to write a script to parse the status file
> (dpkg-query does the grunt work) and determine which packages are
> not a [pre-]depends/recommends/suggests of any other. You may or
> may not want to prune the resulting list, removing the names of any
> packages that the d-i has no option but to install itself.
>
> How kindly apt will take to be presented with such a list of packages
> for installation at one sitting, I don't know.
>
> Cheers,
> David.

I wasn't aiming for tool to automajically recreate an existing system.
I was going more for a tool to allow me to visualize specifying what my
new system would/should look like.
'apt-mark ...' seems to be aimed at my goals.
Thanks.




Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

Marco Möller
In reply to this post by Richard Owlett-3
Once you have your list of packages in a text file, for each package one
line, you could apply the list like this:

     apt install $(< mylist.txt)

Consider to first do a simulation run for finding problems in the list:
     -s

Consider to use the following flag in order to not draw in a maybe
gigantic tail of not really wanted stuff:
     --no-install-recommends

If you from time to time would like to include some package from the
maybe gigantic tail of recommends, then simply add it individually to
the list!

I maintain manually the list, which you are about to generate now, and
have very good experience starting with the following complete command:
     apt install -s --no-install-recommends $(< mylist.txt)

When installing the basic Debian system from the official media,
you may want to do this in mode "expert install", and you may want to
know that all offered graphical Desktop Environments are full fledged
installs coming with many (IMHO too much!) applications, while
afterwards going for a manual install of the desktop environment outside
of the Debian installer will allow you to go for a fine grained
"minimal" Desktop Environment installation only coming with the really
needed functionality and without the applications.
I therefore recommend to have your personally selected packages for the
wished Desktop Environment (KDE, Gnome, XFCE, etc.) included in your
personal list, and when arriving at the Debian installer's "tasksel"
menu do not only deactivate all the listed Desktop Environments, which
become offered, but even do deactivate the head entry “Debian desktop
environment” itself, otherwise this would install a full featured GNOME
desktop although below in the list you have had deactivated GNOME. I
usually only select here “standard system utilities” and “print server”,
and then have all the needed packages for "my" Desktop Environment in my
list.
If following this path, then be aware that for a laptop essential or
nice to have tools like wpasupplicant or network-manager (and many
more!) will not become automatically installed for you! Setting up your
personal list also concerning the Desktop Environment manually will thus
require some extended effort. Unfortunately, the Debian installer is
only offering to install full fledged Desktop Environments, but not
minimal versions of it.

Good luck! Marco.

Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

Richard Owlett-3
On 04/05/2020 01:29 PM, Marco Möller wrote:
> Once you have your list of packages in a text file, for each package one
> line, you could apply the list like this:
>
>      apt install $(< mylist.txt)

Good to know that is known to work.

>
> Consider to first do a simulation run for finding problems in the list:
>      -s
>
> Consider to use the following flag in order to not draw in a maybe
> gigantic tail of not really wanted stuff:
>      --no-install-recommends

That had been my intention for that reason.
A few years ago someone suggested additional methods to install a
minimum number of packages. I'll have to search my old emails - possibly
on a semi-retired machine.

>
> If you from time to time would like to include some package from the
> maybe gigantic tail of recommends, then simply add it individually to
> the list!
>
> I maintain manually the list, which you are about to generate now, and
> have very good experience starting with the following complete command:
>      apt install -s --no-install-recommends $(< mylist.txt)
>
> When installing the basic Debian system from the official media,
> you may want to do this in mode "expert install",

That has been my procedure.

> and you may want to
> know that all offered graphical Desktop Environments are full fledged
> installs coming with many (IMHO too much!) applications,

You're preaching to the choir ;/

> while
> afterwards going for a manual install of the desktop environment outside
> of the Debian installer will allow you to go for a fine grained
> "minimal" Desktop Environment installation only coming with the really
> needed functionality and without the applications.
> I therefore recommend to have your personally selected packages for the
> wished Desktop Environment (KDE, Gnome, XFCE, etc.) included in your
> personal list, and when arriving at the Debian installer's "tasksel"
> menu do not only deactivate all the listed Desktop Environments, which
> become offered, but even do deactivate the head entry “Debian desktop
> environment” itself, otherwise this would install a full featured GNOME
> desktop although below in the list you have had deactivated GNOME. I
> usually only select here “standard system utilities” and “print server”,
> and then have all the needed packages for "my" Desktop Environment in my
> list.

I've a collection of experimental preseed.cfg files aimed at that.

> If following this path, then be aware that for a laptop essential or
> nice to have tools like wpasupplicant or network-manager (and many
> more!) will not become automatically installed for you! Setting up your
> personal list also concerning the Desktop Environment manually will thus
> require some extended effort. Unfortunately, the Debian installer is
> only offering to install full fledged Desktop Environments, but not
> minimal versions of it.

Since I saw a site that would create a custom preseed.cfg in response to
a series of questions resembling those the standard installer asks. That
site is obsolete as it predates Squeeze IIRC. In any case I would do it
in TCL/Tk as a educational coding exercise.

>
> Good luck! Marco.
>
>
>

Thank you.



Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

Andrei POPESCU-2
In reply to this post by The Wanderer
On Du, 05 apr 20, 11:52:32, The Wanderer wrote:
>
> From what I've seen, it looks as if debian-installer also flags some
> packages as manually installed, during initial install of the Debian
> system. I don't know which ones do and don't get that treatment.

At least the packages installed during the 'debootstrap' phase. Possibly
also those installed by tasks (if one is selected) as well as linux
image(s) and bootloader (grub).

Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

Andrei POPESCU-2
In reply to this post by Richard Owlett-3
On Du, 05 apr 20, 11:59:17, Richard Owlett wrote:
>
> I did a test run. I think I see the pattern of which packages it marks as
> manual.
> E.G. It shows systemd related items as "manual". But for *MY* purposes I
> would class them as "auto". That will not be a problem in practice. I'll
> just do what the installer thinks is an absolute minimal install and run
> 'apt-mark showmanual' against it. I'll save that list. Then create a script
> to ignore those lines when 'apt-mark showmanual' is run against my actual
> system.

Something I used to do for similar purposes[1] was to mark everything as
auto installed and then mark as manual the packages I was actually using
(typically things like mutt, GUI/TUI browser, etc.).

This was done using aptitude's interactive mode (not sure if apt-mark
even existed at the time), as I could easily see what aptitude wanted to
keep or remove.

If you intend to also *remove* packages based on this approach do mind
the settings for "APT::AutoRemove::RecommendsImportant" and
"APT::AutoRemove::SuggestsImportant".

[1] I stopped doing this because I found the optimization was not worth
the time spent.

Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Goal: a specialized inventory of installed packages

David Wright-3
In reply to this post by Richard Owlett-3
On Sun 05 Apr 2020 at 13:17:37 (-0500), Richard Owlett wrote:

> On 04/05/2020 12:00 PM, David Wright wrote:
> > On Sun 05 Apr 2020 at 10:30:41 (-0500), Richard Owlett wrote:
> >
> > > I currently have a configuration of Stretch that meets most of my needs.
> >
> > > I'm setting out to do an _extremely_ custom *minimal* install of Buster.
> > > The desired inventory shall list *ONLY* top level packages.
> > > [ E.G. if gfortran was purposely installed, the ~dozen packages
> > > installed because they were tagged as depends, recommends, or suggests
> > > wold *NOT* be listed. ]
> > >
> > > Is there a suitable tool? [Synaptic's History menu item is closest
> > > I've found].
> >
> > I assume that what you're after is a command like
> >    apt-get install a b c d …
> > where the list is the shortest that would install the said system.
> >
> > In which case, I would presume the answer is no.
> >
> > But it's not too onerous to write a script to parse the status file
> > (dpkg-query does the grunt work) and determine which packages are
> > not a [pre-]depends/recommends/suggests of any other. You may or
> > may not want to prune the resulting list, removing the names of any
> > packages that the d-i has no option but to install itself.
> >
> > How kindly apt will take to be presented with such a list of packages
> > for installation at one sitting, I don't know.
>
> I wasn't aiming for tool to automajically recreate an existing system.
> I was going more for a tool to allow me to visualize specifying what
> my new system would/should look like.
> 'apt-mark ...' seems to be aimed at my goals.
> Thanks.

Well, it does appear from your response, "I'll just do what the
installer thinks is an absolute minimal install and run 'apt-mark
showmanual' against it", that you're still enjoying doing installs,
so you can probably iterate towards a solution by by trying your
minimal list and seeing what results (comparing it with your
"reference" installation).

And it does appear from your other response that minimalism is more
important than function, if you exclude Recommends.

But in any case, I was assuming you wanted a reverse tool, to work out
which packages on your system are top-level, in the sense that they
are not dependent on any other package that you've installed.

I've just cannibalised a function that I use for kind of similar
purposes, because with being kept at home it seemed a good time to try
out bash's associative arrays. (I've always dropped into Python in the
past.) You might want to try it out: its default is to include
Recommends, but you can generate a longer list by using strict Depends.

It doesn't handle alternatives ( | is treated as , ) and it ignores
Pre-Depends. It also works ab initio, as if you had to drive the
d-i selection yourself. But one could modify it to merge in a
dpkg-query listing from a system that's untouched post-install.

It uses trash files (as I often do) rather than pipes so that you can
inspect partial results. As I say, it was just to get my hands dirty
with bash associative arrays. Strictly, they're not functionally
required here, but they're meant to be fast to search than a list
of an indexed array. Anyway:

function my-packages-toplevels {
    [ "$1" = "-?" ] && printf '%s\n' "Usage: $FUNCNAME [Depends-only]
        lists the packages installed on this system that are not dependent (and,
        if lacking any argument, not recommended) by any other package." >&2 && return 1
    local -A Depsrecs
    local Pkg Format=' ${Depends} ${Recommends} \n'
    [ -n "$1" ] && Format=' ${Depends} \n'
    local Unique1="$(mktemp ${Uniquetrash:-/tmp}/$FUNCNAME-"$(date +%s)"-1some-XXXX)"
    local Unique2="$(mktemp ${Uniquetrash:-/tmp}/$FUNCNAME-"$(date +%s)"-2all-XXXX)"
    dpkg-query -W --showformat "$Format" | sed -e 's/([^)]\+)//g;s/,/ /g;s/|/ /g;s/[ ]\+/ /g;s/ /\n/g;' | sort -u | tail -n +2 >| "$Unique1"
    while IFS=$'\n' read -r Pkg; do
        Depsrecs["$Pkg"]="yes"
    done < "$Unique1"
    dpkg-query -W --showformat '${Package}\n' | sort >| "$Unique2"
    while IFS=$'\n' read Pkg; do
        [[ ! -n "${Depsrecs[$Pkg]}" ]] && printf '%s\n' "$Pkg"
    done < "$Unique2"
    return 0
}

Cheers,
David.