Bug#987017: recommends 3 different ways to find obsolete packages, pick one

[Antoine Beaupre]

Package: release-notes
Severity: minor

The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
*different* ways of finding what are essential orphaned packages:

aptitude search '~o'
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
apt-forktracer | sort

Then I also know of those:

apt-show-versions | grep -v /bullseye
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'

I frankly don't quite know where I stand with all this anymore, but I
am getting the strong feeling we're sending an incoherent message
here. :)

In my personal documentation, I've settled on `apt-forktracer`, but I
suspect we might want to stick with `aptitude search '~obsolete'`
because that matches other documentation in the release notes (and
allows for easy purging).

Is there any reason why we have all that diversity?

What's the right way to do what we actually want here?

-- System Information:
Debian Release: 10.9
APT prefers stable-debug
APT policy: (500, 'stable-debug'), (500, 'stable'), (1, 'experimental'), (1, 'unstable'), (1, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.19.0-16-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_CA.UTF-8, LC_CTYPE=fr_CA.UTF-8 (charmap=UTF-8), LANGUAGE=fr_CA.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

[Antoine Beaupré]

I actually forgot that bullseye itself introduces yet another one:

apt list ~obsolete

Apparently, those are also a thing:

comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u)
apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'

[Paul Gevers]

Hi

Post by Antoine Beaupré
apt list ~obsolete

But (I'm running bookworm):

***@mulciber ~/release-team/release-notes $ apt list ~obsolete
Listing... Error!
E: input:2-9: error: Expected end of file
~obsolete
^^^^^^^

Did this work in the past?

Paul

[Stuart Prescott]

Post by Paul Gevers
Hi

     apt list ~obsolete

Listing... Error!
E: input:2-9: error: Expected end of file
   ~obsolete
     ^^^^^^^
Did this work in the past?

Doubt that ever worked as written — that should be « apt list '~o' ».
(The ~ patterns are single letters followed by arguments.)

To tail back through this bug, there's a mixture of different recipes
considered here. They do different things and will produce very
different results depending on whether they are run before or after
editing the sources.list+apt update step.

"obsolete" packages are packages that are currently installed but that
package name is not available in any current apt source. The version of
the package installed can be higher or lower than what is currently
available.
- If you s/bullseye/bookworm/+apt update, packages that are upgradable
are not obsolete; the package names that have disappeared completely
are, obsolete.
- If you installed packages from bullseye-backports and then removed
bullseye-backports from your sources.list, those packages aren't
available any more, but are also not obsolete.

Obsolete packages are matched by both the aptitude and apt searches ~o.
Locally built/manually downloaded packages that are in no apt source
will also match ~o; locally built/manually downloaded packages where the
package name is also in an apt repo will _not_ be included.

There's a second collection of packages where the currently installed
version cannot be located in any available apt repository. While
'obsolete' packages was only about the package name, this is about the
(name, version, architecture) tuple. This set of "not available"
packages includes the yet-to-be upgraded packages, and the ones that are
somehow at a higher version than what is currently available.


So... the actual commands discussed:


aptitude search '~o'
apt list '~o'
Both of these look for obsolete packages as defined above. On this
machine it lists gcc-8-base linux-image-5.16.0-2-amd64-unsigned (left
over from a previous release and churn in sid) but also "dpkg -i"
installed packages like crossover and and draw.io (where those package
names do not exist in Debian). My own backports are not included because
the package names exist. Likewise packages like rclone I've grabbed the
upstream .deb and installed it on this machine are not included.

aptitude search
'?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'
apt list | grep 'installed,local'
This is a monster (it may well have been reinvented multiple times,
but I will own up to responsibility for creating it at least one time
c2009). It's filtering out apt's magic repo of 'now', which accompanies
all installed packages (a bit like /var/lib/dpkg/status in the output of
apt-cache policy foo), to find packages that are currently installed but
for which (name, version, architecture) doesn't exist in a currently
configured repo. On this machine, it matches some 3rd party packages
that are installed via dpkg -i and for which there is no apt repo, like
crossover (just like 'obsolete') as well as local backports that I've
made that are not sitting in any repo, and those upstream .debs like
rclone. Backports that I poked into my own local repo are not included
because they are available. Packages that I installed directly out of
sid onto this bullseye machine and have never upgraded are also on the
list because the version part of the (name, version, arch) tuple doesn't
match. (The treatment of local repositories is slightly different for
the two versions of this command given above)

aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
apt list '?narrow(?installed, ?not(?origin(Debian)))'
This looks for packages not currently installable from a repo with
'Debian' as the name in the Release file, which includes the case where
(name, version, arch) doesn't match. Because it's matching on the
version string of the package, it spots that upstream .deb for rclone,
the local backports, etc. It also adds in 3rd party packages that are in
apt repositories, so on this machine signal-desktop and slack-desktop.
Note that lots of 3rd party repositories are misconfigured which is
unhelpful.


Which of the above is useful depends on the stage of the upgrade.
Looking for obsolete packages after the upgrade is an important part of
the cleanup process. Looking for 3rd party packages before the upgrade
is an important part of the preparation.

Importantly, the output of any of these commands will depend on whether
the sources.list has been edited; we see that quite frequently in
#debian, where someone has edited the sources.list for the upgrade and
is then freaked out by absolutely every single package suddenly being
'not available'.

cheers
Stuart

[Justin B Rye]

Post by Antoine Beaupre
The release notes, in sections 4.2.2 and 4.8, actually suggest *three*

I don't think you mean "orphan" in either of the senses known to
"https://wiki.debian.org/Glossary#orphan".

Post by Antoine Beaupre
aptitude search '~o'

This is nice and simple, but frankly as an aptitude user I wouldn't
bother. Instead I'd do what the text just above mentions - launch
aptitude, notice that there was a category for "Obsolete and Locally
Created Packages" (which would tell me I'd somehow lost track of my
kernel packages), and purge everything in that category from the
full-screen interface, without going back to the commandline.

Post by Antoine Beaupre
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

This one (apparently an improvement on the '~i(!~ODebian)' search that
was recommended for buster, though the logic is too subtle for me to
remember) is looking for a slightly different thing at a different
stage in the upgrade process: it's part of the section about getting
rid of "non-Debian packages" *before* the upgrade. The '~o' and
'!~ODebian' searches find different kinds of "unwanted" package.

Post by Antoine Beaupre
apt-forktracer | sort

I've never quite been able to understand how it is that anybody
could get themselves into the situation of *needing* this specialised
package installed to work around the weirdness of their setup, but
still need to be told what it is that's unusual about their system.

Post by Antoine Beaupre
apt-show-versions | grep -v /bullseye

This is another package I've never needed to install on my stable
desktop precisely because it's my stable desktop. If I had a reason
to install it, presumably I'd already know about the reason, and step
one in the bullseye upgrade process should be to get rid of that.

Post by Antoine Beaupre
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

Yes, that's the one you listed above.

Post by Antoine Beaupre
aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'

Does that do something similar to the above, but less intelligibly, or
something different?

Post by Antoine Beaupre
I frankly don't quite know where I stand with all this anymore, but I
am getting the strong feeling we're sending an incoherent message
here. :)

It's two different messages in two different parts of the release
notes. The first message is roughly "before the upgrade, look at what
you've got installed. If it's a mix of complex pins and PPA packages
and nonsense like that, start by getting rid of all the crazy stuff".
Unfortunately, this relies on the reader to apply some common sense,
so we've fallen into the trap of replacing "subjective" advice with
"objective" diagnostic commandlines.

The second message is "after the upgrade, throw out all the stuff
that isn't supported any longer". This really is trivially easy to
automate, as long as you don't confuse it with the previous task.

Post by Antoine Beaupre
In my personal documentation, I've settled on `apt-forktracer`,

I'd be interested to know what you find it useful for.

Post by Antoine Beaupre
but I
suspect we might want to stick with `aptitude search '~obsolete'`
because that matches other documentation in the release notes (and
allows for easy purging).

But it isn't looking for the same thing.

Post by Antoine Beaupre
Is there any reason why we have all that diversity?
What's the right way to do what we actually want here?

[---suture---]

Post by Antoine Beaupre
apt list ~obsolete

And indeed for section 4.8, which is dealing with tidying up *after*
the upgrade, it might make sense to recommend the use of apt instead
of aptitude here.

Post by Antoine Beaupre
comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u)
apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'

If they're not getting shorter, you're going the wrong way.

[Antoine Beaupré]

Post by Justin B Rye

Post by Antoine Beaupre
The release notes, in sections 4.2.2 and 4.8, actually suggest *three*

I don't think you mean "orphan" in either of the senses known to
"https://wiki.debian.org/Glossary#orphan".

I should have said "obsolete", I think.

Post by Justin B Rye

Post by Antoine Beaupre
aptitude search '~o'

This is nice and simple, but frankly as an aptitude user I wouldn't
bother. Instead I'd do what the text just above mentions - launch
aptitude, notice that there was a category for "Obsolete and Locally
Created Packages" (which would tell me I'd somehow lost track of my
kernel packages), and purge everything in that category from the
full-screen interface, without going back to the commandline.

I think the point here is that you can do:

aptitude purge '~o'

... to avoid loading the GUI.

Post by Justin B Rye

Post by Antoine Beaupre
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

This one (apparently an improvement on the '~i(!~ODebian)' search that
was recommended for buster, though the logic is too subtle for me to
remember) is looking for a slightly different thing at a different
stage in the upgrade process: it's part of the section about getting
rid of "non-Debian packages" *before* the upgrade. The '~o' and
'!~ODebian' searches find different kinds of "unwanted" package.

Maybe it would be worth clarifying that distinction then?

It might help if the former `~o` is expanded to `?obsolete` in the text,
to make it clearer how it compares with the latter.

Post by Justin B Rye

Post by Antoine Beaupre
apt-forktracer | sort

I've never quite been able to understand how it is that anybody
could get themselves into the situation of *needing* this specialised
package installed to work around the weirdness of their setup, but
still need to be told what it is that's unusual about their system.

I actually find the output of apt-forktracer to be quite handy.

Post by Justin B Rye

Post by Antoine Beaupre
apt-show-versions | grep -v /bullseye

This is another package I've never needed to install on my stable
desktop precisely because it's my stable desktop. If I had a reason
to install it, presumably I'd already know about the reason, and step
one in the bullseye upgrade process should be to get rid of that.

Yeah, although I guess that's the point of those commands: figure out if
there's a mess in there.

I find apt-show-versions to be comparable to apt-forktracer, but a bit
more flexible.

Post by Justin B Rye

Post by Antoine Beaupre
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

Yes, that's the one you listed above.

Ah yes, sorry for the dupe.

Post by Justin B Rye

Post by Antoine Beaupre
aptitude search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'

Does that do something similar to the above, but less intelligibly, or
something different?

I frankly have no idea anymore.

Post by Justin B Rye

Post by Antoine Beaupre
I frankly don't quite know where I stand with all this anymore, but I
am getting the strong feeling we're sending an incoherent message
here. :)

It's two different messages in two different parts of the release
notes. The first message is roughly "before the upgrade, look at what
you've got installed. If it's a mix of complex pins and PPA packages
and nonsense like that, start by getting rid of all the crazy stuff".
Unfortunately, this relies on the reader to apply some common sense,
so we've fallen into the trap of replacing "subjective" advice with
"objective" diagnostic commandlines.

I see. Maybe a quick oneliner explanation before the command could help
alleviate that confusion then...

Post by Justin B Rye
The second message is "after the upgrade, throw out all the stuff
that isn't supported any longer". This really is trivially easy to
automate, as long as you don't confuse it with the previous task.

Same here, I guess.

Post by Justin B Rye

Post by Antoine Beaupre
In my personal documentation, I've settled on `apt-forktracer`,

I'd be interested to know what you find it useful for.

I like that it shows the related versions in the archive, and that it
has very terse output.

Post by Justin B Rye

Post by Antoine Beaupre
but I
suspect we might want to stick with `aptitude search '~obsolete'`
because that matches other documentation in the release notes (and
allows for easy purging).

But it isn't looking for the same thing.

Okay, so just stick to aptitude in both cases then, and don't introduce
apt-forktracer. :)

Post by Justin B Rye

Post by Antoine Beaupre
Is there any reason why we have all that diversity?
What's the right way to do what we actually want here?

[---suture---]

Post by Antoine Beaupre
apt list ~obsolete

And indeed for section 4.8, which is dealing with tidying up *after*
the upgrade, it might make sense to recommend the use of apt instead
of aptitude here.

Yeah, and then we get rid of aptitude in the docs in bullseye +1 :)

So, TL;DR: we should have this before, to find and cleanup foreign
packages:

aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

Presumably `apt list` might support that as well in bullseye?

Then after the upgrade to bullseye, we use this to find unsupported
/ removed packages:

apt list ~obsolete

And the release notes could use their own TL;DR: before those commands. :)

Post by Justin B Rye

Post by Antoine Beaupre
comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u)
apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'

If they're not getting shorter, you're going the wrong way.

Definitely. That was mostly meant as a joke.

[Justin B Rye]

Post by Antoine Beaupré

Post by Justin B Rye

Post by Antoine Beaupre
aptitude search '~o'

This is nice and simple, but frankly as an aptitude user I wouldn't
bother. Instead I'd do what the text just above mentions - launch
aptitude, notice that there was a category for "Obsolete and Locally
Created Packages" (which would tell me I'd somehow lost track of my
kernel packages), and purge everything in that category from the
full-screen interface, without going back to the commandline.

aptitude purge '~o'
... to avoid loading the GUI.

I'd imagine it's to avoid having to deal with that interface - it's
much more powerful (and indeed faster) if you're used to it, but we
don't want novice users to need to guess it's "↓↓↓_gg"!

Post by Antoine Beaupré

Post by Justin B Rye

Post by Antoine Beaupre
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

This one (apparently an improvement on the '~i(!~ODebian)' search that
was recommended for buster, though the logic is too subtle for me to
remember) is looking for a slightly different thing at a different
stage in the upgrade process: it's part of the section about getting
rid of "non-Debian packages" *before* the upgrade. The '~o' and
'!~ODebian' searches find different kinds of "unwanted" package.

Maybe it would be worth clarifying that distinction then?

I don't know how we make texts more separate than printing them
in two different places with explanatory paragraphs that talk
about different things! But let me have a look.

# 4.2.2. Remove non-Debian packages
#
# Below there are two methods for finding installed packages that did
# not come from Debian, using either aptitude or apt-forktracer.
# Please note that neither of them are 100% accurate (e.g. the
# aptitude example will list packages that were once provided by
# Debian but no longer are, such as old kernel packages).

Now that you come to mention it I've always thought that was a bad
example, since after all it isn't exactly a false-positive - old
linux-images really are no-longer-Debian packages, and if you've got
some lying around even before the upgrade, this would be an
appropriate time to get rid of them, as we go on to say a little
later. Wait, why *is* 4.2.2 before 4.2.3? shouldn't it be
* sort out the package database (pending actions or whatever)
* make sure you're at the latest point release
* remove any non-Debian packages
* similarly but separately, remove any obsolete packages
* tidy any relic configs
[...]
Maybe this is turning into the sort of disruptive low-priority change
that should wait for next time.

Post by Antoine Beaupré
It might help if the former `~o` is expanded to `?obsolete` in the text,
to make it clearer how it compares with the latter.

Unfortunately people also use "obsolete" to mean "autoremovable"!
And this also makes the search less simple and easier to confuse with
the one in 4.2...

[...]

Post by Antoine Beaupré

Post by Justin B Rye

Post by Antoine Beaupre
In my personal documentation, I've settled on `apt-forktracer`,

I'd be interested to know what you find it useful for.

I like that it shows the related versions in the archive, and that it
has very terse output.

Yes, but how do you come to be running a system with non-Debian
repositories in your sources and installing packages to inspect the
gory details without already realising you've done that?

Now that we've got "https://deb.debian.org/debian/", we're close to
being able to say that standard procedure is "for the duration of the
upgrade, comment out any lines that don't match that URL".

Post by Antoine Beaupré

Post by Justin B Rye
[---suture---]

Post by Antoine Beaupre
apt list ~obsolete

And indeed for section 4.8, which is dealing with tidying up *after*
the upgrade, it might make sense to recommend the use of apt instead
of aptitude here.

Yeah, and then we get rid of aptitude in the docs in bullseye +1 :)

Aptitude may no longer be recommended for dist-upgrades, but there are
still reasons to prefer it for day-to-day admin on stable systems.
For bullseye-to-bookworm we'll probably want to use apt recipes
wherever possible, with at most a mention that aptitude can also do
this from the fullscreen mode. But since we also point at 4.8 from
4.2, we can't do that yet; we don't want the extra complication of
having to say "if you haven't upgraded yet then use aptitude".

Post by Antoine Beaupré
So, TL;DR: we should have this before, to find and cleanup foreign
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
Presumably `apt list` might support that as well in bullseye?

Afraid not - "E: Regex compilation error".

[Hendrik Boom]

Post by Justin B Rye
# 4.2.2. Remove non-Debian packages
#
# Below there are two methods for finding installed packages that did
# not come from Debian, using either aptitude or apt-forktracer.
# Please note that neither of them are 100% accurate (e.g. the
# aptitude example will list packages that were once provided by
# Debian but no longer are, such as old kernel packages).
Now that you come to mention it I've always thought that was a bad
example, since after all it isn't exactly a false-positive - old
linux-images really are no-longer-Debian packages, and if you've got
some lying around even before the upgrade, this would be an
appropriate time to get rid of them, as we go on to say a little
later.

Old kernels are sometimes kept around as a kind of backstop in
case a new kernel turns out not to work properly.

...
...

Post by Justin B Rye
Yes, but how do you come to be running a system with non-Debian
repositories in your sources and installing packages to inspect the
gory details without already realising you've done that?

You may have forgotten.

You may have long ago enabled a nonDebian repository to install some
nonDebian package. Unbeknownst to you, that repository also contained
variants of debian packages which ended up replacing the Debian packages
you expected to keep.

A real mess. They look like Debian packages, but they are not.

Th the extent that the new packages have more recent version numbers than
the intruding packages, things may still go well.

Post by Justin B Rye
Now that we've got "https://deb.debian.org/debian/", we're close to
being able to say that standard procedure is "for the duration of the
upgrade, comment out any lines that don't match that URL".

Sounds like a valid thing to do, anyway.

-- hendrik

[Justin B Rye]

Post by Hendrik Boom

Post by Justin B Rye
# Please note that neither of them are 100% accurate (e.g. the
# aptitude example will list packages that were once provided by
# Debian but no longer are, such as old kernel packages).
Now that you come to mention it I've always thought that was a bad
example, since after all it isn't exactly a false-positive - old
linux-images really are no-longer-Debian packages, and if you've got
some lying around even before the upgrade, this would be an
appropriate time to get rid of them, as we go on to say a little
later.

Old kernels are sometimes kept around as a kind of backstop in
case a new kernel turns out not to work properly.

Sure, that can be how you come to have old ones unpurged; but you
can't safely treat a kernel from what's now oldstable as "known good",
because when you reboot into it you might quite easily discover that
it's incompatible with the stable udev/libc/whatever.

Post by Hendrik Boom

Post by Justin B Rye
Yes, but how do you come to be running a system with non-Debian
repositories in your sources and installing packages to inspect the
gory details without already realising you've done that?

You may have forgotten.
You may have long ago enabled a nonDebian repository to install some
nonDebian package. Unbeknownst to you, that repository also contained
variants of debian packages which ended up replacing the Debian packages
you expected to keep.
A real mess. They look like Debian packages, but they are not.
Th the extent that the new packages have more recent version numbers than
the intruding packages, things may still go well.

It's a pity "apt policy" is no easier to interpret than the old
apt-cache equivalent. But this does begin to make a bit more sense
when I consider the issue of pinning - you can't check your system is
sane *just* by a glance at /etc/apt/sources.list. Plus, tools like
this can make obvious sense on a testing/unstable development system,
so you might have got used to it there and then installed it on your
stable desktop as well.

Post by Hendrik Boom

Post by Justin B Rye
Now that we've got "https://deb.debian.org/debian/", we're close to
being able to say that standard procedure is "for the duration of the
upgrade, comment out any lines that don't match that URL".

Sounds like a valid thing to do, anyway.

We might (next time round) say something to the effect that "the
recommended setup is to have a basic sources.list as similar to fig 1
as possible. Although many variations are possible and will work
reliably on a stable system, you should consider simplifying things
for the upgrade".

[Antoine Beaupré]

[...]

Post by Justin B Rye

Post by Antoine Beaupré

Post by Justin B Rye

Post by Antoine Beaupre
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'

This one (apparently an improvement on the '~i(!~ODebian)' search that
was recommended for buster, though the logic is too subtle for me to
remember) is looking for a slightly different thing at a different
stage in the upgrade process: it's part of the section about getting
rid of "non-Debian packages" *before* the upgrade. The '~o' and
'!~ODebian' searches find different kinds of "unwanted" package.

Maybe it would be worth clarifying that distinction then?

I don't know how we make texts more separate than printing them
in two different places with explanatory paragraphs that talk
about different things! But let me have a look.
# 4.2.2. Remove non-Debian packages
#
# Below there are two methods for finding installed packages that did
# not come from Debian, using either aptitude or apt-forktracer.
# Please note that neither of them are 100% accurate (e.g. the
# aptitude example will list packages that were once provided by
# Debian but no longer are, such as old kernel packages).
Now that you come to mention it I've always thought that was a bad
example, since after all it isn't exactly a false-positive - old
linux-images really are no-longer-Debian packages, and if you've got
some lying around even before the upgrade, this would be an
appropriate time to get rid of them, as we go on to say a little
later. Wait, why *is* 4.2.2 before 4.2.3? shouldn't it be
* sort out the package database (pending actions or whatever)
* make sure you're at the latest point release
* remove any non-Debian packages
* similarly but separately, remove any obsolete packages
* tidy any relic configs
[...]
Maybe this is turning into the sort of disruptive low-priority change
that should wait for next time.

Indeed. It does seem like 4.2.3 should be before 4.2.2.

I am not sure we should tell people to "remove any non-Debian package"
before the upgrade. They might have legitimate reasons to have
third-party package repositories...?

Post by Justin B Rye

Post by Antoine Beaupré
It might help if the former `~o` is expanded to `?obsolete` in the text,
to make it clearer how it compares with the latter.

Unfortunately people also use "obsolete" to mean "autoremovable"!
And this also makes the search less simple and easier to confuse with
the one in 4.2...

I just meant to avoid using the shortcut, obscuring it won't help with
this sort of confusiong.

Post by Justin B Rye
[...]

Post by Antoine Beaupré

Post by Justin B Rye

Post by Antoine Beaupre
In my personal documentation, I've settled on `apt-forktracer`,

I'd be interested to know what you find it useful for.

I like that it shows the related versions in the archive, and that it
has very terse output.

Yes, but how do you come to be running a system with non-Debian
repositories in your sources and installing packages to inspect the
gory details without already realising you've done that?

You have forgotten, it's been years since you messed around with your
sources.list.

Post by Justin B Rye
Now that we've got "https://deb.debian.org/debian/", we're close to
being able to say that standard procedure is "for the duration of the
upgrade, comment out any lines that don't match that URL".

I am not sure we want the release notes to standardize on that as
well... But I have considered writing a parser that checks sources.list
Origin headers and fixes things up correctly. Because my checklist is
getting ridiculous:

: Check for pinned, on hold, packages, and possibly disable &&
rm -f /etc/apt/preferences /etc/apt/preferences.d/* &&
rm -f /etc/apt/sources.list.d/backports.debian.org.list &&
rm -f /etc/apt/sources.list.d/backports.list &&
rm -f /etc/apt/sources.list.d/bullseye.list &&
rm -f /etc/apt/sources.list.d/buster-backports.list &&
rm -f /etc/apt/sources.list.d/experimental.list &&
rm -f /etc/apt/sources.list.d/incoming.list &&
rm -f /etc/apt/sources.list.d/proposed-updates.list &&
rm -f /etc/apt/sources.list.d/sid.list &&
rm -f /etc/apt/sources.list.d/testing.list &&
apt update && apt -y upgrade &&

Post by Justin B Rye

Post by Antoine Beaupré

Post by Justin B Rye
[---suture---]

Post by Antoine Beaupre
apt list ~obsolete

And indeed for section 4.8, which is dealing with tidying up *after*
the upgrade, it might make sense to recommend the use of apt instead
of aptitude here.

Yeah, and then we get rid of aptitude in the docs in bullseye +1 :)

Aptitude may no longer be recommended for dist-upgrades, but there are
still reasons to prefer it for day-to-day admin on stable systems.

I'm not arguing for deprecating aptitude altogether, but it would seem
to me that using less tools in the release notes would be better.

Post by Justin B Rye
For bullseye-to-bookworm we'll probably want to use apt recipes
wherever possible, with at most a mention that aptitude can also do
this from the fullscreen mode. But since we also point at 4.8 from
4.2, we can't do that yet; we don't want the extra complication of
having to say "if you haven't upgraded yet then use aptitude".

Agreed.

Post by Justin B Rye

Post by Antoine Beaupré
So, TL;DR: we should have this before, to find and cleanup foreign
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
Presumably `apt list` might support that as well in bullseye?

Afraid not - "E: Regex compilation error".

That's really too bad! ;)

a.

[Andrei POPESCU]

Post by Antoine Beaupré
I am not sure we should tell people to "remove any non-Debian package"
before the upgrade. They might have legitimate reasons to have
third-party package repositories...?

A dist-upgrade is difficult enough, reducing complexity introduced by
non-Debian packages and/or repositories (of questionable quality) seems
obvious to me.

Kind regards,
Andrei

[Richard Lewis]

Post by Antoine Beaupre
The release notes, in sections 4.2.2 and 4.8, actually suggest *three*
aptitude search '~o'
aptitude search '?narrow(?installed, ?not(?origin(Debian)))'
apt-forktracer | sort

I think this bug has been fixed.

Current text is pretty clear there are two separate issues:
- "non-debian packages" (4.2.2, which has the second two commands):
remove before upgrading
(and (perhaps no-longer?) mentions orphaned packages at all)
- "removed from debian" (4.8, which has the first command): can only
be done after upgrading

It is perhaps worth considering more clearly splitting "before the release" and
"after the release" actions into separate chapters? (post bullseye?) -
eg 4.7 and 4.8 could be put at the start of chapter 5
as they seem like 'issues to be aware of for the new release' as much
as the are about 'upgrading'.

[Manny]

Package: release-notes
Followup-For: Bug #987017
X-Debbugs-Cc: debbug.release-***@sideload.33mail.com

@ Antoine Beaupre

Post by Antoine Beaupre
Is there any reason why we have all that diversity?
…
I'm not arguing for deprecating aptitude altogether, but it would seem
to me that using less tools in the release notes would be better.

As an aptitude user, I was bothered by the lack of aptitude ways of
doing things in the upgrade guide. I had to research and use guesswork
to work out what is the aptitude-equivalent command, which complicated
my effort. For the minimal system upgrade, users were directed to run
“apt upgrade --without-new-pkgs”. I thought why is the aptitude
approach missing? Is aptitude incapable of this? I then went to the
man page and found “aptitude safe-upgrade”. I am still not sure if
that is the equivalent command. And if so, is it exacly the same or
are there differences?

Whether someone wants to know a bit of many tools or be a master of
few tools is a matter of preference, but the docs would ideally
accommodate both kinds of users (though not necessarily in the same
doc… that’s another matter - but if the different methods are
side-by-side in the same doc it helps users learn about the
equivalences and makes it easier for them to settle on a preferred
method). But certainly it’s sensible to drop methods that have no
advantage of any kind.

The advice I was given early in my Debian years was that apt-get was a
more primitive command and aptitude was more complete/comprehensive,
that it logs or tracks more things and generally the better tool
according to folks giving IRC support. I think aptitude calls apt
IIRC, which makes it a higher level tool.

Post by Antoine Beaupre
I am not sure we should tell people to "remove any non-Debian package"
before the upgrade. They might have legitimate reasons to have
third-party package repositories...?

Concur. I’m not sure what the past release notes said, but the
Bookworm release notes simply bluntly direct users to “Remove
non-Debian packages”. This was frustrating for me. **Why?** I want to
know why I am doing something. The list of non-Debian packages
contained pkgs *I need*. Users need to know why they are directed to
destroy something they need.

Is there a real likeliness that a non-Debian pkg will make a mess or
disaster of the upgrade? Or is this step a generic “we only
officially support our officially distributed software” scenario?

I decided to go against the guidance. There was one non-Debian pkg
that I no longer used, so removal was a trivial choice for that
one. But I left the other non-Debian pkgs alone. Some of them broke
and some survived.

The guide should probably suggest removing any non-Debian pkgs that
are not needed to mitigate dependency complications, but simply warn
that non-Debian pkgs allowed to persist might not run correctly and
should be also treated with low priority if conflicts arise.

@ Justin B Rye

Post by Antoine Beaupre

Post by Antoine Beaupre
aptitude search '~o'

This is nice and simple, but frankly as an aptitude user I wouldn't
bother. Instead I'd do what the text just above mentions - launch
aptitude, notice that there was a category for "Obsolete and Locally
Created Packages"

If the guide is intended to help train the user and advance their
Debian skills, then the CLI advice is probably favorable because it’s
more likely to improve the user’s knowledge than a UI that needs no
manual.

If the guide is intended to just execute steps to do a job (git’r
done), then the CLI is still the winner because it’s just one line for
a user to copy-paste and get instant results with minimal text and
minimal reading.

Briefly mentioning the UI option probably doesn’t hurt though in
addition to the CLI.

Post by Antoine Beaupre

Post by Antoine Beaupre
apt-forktracer | sort

I've never quite been able to understand how it is that anybody
could get themselves into the situation of *needing* this specialised
package installed to work around the weirdness of their setup, but
still need to be told what it is that's unusual about their system.
…

Post by Antoine Beaupre
In my personal documentation, I've settled on `apt-forktracer`,

I'd be interested to know what you find it useful for.

For me, apt-forktracer gives 1 pkg additional output:

$ apt-forktracer | sort
linux-image-5.10.0-16-amd64 (5.10.127-2)
linux-image-5.10.0-19-amd64 (5.10.149-2)
linux-image-5.10.0-6-amd64 (5.10.28-1)
linux-image-5.10.0-7-amd64 (5.10.40-1)
linux-image-5.10.0-8-amd64 (5.10.46-5)
mastodon-archive (1.4.4-izzy1)
openjdk-11-jre (11.0.23+9-1~deb11u1) [Debian: 11.0.22+7-1~deb11u1]
openjdk-11-jre-headless (11.0.23+9-1~deb11u1) [Debian: 11.0.22+7-1~deb11u1]
ungoogled-chromium (90.0.4430.212-1.sid1)
ungoogled-chromium-common (90.0.4430.212-1.sid1)
ungoogled-chromium-sandbox (90.0.4430.212-1.sid1)
ungoogled-chromium-shell (90.0.4430.212-1.sid1)
wire-desktop (3.35.3348-3348)
xdiskusage (1.60-4~bpo12+1) [Debian Backports: 1.60-4~bpo12+1] [Debian: 1.48-10.1+b1]

The “apt list '?narrow(?installed, ?not(?origin(Debian)))'” command
excludes the last line (xdiskusage). If those two commands are
expected to have identical results, then maybe we have a bug.

Post by Antoine Beaupre

Post by Antoine Beaupre
comm -23 <(dpkg-query -W -f '${db:Status-Abbrev}\t${Package}\n' | grep '^.[^nc]' | cut -f2 | sort) <(apt-cache dumpavail | sed -rn 's/^Package: (.*)/\1/p' | sort -u)
apt list --installed | awk -F/ '/\[installed,local\]/{print $1}'

If they're not getting shorter, you're going the wrong way.

Yes, assuming the intent of the guide is purely to get a job
done. OTOH, the long versions are a good learning opportunity.

If this were a PDF-formatted textbook, it would be tempting to put the
lengthy but academically rich versions in a distinctly colored
sideline box maybe with a graduation hat or some nerdy symbol in the
corner to make it easy for people in git’r done mode to ignore and
simultaneously easy for curious minds to study.

Post by Antoine Beaupre
Aptitude may no longer be recommended for dist-upgrades, but there are
still reasons to prefer it for day-to-day admin on stable systems.
For bullseye-to-bookworm we'll probably want to use apt recipes
wherever possible, with at most a mention that aptitude can also do
this from the fullscreen mode. But since we also point at 4.8 from
4.2, we can't do that yet; we don't want the extra complication of
having to say "if you haven't upgraded yet then use aptitude".

Can you elaborate? Why would aptitude lose ground on dist upgrades?
Is it deficient in some way?

As an aptitude user, my temptation is to look for the aptitude
approach. So merely omitting aptitude from the guide only encumbers
aptitude users. If there is a good reason for omitting an aptitude
approach, the guide should state why. Otherwise users might question
the quality and comprehensiveness of the guide.

[Richard Lewis]

Hi,
[Release Team member hat on]

no hats here, but wanted to agree

Post by Manny
As an aptitude user, I was bothered by the lack of aptitude ways of
doing things in the upgrade guide.

I anything, I prefer the Release Notes to move to using one tool in
the instructions, without insinuating that it's the only way. I think
that tool should be apt nowadays. We've made steps in that direction
during the last release cycle, i.e. we moved away from aptitude.

I agree. i myself use aptitude for upgrading to stable - it's a lot more
work than using apt, and I dont think it is suitable approach for new
users. (eg i am not sure that "aptitude safe-upgade" is actually
entirely equivalent to the first stage of apt, in some subtle ways).

I definitely dont want the release-notes to list every single possibile
way of doing things. It would make everything seem more complicated and
be very confusing - and need a lot of maintenance: i think that the
release-notes already try to do too much in some places.

Debian should be recomending a way to do things in a simple way that
will work. Advanced users can always do their own thing, but the
official release-notes should be simple -- it also helps people who want
to do their own thing if "the official thing" is clearly documented.

So I think release-notes should only use apt, with a brief note that
says that other front-ends like aptitude, synaptic are avialable. The
developers and users of those tools can write their own guide, if they
want, which could be linked to.

Post by Manny
If the guide is intended to help train the user and advance their
Debian skills, then the CLI advice is probably favorable because it’s
more likely to improve the user’s knowledge than a UI that needs no
manual.

That's not the purpose of the Release Notes.

(is there a case to think of something like this as a secondary
objective?, one to be kept in mind where possible, and only where it
doesnt conflict with a clear, short document. either way, documenting
objectives would help contributors)

Post by Manny
As an aptitude user, my temptation is to look for the aptitude
approach. So merely omitting aptitude from the guide only encumbers
aptitude users. If there is a good reason for omitting an aptitude
approach, the guide should state why. Otherwise users might question
the quality and comprehensiveness of the guide.

We could add a statement that while more tools exist. All automated
testing of upgrades that I know of use apt-get, so that's the obvious
choice. aptitude doesn't get as much testing.

i fear aptitude is mostly bugfix-only, whereas apt is actively
maintained with new features.

I really like the text interface in aptitude as it makes it easier to
track manual/automatic status and helps when you have complex
requirements like "install diffoscope-minimal but only some all its ~1GB
of transitive recommends" - but when I just want "the default" (the vast
majority of the time!), it doesnt actually add much over apt.

[R Lewis]

Post by Manny
As an aptitude user, I was bothered by the lack of aptitude ways of
doing things in the upgrade guide.

I anything, I prefer the Release Notes to move to using one tool in the
instructions, without insinuating that it's the only way. I think that
tool should be apt nowadays.

This bug got side-tracked. It was originally about the commands to search
for old/obsolete packages. The consensus was that we list 2 commands in 2
sections as they do different things. So no change needed


There was also consesnsue to swap sections around 4.2.2 (remove non-debian
packages) and 4.2.3 (title not stated). However, current 4.2.2 is upgrade
to the latest point release, and 4.2.3 is remove debian backports.
Non-debian packages is now down at 4.2.6 -- so i think this bit has been
solved.

It diverged into a discussion of whether to also mention aptitude variants.
The consensus was to stick with apt, as much as possible, but dont say
other options cannot be used. So no change needed.


Did i miss anything or should we close this bug?