Bug#1033065: release-notes: i386 notes should specify minimum CPU requirements

[Martin-Éric Racine]

Package: release-notes
Severity: important

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

The release notes for i386 should specify the minimum CPU requirements.

While a kernel compiled for Geode LX (essentially a basic i686 without the optional CPU features) still ships in Debian, many packages enforce CPU flags that barf on anything older than 686-PAE.

One such package is 'sudo' (Bug #1004894) where upstream unilaterally enabled -fcf-protection and the Debian maintainer refuses to comment out the corresponding ./configure macros.

Additionally, other packages apparently disregard build-essentials' GCC defaults and instead configure flags to match the CPU features of the build host, which produces binaries that segfault on a Geode.

Martin-Éric

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEyJACx3qL7GpObXOQrh+Cd8S017YFAmQTa+AACgkQrh+Cd8S0
17YqHRAAtsb991JnECg5MgWkkH2XwYoGgSh2N15aPOta1Gt/MqP2pdYjfDEPvsqm
Mxsq8Q+OMdNXd/i5ZglqRbpRWUO8TGOQzL+keD3U3RBss9zhFqSiv47S6eZM6R1N
RefspxicGI/7WmMUW8tdnU6pmtnEiw1czjLlZ2WICZ1kYB3MknTxlLpxp8IxnDEX
cPyKbVPd/jtdzOysNILfPZiFFsPBcOsPzYSgLxvZRZk0XzoLQQLXK5ZNb306Z8hv
wDW47WB+iHBVNd0WiZaFod8fJMELIdqzcLUgv6I1fHzB0VnwkuECdr0RFSG+vuL8
C6APU+63p9C3Py9gmwI1fRfThBdX4c1qgKCY+kDxoAf8CGjma0vgdS65bJb7Lfes
JwpyxXX5SXqdHGWuqd3pqVYO523cKOzZSWoFiwhFaopyjxLWrTMqvVXuWLl7Mawp
oqrkjWdgogyPiZGWa/xTqccuKDjqClUQM3ZjcEPKx4/R4m2iMVMTsbqZfMgRlU/H
Z5zTQUleGH4EsE0XnTF0R2gf2OCJrlxT2xyqEx72c1/edZwKfQdekXraFOv+TudZ
JKgL43lbqem90+0+QoIjWDl2PV9Uy1R1PEg0ouHrinW6ZpZlJN9M4JTTvfXg6JEJ
RHvV/8wEx4UpUOlx9Zw2mkruHKKEwo

[James Addison]

Package: release-notes
Followup-For: Bug #1033065
X-Debbugs-Cc: ***@bugs.debian.org
Control: affects 1005863 net-tools
Control: tags -1 moreinfo

Post by Martin-Éric Racine
While a kernel compiled for Geode LX (essentially a basic i686 without the optional CPU features) still ships in Debian, many packages enforce CPU flags that barf on anything older than 686-PAE.

Possibly a naive question: could you provide a command/script to check input
binaries (such as those in the i386 archive) to see whether they are likely to
be affected by this?

(my thinking there is: if that flags a small _number_ of packages then
individual patches could make sense while we await upstream activity.
conversely, if it's a high _percentage_ of packages, then there is an important
decision to make)

I don't have a sense for what proportion of Debian's i386 installs could be on
Geode systems, but i386 is our second-most-widely deployed architecture
(between amd64 and arm64) today according to popcon.

[Debian Bug Tracking System]

Post by James Addison
affects 1005863 net-tools

Bug #1005863 [gcc-11] gcc-11: invalid opcode for Geode LX on i386
Added indication that 1005863 affects net-tools

Post by James Addison
tags -1 moreinfo

Bug #1033065 [release-notes] release-notes: i386 notes should specify minimum CPU requirements
Added tag(s) moreinfo.

[James Addison]

Package: release-notes
Followup-For: Bug #1033065
X-Debbugs-Cc: martin-***@iki.fi

Hi Martin-Éric - I intended to send my previous comment to you, but forgot to
add you to on carbon-copy.

Roughly speaking: I'm wondering whether there is a way that we can scan i386
architecture packages in (bookworm only?) archives to figure out th

[James Addison]

Package: release-notes
Followup-For: Bug #1033065
X-Debbugs-Cc: martin-***@iki.fi

I've used the following commands to confirm that the i386 sudo/1.9.9-1 package
contains the bugreport-relevant NOPL opcode:

# obtain an archived copy of the affected binary package
$ wget2 http://snapshot.debian.org/archive/debian/20220202T154459Z/pool/main/s/sudo/sudo_1.9.9-1_i386.deb

# unpack the contents of the package into a corresponding directory name
$ dpkg -x sudo_1.9.9-1_i386.deb sudo_1.9.9-1_i386

# iterate over each executable file extracted from the package
# - print each filename so that we can identify potentially-affected files
# - check for the 'NOPL' opcode in all i386 code sections of the file
$ while IFS= read -r -d '' file; do

echo "$file"
objdump --architecture=i386 --disassemble-all "$file" | grep -w "nopl"
done < <(find . -type f -executable -print0)

It should be possible to generalise this further to scan a larger set of
packages.

The 'grep' for the word 'nopl' seems potentially fragile. If there's a
more-precise and/or less-false-positive-prone way to check whether each file
contains the 'nopl' opcode (and I'd expect that there is), then that'd be a
welcome improvement.

[Paul Wise]

The 'grep' for the word 'nopl' seems potentially fragile.  If there's a
more-precise and/or less-false-positive-prone way to check whether each file
contains the 'nopl' opcode (and I'd expect that there is), then that'd be a
welcome improvement.

Broadly speaking, detecting non-baseline instruction usage isn't
possible without false positives, because the program could use runtime
instruction selection based on the current CPU to avoid currently
unavailable instructions, while the binary would still contain those
instructions for use on other CPUs.

https://wiki.debian.org/InstructionSelection

Of course you could do the scanning and then use autopkgtests or manual
tests to find out if the found programs work on the relevant CPUs.

Perhaps lintian could add classification tags for the relevant CPU
instructions and then the i386 port could have extra autopkgtest nodes
that only process the packages detected by lintian.

[Paul Gevers]

Hi Martin-Éric,

Post by Martin-Éric Racine
The release notes for i386 should specify the minimum CPU requirements.

Thanks for letting us know. If we raise the baseline, historically we
mention it in the release notes. I (as a member of the release team)
wasn't aware we might have done so until now.

Post by Martin-Éric Racine
Additionally, other packages apparently disregard build-essentials' GCC defaults and instead configure flags to match the CPU features of the build host, which produces binaries that segfault on a Geode.

Those are in principle all RC bugs and don't only impact Geode. Please
file bugs when you find them.

Paul (who is trying to get some more background on the topic)

[Martin-Éric Racine]

Post by Paul Gevers

Post by Martin-Éric Racine
The release notes for i386 should specify the minimum CPU requirements.

Thanks for letting us know. If we raise the baseline, historically we
mention it in the release notes. I (as a member of the release team)
wasn't aware we might have done so until now.

I purposely filed a bug for this because a number of Debian
maintainers seem to be under the impression that Geode LX hasn't been
supported for a couple of Debian releases. Pointing out that the -686
kernel is build for exactly that doesn't seem to convince them. I
therefore feel that this needs to be made more explicit in the release
notes.

$ grep MGEODE_LX /boot/config-6.1.0-6-686
CONFIG_MGEODE_LX=y

Post by Paul Gevers

Post by Martin-Éric Racine
Additionally, other packages apparently disregard build-essentials' GCC defaults and instead configure flags to match the CPU features of the build host, which produces binaries that segfault on a Geode.

Those are in principle all RC bugs and don't only impact Geode. Please
file bugs when you find them.

I regularly do. See above for the usual response I get.

Martin-Éric

[James Addison]

Package: release-notes
Followup-For: Bug #1033065
X-Debbugs-Cc: ***@debian.org, ***@debian.org

Dear Maintainer and Éric-Martin (with Bill on carbon copy),

Please find linked below a previous release note from Debian 9.0 (stretch)
that we could use to provide relevant user guidance:

https://www.debian.org/releases/stretch/i386/release-notes/ch-information.html#i386-is-now-almost-i686

(I discovered this while reading a 2019 mailing list discussion[1])

Post by Paul Wise
Broadly speaking, detecting non-baseline instruction usage isn't
possible without false positives, because the program could use runtime
instruction selection based on the current CPU to avoid currently
unavailable instructions, while the binary would still contain those
instructions for use on other CPUs.
https://wiki.debian.org/InstructionSelection
Of course you could do the scanning and then use autopkgtests or manual
tests to find out if the found programs work on the relevant CPUs.

Thank you, that makes sense.

I've run some ad-hoc script analysis[2] on a recent mirror of the bookworm i386
archive, and it appears that ~20% or so of packages are potentially affected in
that (so, in all likelihood, Debian is currently uninstallable and/or unusable
on Geode LX).

In theory I would like to run a comparative analysis against the snapshot
archives from previous points in time, but am not sure whether I'll get around
to doing that.

Post by Paul Wise
Perhaps lintian could add classification tags for the relevant CPU
instructions and then the i386 port could have extra autopkgtest nodes
that only process the packages detected by lintian.

That's not a bad idea. Are there any reasons that that might _not_ be a good
idea before filing a wishlist bug? (performance, implications of scanning
binary packages, ...)


[1] - https://lists.debian.org/debian-user/2019/04/msg01091.html

[2] - https://bugs.debian

[Paul Wise]

That's not a bad idea.  Are there any reasons that that might _not_ be a good
idea before filing a wishlist bug?  (performance, implications of scanning
binary packages, ...)

binutils isn't security supported, so using objdump in lintian probably
isn't a good idea, especially since it is run on ftp-master.debian.org.

In addition disassembling binaries is going to have an impact on the
performance of lintian, especially for larger packages.

[James Addison]

Package: release-notes
Followup-For: Bug #1033065
X-Debbugs-Cc: ***@debian.org
Control: severity -1 serious

Increasing this bug's severity to a release-critical, based on mailing list
discussion[1].

Paul: bug #1005863 has most of the relevant context for Debian, although I'd
recommend the following wiki page as a more concise, purpose-written summary:

https://www.jookia.org/wiki/Nopl

[1] - https://lists.debian.org/debian-doc/2023/03/msg00012.html

[James Addison]

Followup-For: Bug #1033065

Post by James Addison

Post by Paul Wise
Perhaps lintian could add classification tags for the relevant CPU
instructions and then the i386 port could have extra autopkgtest nodes
that only process the packages detected by lintian.

That's not a bad idea. Are there any reasons that that might _not_ be a good
idea before filing a wishlist bug? (performance, implications of scanning
binary packages, ...)

This seems logistically problematic.
Is lintian actually ran on i386 binaries anymore ?
lintian.debian.org only lists reports for amd64 packages.

Yep, some i386 binary package analysis does still take place I believe, based
on running an UDD query[1] that found results with a recent version of lintian
(since 2023-02-05) including some i386 shared libraries in the output.

The data on lintian.debian.org (detagtive) appears out-of-date, though. There
is an existing issue report[2] about that.

I am not sure it is worth the trouble, frankly. I do not see what this would
bring us.

A lintian check could help to notify maintainers about architecture baseline
compatibility issues. Running autopkgtests for those could help to (although
is not guaranteed to) confirm whether the identified packages are broken,
without rerunning tests across all packages.

[1] - https://udd.debian.org/lintian/?email1=&email2=&email3=&packages=gcc-12-cross&ignpackages=&format=html&lt_information=on&lintian_tag=static-library-has-unneeded-sections#all

[2] - https://salsa.debian.org/lintian/detagtive/-/issues/22

[Lucas Nussbaum]

Hi,

Post by James Addison

This seems logistically problematic.
Is lintian actually ran on i386 binaries anymore ?
lintian.debian.org only lists reports for amd64 packages.

Yep, some i386 binary package analysis does still take place I believe, based
on running an UDD query[1] that found results with a recent version of lintian
(since 2023-02-05) including some i386 shared libraries in the output.
The data on lintian.debian.org (detagtive) appears out-of-date, though. There
is an existing issue report[2] about that.

I confirm that the UDD implementation of lintian runs checks on all
architectures.

Lucas

[Paul Gevers]

Control: tags -1 patch pending

Hi,

Please review my proposal here:

https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/166

Paul

[Paul Gevers]

Hi,

Post by Paul Gevers
https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/166

The release notes now document that the baseline has been bumped.

Paul

[Debian Bug Tracking System]

Post by Paul Gevers
tags -1 patch pending

Bug #1033065 [release-notes] release-notes: i386 notes should specify minimum CPU requirements
Added tag(s) pending and patch.

[Martin-Éric Racine]

On Sun, May 28, 2023 at 7:15 PM Martin-Éric Racine

Post by Paul Gevers

Post by Paul Gevers
https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/166

The release notes now document that the baseline has been bumped.

Please note that I never received the previous message you quoted.
Anyhow, I commented. on Salsa.

See my other comments in the same thread on Salsa.

This should be made more explicit. Merely stating "something with
NOPL" really won't help anyone who isn't familiar with the intricacies
of x86 variants.

If the expected baseline nowadays is hardware that can run the
-686-pae kernel, it should be explicitly stated in the release notes.

I'm not sure of whether there is any way to grep /proc/cpuinfo to
check whether a host meets the minimal requirement for Bookwormor not,
let alone which package should perform that check, before allowing a
dist-upgrade, but this definitely needs to be implemented now before
Bookworm is released.

Anyhow, given the baseline bump, it appears that the last of the Geode
processors is no longer supported on i386 (even though -686 is
configured for MGEODE and stil ships with Bookworm). Oh well. It was
fun while it lasted.

Martin-Éric

[Debian Bug Tracking System]

Your message dated Mon, 29 May 2023 06:52:22 +0200
with message-id <80b0928c-6ff4-0014-66d9-***@debian.org>
and subject line Re: Bug#1033065: release-notes: i386 notes should specify minimum CPU requirements
has caused the Debian Bug report #1033065,
regarding release-notes: i386 notes should specify minimum CPU requirements
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)

[Martin-Éric Racine]

Post by Paul Gevers

Post by Paul Gevers
https://salsa.debian.org/ddp-team/release-notes/-/merge_requests/166

The release notes now document that the baseline has been bumped.

I ment this message to close the bug, but I failed to adjust the mail To
header.

As commented both on Salsa and in response to your previous e-mail, I
still don't think that the release notes adequately describe the
updated baseline for i386.

As previously stated, the Geode LX (but not older Geodes) does fulfill
the baseline requirement for i686. NOPL, PAE and others were marked by
Intel as optional features. If what the new Debian baseline really
means is something that can run the -686-pae kernel, the release notes
should explicitly say so.

On a related issue, something in dpkg or apt should check the CPU
features and refuse to perform an upgrade/dist-upgrade on an host
whose CPU doesn't meet the new baseline requirements.

Martin-Éric

[Paul Gevers]

Hi Martin-Éric,

Post by Martin-Éric Racine
As previously stated, the Geode LX (but not older Geodes) does fulfill
the baseline requirement for i686. NOPL, PAE and others were marked by
Intel as optional features. If what the new Debian baseline really
means is something that can run the -686-pae kernel, the release notes
should explicitly say so.

I believe you are more aware of the problems with the Geode than I, as
it were your bugs that lead to the update in the release notes. If I
understand correctly the kernel still runs fine, but a bunch of
important binaries use NOPL and hence fail on the Geode. We could add a
check for NOPL, like I mentioned in the MR.

Post by Martin-Éric Racine
On a related issue, something in dpkg or apt should check the CPU
features and refuse to perform an upgrade/dist-upgrade on an host
whose CPU doesn't meet the new baseline requirements.

Please file a bug with the respective packages. Mentioning it in a bug
against the release notes isn't going to achieve that feature.

Paul

[Paul Gevers]

Hi,

Post by Paul Gevers

Post by Martin-Éric Racine
On a related issue, something in dpkg or apt should check the CPU
features and refuse to perform an upgrade/dist-upgrade on an host
whose CPU doesn't meet the new baseline requirements.

Please file a bug with the respective packages. Mentioning it in a bug
against the release notes isn't going to achieve that feature.

Oh, totally. I'm just not sure of which one typically handles this.
Last time the baseline was raised from 586 to 686 (before that from
486 to 586), something in the upgrade process performed the CPU check
and loudly aborted the proceedings. Something similar was done when
the baseline was raised on SPARC ages ago; upgrade aborted early if
the host hardware didn't meet the new baseline.

I wasn't aware of that. I *think* dpkg just installs what apt feeds it,
so from those two, I'd guess apt. However, it's too late for bookworm to
add that now. I thought you meant this mostly as a future enhancement.

Paul

[Martin-Éric Racine]

Post by Paul Gevers

Post by Paul Gevers

Post by Martin-Éric Racine
On a related issue, something in dpkg or apt should check the CPU
features and refuse to perform an upgrade/dist-upgrade on an host
whose CPU doesn't meet the new baseline requirements.

Please file a bug with the respective packages. Mentioning it in a bug
against the release notes isn't going to achieve that feature.

Oh, totally. I'm just not sure of which one typically handles this.
Last time the baseline was raised from 586 to 686 (before that from
486 to 586), something in the upgrade process performed the CPU check
and loudly aborted the proceedings. Something similar was done when
the baseline was raised on SPARC ages ago; upgrade aborted early if
the host hardware didn't meet the new baseline.

I wasn't aware of that. I *think* dpkg just installs what apt feeds it,
so from those two, I'd guess apt. However, it's too late for bookworm to
add that now. I thought you meant this mostly as a future enhancement.

Then the baseline will have to remain at 686 WITHOUT NOPL for Bookworm.

Sorry, but a baseline bump really has to be better planned and better
documented than this. Merely deciding to update the release notes at
the last minute because everyone is too lazy to fix the 20% of
packages or so that --configure NOPL or some more recent flag really
won't do.

Martin-Éric