security support in buster and the release notes

[Paul Gevers]

Hi all,

This is a kind ping for a reply. There was a note on IRC for a reply,
but I fear it was forgotten or ENOTIME.

Dear release team,
I am reaching out to you to align on the security support that users can
expect during the lifetime of buster and how this is covered in the
release notes.
The release notes currently contain a section on "Limitations in
* web browsers and their rendering engines
* ecosystem around libv8 and Node.js
Do these subjects still cover your current view of the support for
buster. Especially about the second item I am not sure if it still
applies (although I expect so). Are there other concerns or warnings and
should they already be mentioned in the release notes?
On top of the above questions, of course it would be great if you would
check the wording of the current text [1].
On behalf of the release team,
Paul
[1]
https://salsa.debian.org/ddp-team/release-notes/blob/master/en/issues.dbk#L215
(Line number where we are currently)

Paul

[Paul Gevers]

Hi,
Done.

[...]

IIUC, there're two concerns for Go packages.

[...]

2. binNMU without full source upload for security-master.
It's still not possible, and I don't know there's any effort to
change the dak.
But I want to know how security team handles other static linked
languages, like rust, haskell, ocaml, etc.

With respect to binNMU'ing, static linking is not a problem, only
arch:all is. Most haskell (4 vs 1048) and ocaml (21 vs 233) aren't
arch:all. haskell and ocaml have a framework in place to at least know
the status in unstable/testing. See e.g. the "permanent trackers" at
https://release.debian.org/transitions/ I don't know yet what this means
for security support. Neither do I know what it means for rust.

It's not the issue for only Go packages.

But most haskell and ocaml packages can be binNMU'd.

The easiest probably is to binNMU in stable-pu.

I don't understand what you mean by this last sentence. You mean to not
do a binNMU but a full NMU for all the arch:all packages? I think the
problem of the security team is that they don't want to commit to that.

[bug 928227]
On 05-05-2019 18:00, Shengjing Zhu wrote:> Hi,

[...]

Please unblock package golang-golang-x-net-dev
Upstream has provided patches addressing security issues
CVE-2018-17846 / CVE-2018-17847 / CVE-2018-17848
(Debian bug #911795).

How will unblocking this fix these issues? golang-golang-x-net-dev is

embedded

in a number of packages in buster. If they are not updated, the

unblock will

not fix anything. How will this be handled?

All the reverse depends need binNMU.
Since the Go packages are using(abusing) Built-Using tag, probably the
release team will binNMU all outdated Built-Using packages at this
period(before release)?

I think the rebuild (or at least a big chunk of it) has already been
done. And, as noted above, that we can't binNMU arch:all yet. Will you
source upload those and add the list to bug 928227 and tell us which
additional packages need to be scheduled for a binNMU?

Just wondering, does anybody already have tooling/scripts/urls do check
the current status? If not, I'll cook up something to assess the
situation for myself. I'll update bug 928227 when I have some data.

Maybe we can keep the conversation at

Done.

Paul

[Moritz Muehlenhoff]

Post by Paul Gevers

2. binNMU without full source upload for security-master.
It's still not possible, and I don't know there's any effort to
change the dak.
But I want to know how security team handles other static linked
languages, like rust, haskell, ocaml, etc.

With respect to binNMU'ing, static linking is not a problem, only
arch:all is. Most haskell (4 vs 1048) and ocaml (21 vs 233) aren't
arch:all. haskell and ocaml have a framework in place to at least know
the status in unstable/testing. See e.g. the "permanent trackers" at
https://release.debian.org/transitions/ I don't know yet what this means
for security support. Neither do I know what it means for rust.

There's the additional issue that ftp-master and security-master don't
share tarballs; binNMUs are only possible for packages which are on
security-master, so we'd need to do manual source uploads for every
affected go package.

Post by Paul Gevers
But most haskell and ocaml packages can be binNMU'd.

This issue has been around for Haskell and Ocaml for a long time, but it
hasn't been an issue in practice. Ideally we can use the same mechanism
found for Go, also for Ocaml or Haskell if the need should ever arise.

For Go it's different, it has already bitten us for the limited subset
of Go applications in Stretch (and #922170 which is needed to kludge
around it), but with the increased footprint of Go stuff in buster we
need something better.

The same issue will hit us with Rust as well, but for buster we have
only two relevant packages (librsvg and firefox which use vendored
libs, which seems manageable). For bullseye we also need a proper, generic
solution for Rust.

Post by Paul Gevers
I think the
problem of the security team is that they don't want to commit to that.

Yeah, that won't work, this could amount to hundreds of packages.

Cheers,
Moritz

[Shengjing Zhu]

Hi the security team,

On Thu, May 9, 2019 at 1:53 AM Moritz Muehlenhoff <***@inutil.org> wrote:
[...]

Post by Moritz Muehlenhoff
There's the additional issue that ftp-master and security-master don't
share tarballs; binNMUs are only possible for packages which are on
security-master, so we'd need to do manual source uploads for every
affected go package.

I probably lack of some historical background, have you ever think of
merging ftp-master and security-master?

[Salvatore Bonaccorso]

Hi,

Post by Shengjing Zhu
Hi the security team,
[...]

Post by Moritz Muehlenhoff
There's the additional issue that ftp-master and security-master don't
share tarballs; binNMUs are only possible for packages which are on
security-master, so we'd need to do manual source uploads for every
affected go package.

I probably lack of some historical background, have you ever think of
merging ftp-master and security-master?

The security team does not manage dak on security-master, this is
actually ftp-masters domain. The separation has as disadantages and
advantages, one which comes to my mind idepenently on the aspect of
the one beeing security-master is to have a fallback updateing channel
in case one or the other cannot be used.

But okay that's not the point.

There is #823820 for one Built-Using aspect, but the idea might be
possible to generalize to have on every time orig tarballs from
main archive available on security-master as well.

Regards,
Salvatore

[Paul Wise]

Post by Paul Gevers
With respect to binNMU'ing, static linking is not a problem, only
arch:all is. Most haskell (4 vs 1048) and ocaml (21 vs 233) aren't
arch:all. haskell and ocaml have a framework in place to at least know
the status in unstable/testing. See e.g. the "permanent trackers" at
https://release.debian.org/transitions/ I don't know yet what this means
for security support. Neither do I know what it means for rust.

I think there is something that the release/security teams might be
missing here:

The Go/Rust arch:all packages (the golang-*-dev ones) contain only
source code (ideally things would support build-deps on foo:src, the
current Go/Rust binary packages are a workaround for that being
missing), so they do not need to be changed after a security upload.
Only the packages containing statically linked Go/Rust code need to be
binNMUed and those ones should be arch:any since they contain
architecture-specific binaries. In addition the arch:all packages do
not have Built-Using, only the statically linked ones do.

So the workflow seems to be quite manageable, modulo the
security-master binNMU issue: fix the security issue where it
originated, then binNMU anything that has Built-Using on any version
less than the fixed version.

[Paul Gevers]

Hi all,

[@ftp-master: we're discussing the biggest blocker for picking a buster
release date]

TL;DR; With this mail I'd like to ask technical background and/or
reasoning of why the security archive can't do binNMUs for packages that
weren't sourcefully uploaded there and to search for concrete potential
solutions (I assume there are more) to solve the technical problem of
binNMU-ing in the security archive.

2. binNMU without full source upload for security-master.
It's still not possible, and I don't know there's any effort to
change the dak.

I am all to new into this business, so ignore my ignorance and please
educate me on any mistake. Can somebody please try to explain what the
exact problem is or point me to documentation or earlier discussion that
do that? I'll try to write down potential descriptions as I see them
(from quite a bit of guessing).

IIUC from this thread so far, ftp-master and security-master are two
separate archives [1]. The security-master archive only contains sources
and binaries that were uploaded to security. A potential solution to the
problem at hand seems to be to sync all sources from ftp-master into
security-master, but I guess that is undesirable because of the massive
size increase of the security archive in that case. If this is going to
be the solution, is this still dak domain?

Another solution already raised by Shengjing is to merge the archives. I
*guess* that is undesirable due to the fact that the security archive
often has embargoed sources and binaries. Am I right there?

Another direction, as I am wondering: when packages are build for the
security archive, the binary packages from the regular archive are
available to the buildds, right? So is the problem really that the
security archive doesn't have the sources, or could/should wanna-build
(or the code really responsible for creating the proper binNMU input) be
enhanced to support this use case with the archives as they are?

I have been thinking that code could be made to do the required stuff
out-of-band and basically do sourcefull uploads automatically when
required for a set of packages. It feels like a great hack, but I guess
such a tool could cover the time until the archive tooling can properly
support it.

[Here your solution direction]

Paul

[1] I wonder how that matches with the drawing on
https://wiki.debian.org/DebianDak as that suggests that the security
archive is just another pool like unstable and testing.

[Ansgar]

Post by Paul Gevers

2. binNMU without full source upload for security-master.
It's still not possible, and I don't know there's any effort to
change the dak.

I am all to new into this business, so ignore my ignorance and please
educate me on any mistake. Can somebody please try to explain what the
exact problem is or point me to documentation or earlier discussion that
do that? I'll try to write down potential descriptions as I see them
(from quite a bit of guessing).
IIUC from this thread so far, ftp-master and security-master are two
separate archives [1]. The security-master archive only contains sources
and binaries that were uploaded to security. A potential solution to the
problem at hand seems to be to sync all sources from ftp-master into
security-master, but I guess that is undesirable because of the massive
size increase of the security archive in that case. If this is going to
be the solution, is this still dak domain?

I though about importing the full source to security-master already for
a different reason: `Built-Using` leads to a similar problem as binNMUs
in that uploads require source that is not already present in the
archive.

It is not necessary to push all sources to the public mirrors.

Post by Paul Gevers
Another solution already raised by Shengjing is to merge the archives. I
*guess* that is undesirable due to the fact that the security archive
often has embargoed sources and binaries. Am I right there?

That doesn't work as dak doesn't try to keep secrets. There are various
ways information would be leaked about embargoed issues (mails,
database, web interface (rmadison), ...).

I personally also don't find it too bad to have a fallback: if one of
the hosts is broken at the same time we have to release a critical
update, we can still do so by publishing via the "wrong" archive.

Ansgar

[Paul Gevers]

Hi Ansgar,

Post by Ansgar
I though about importing the full source to security-master already for
a different reason: `Built-Using` leads to a similar problem as binNMUs
in that uploads require source that is not already present in the
archive.
It is not necessary to push all sources to the public mirrors.

Does this mean you think it is feasible to do/fix this in the near future?

Post by Ansgar

Post by Paul Gevers
Another solution already raised by Shengjing is to merge the archives. I
*guess* that is undesirable due to the fact that the security archive
often has embargoed sources and binaries. Am I right there?

That doesn't work as dak doesn't try to keep secrets. There are various
ways information would be leaked about embargoed issues (mails,
database, web interface (rmadison), ...).
I personally also don't find it too bad to have a fallback: if one of
the hosts is broken at the same time we have to release a critical
update, we can still do so by publishing via the "wrong" archive.

Regarding my other direction with wanna-build, I learned yesterday via
another bug (#894441 binNMUs should be replaced by easy no-change
uploads) that wanna-build is not in the place to fix this because
uploads need to be signed.

Paul

[Ansgar]

Post by Paul Gevers

Post by Ansgar
I though about importing the full source to security-master already for
a different reason: `Built-Using` leads to a similar problem as binNMUs
in that uploads require source that is not already present in the
archive.
It is not necessary to push all sources to the public mirrors.

Does this mean you think it is feasible to do/fix this in the near future?

Importing sources once is probably doable; writing something to continue
updating them (at point release time and similar) takes more time which
I currently can't commit to.

There is also the question of storage: the full buster source is
something like 60 GB+, but security-master only has 63 GB free storage
right now (need to check with DSA by how much that could increase).
Though we might need more storage over the lifetime of Buster anyway as
eventually oldstable will have debug symbols too...

If storage is a problem, we would need to look at importing only
subsets, but I don't really like treating packages differently.

Ansgar

[Paul Gevers]

Control: tags -1 patch

package: release-notes

This is already an issue in Stretch (e.g. #922170), but will be much
worse in Buster, so unless someone reliably commits to work on
this ASAP the available options are to drop everything Go apart
from the toolchain packages from buster or exclude of all that mess
from security updates so that people know what they can expect.

filing a bug (in coordination with Moritz) so this doesnt get forgotten.

I have pushed a first version (attached).

Paul

[Debian Bug Tracking System]

Post by Paul Gevers
tags -1 patch

Bug #928026 [release-notes] release-notes: document the state of security support for golang packages in Buster
Added tag(s) patch.

[Justin B Rye]

+ <section id="golang-static-linking">
+ <title>Go based packages</title>
+ <para>
+ The Debian infrastructure currently doesn't properly enable rebuilding
+ packages that statically link parts of other packages on a large
+ scale. Until buster that hasn't been a problem in practice, but with the

(I'm adding a stretch-to-buster tag, since if this is still true for
buster-to-bullseye then it'll at least need a change from "hasn't
been" to "wasn't".)

+ growth of the Go ecosystems it means that Go based packages won't be

That should probably be "ecosystem", singular.

+ covered by regular security support until the infrastructure is improved
+ to cope with these kind of packages in a maintainable manner.

"These kind of" is normal in spoken English, but formal written
English prefers either "this kind of package" or "these kinds of
packages". Or maybe we just want:

to deal with them maintainably.

+ </para>
+ <para>
+ If updates are warranted, they can only come via regular point releases
+ and thus may be deployed late.

To avoid implying "too late" we might phrase this as:

If updates are warranted, they can only come via regular point releases,
which may be slow in arriving.