Bug#931130: Document that local configuration for dummy and bonding modules are getting overwritten by systemd

[Baptiste BEAUPLAT]

Package: release-notes

Dear maintainers,

Systemd, in buster, will ship the file /lib/modprobe.d/systemd.conf,
which contains the following:

options bonding max_bonds=0

# Do the same for dummy0.

options dummy numdummies=0

This breaks any configuration that an administrator could have added to
/etc/modprobe.d regarding the dummy and bonding modules.

For instance, a file in /etc/modprobe.d/dummy.conf containing:

options dummy numdummies=1

Will result in the following being executed by modprobe:

insmod /lib/modules/4.19.0-5-amd64/kernel/drivers/net/dummy.ko
numdummies=1 numdummies=0

And the original configuration will be overridden.

The only way to force modprobe to use local configuration is to rename
the /etc/modprobe.d/dummy.conf file to /etc/modprobe.d/systemd.conf.

I thinks this should be documented in the release notes as admins would
need to be aware of that.

Best,

[Justin B Rye]

Checking on a Buster box I see it also defines
"options bonding max_bonds=0"

Oh, yes, you already said that! But anyway: what are the ill effects?

[Justin B Rye]

Post by Baptiste BEAUPLAT
Systemd, in buster, will ship the file /lib/modprobe.d/systemd.conf,
options bonding max_bonds=0
# Do the same for dummy0.
options dummy numdummies=0
This breaks any configuration that an administrator could have added to
/etc/modprobe.d regarding the dummy and bonding modules.

We need more information about why an administrator might have done
this, since otherwise for a start it's impossible to guess what would
go wrong as a result. VMs with sabotaged networking, or what? Is
there some other bugreport where we could read about these symptoms?

Post by Baptiste BEAUPLAT
options dummy numdummies=1
insmod /lib/modules/4.19.0-5-amd64/kernel/drivers/net/dummy.ko
numdummies=1 numdummies=0
And the original configuration will be overridden.
The only way to force modprobe to use local configuration is to rename
the /etc/modprobe.d/dummy.conf file to /etc/modprobe.d/systemd.conf.

This overrides the /lib/modprobe.d/systemd.conf entirely, doesn't it?
In which case you'd lose any other things in that file that systemd
might be depending on. Checking on a Buster box I see it also defines
"options bonding max_bonds=0" - if I want to avoid overriding that,
would it be better to use a name like /etc/modprobe.d/zz-local.conf?

Post by Baptiste BEAUPLAT
I thinks this should be documented in the release notes as admins would
need to be aware of that.

And/or quite possibly about max_bonds=0? I'm afraid I know even less
about what symptoms that might have.

[Baptiste BEAUPLAT]

Hi all,

Post by Justin B Rye

Post by Baptiste BEAUPLAT
Systemd, in buster, will ship the file /lib/modprobe.d/systemd.conf,
options bonding max_bonds=0
# Do the same for dummy0.
options dummy numdummies=0
This breaks any configuration that an administrator could have added to
/etc/modprobe.d regarding the dummy and bonding modules.

We need more information about why an administrator might have done
this, since otherwise for a start it's impossible to guess what would
go wrong as a result. VMs with sabotaged networking, or what? Is
there some other bugreport where we could read about these symptoms?

The dummy module can be used to create virtual interfaces, that can then
be configured for a particular use.

I've seen it a couple of times, usually for router or vpn box. It would
allow to bind some services especially for that virtual interface.

I stumble across this bug just as of today, while testing the upgrade to
buster. I don't think there is yet other bug report on it.

If you think that release-notes is not the place and that I should
report it to another package, I don't mind at all reassigning it.

Post by Justin B Rye

Post by Baptiste BEAUPLAT
options dummy numdummies=1
insmod /lib/modules/4.19.0-5-amd64/kernel/drivers/net/dummy.ko
numdummies=1 numdummies=0
And the original configuration will be overridden.
The only way to force modprobe to use local configuration is to rename
the /etc/modprobe.d/dummy.conf file to /etc/modprobe.d/systemd.conf.

This overrides the /lib/modprobe.d/systemd.conf entirely, doesn't it?
In which case you'd lose any other things in that file that systemd
might be depending on. Checking on a Buster box I see it also defines
"options bonding max_bonds=0" - if I want to avoid overriding that,
would it be better to use a name like /etc/modprobe.d/zz-local.conf?

The current problem is the actual processing order of modprobe.

It should be /lib/modprobe.d then /etc/modprobe.d.

Currently, it merges both directories and then do the sorting.
The 'ill' effect is that it processes /lib/modprobe.d/systemd.conf
_after_ /etc/aa-systemd.conf, overwritting its content.

I now realize that I should have reported the issue directly against
modprobe. I don't think it will be fixed in time for buster release, so
it might be still useful to have the info on the release notes.

Post by Justin B Rye

Post by Baptiste BEAUPLAT
I thinks this should be documented in the release notes as admins would
need to be aware of that.

And/or quite possibly about max_bonds=0? I'm afraid I know even less
about what symptoms that might have.

I'll open a new bug against modprobe with a better explaining of the
problem. I'll wait for maintainers' reply and see then.

Thanks for your quick replies,

Best,

[a***@gmail.com]

Control: tags -1 moreinfo

Post by Baptiste BEAUPLAT
Package: release-notes
Dear maintainers,
Systemd, in buster, will ship the file /lib/modprobe.d/systemd.conf,
options bonding max_bonds=0
# Do the same for dummy0.
options dummy numdummies=0
This breaks any configuration that an administrator could have added to
/etc/modprobe.d regarding the dummy and bonding modules.

Would you have background information on why this is done?
What would be the impact on local configuration (networking unusable,
degraded, etc.)?

Post by Baptiste BEAUPLAT
options dummy numdummies=1
insmod /lib/modules/4.19.0-5-amd64/kernel/drivers/net/dummy.ko
numdummies=1 numdummies=0
And the original configuration will be overridden.
The only way to force modprobe to use local configuration is to rename
the /etc/modprobe.d/dummy.conf file to /etc/modprobe.d/systemd.conf.

Is this documented anywhere else?

Post by Baptiste BEAUPLAT
I thinks this should be documented in the release notes as admins would
need to be aware of that.

Thanks,
Andrei - not a Release Notes Maintainer, just anticipating what
information might be necessary to write a useful entry.

[Michael Biebl]

Post by a***@gmail.com

Post by Baptiste BEAUPLAT
The only way to force modprobe to use local configuration is to rename
the /etc/modprobe.d/dummy.conf file to /etc/modprobe.d/systemd.conf.

Is this documented anywhere else?

According to the docs in man 5 modules-load.d it should be sufficient to
name the local config so that is sorted after systemd.conf.

[Baptiste BEAUPLAT]

Post by Michael Biebl

Post by a***@gmail.com

Post by Baptiste BEAUPLAT
The only way to force modprobe to use local configuration is to rename
the /etc/modprobe.d/dummy.conf file to /etc/modprobe.d/systemd.conf.

Is this documented anywhere else?

According to the docs in man 5 modules-load.d it should be sufficient to
name the local config so that is sorted after systemd.conf.

I've opened the bug #931137 [1] directly against kmod.

Following the prompt reply from the maintainer, it would appear that
it's the only way to go.

This is what I'm going to apply to fix the actual issue on my systems.

I'll let you decided if this new override from systemd to dummy and
bonding modules should be mentioned in the release notes.

I would advocate for it since unaware sysadmins (like me) could easly
break their network configuration because of that.

Best,

[1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931137

[Justin B Rye]

Post by Michael Biebl

Post by a***@gmail.com

Post by Baptiste BEAUPLAT
The only way to force modprobe to use local configuration is to rename
the /etc/modprobe.d/dummy.conf file to /etc/modprobe.d/systemd.conf.

Is this documented anywhere else?

According to the docs in man 5 modules-load.d it should be sufficient to
name the local config so that is sorted after systemd.conf.

Okay, so here's a first rough idea of what the release-notes might
say. It would make sense to put the new section after the existing
one on deprecating old-style interface names. Something like:


_Module configuration for bonding and dummy interfaces_

Administrators who are using channel bonding and/or dummy interfaces
(for instance to configure a machine as a router), and who have set
up module parameters in a file under /etc/modprobe.d, may need to
change the name of the local configuration file used, to avoid these
parameters being ignored after the upgrade to buster.

In buster, udev ships a file /lib/modprobe.d/systemd.conf designed to
make it easier to configure such interfaces using systemd-networkd.[?]
This contains the lines

options bonding max_bonds=0
options dummy numdummies=0

A file in /lib/modprobe.d can be overridden by one with the same name
under /etc/modprobe.d, but the names are processed in alphabetical
order, so /lib/modprobe.d/systemd.conf follows and overrides (for
instance) /etc/modprobe.d/dummy.conf. Make sure that any local
configuration file has a name that sorts after "systemd.conf", such
as "/etc/modprobe.d/zz-local.conf".

[a***@gmail.com]

Post by Justin B Rye
Okay, so here's a first rough idea of what the release-notes might
say. It would make sense to put the new section after the existing
_Module configuration for bonding and dummy interfaces_
Administrators who are using channel bonding and/or dummy interfaces
(for instance to configure a machine as a router), and who have set
up module parameters in a file under /etc/modprobe.d, may need to
change the name of the local configuration file used, to avoid these
parameters being ignored after the upgrade to buster.
In buster, udev ships a file /lib/modprobe.d/systemd.conf designed to
make it easier to configure such interfaces using systemd-networkd.[?]
This contains the lines
options bonding max_bonds=0
options dummy numdummies=0
A file in /lib/modprobe.d can be overridden by one with the same name
under /etc/modprobe.d, but the names are processed in alphabetical
order, so /lib/modprobe.d/systemd.conf follows and overrides (for
instance) /etc/modprobe.d/dummy.conf. Make sure that any local
configuration file has a name that sorts after "systemd.conf", such
as "/etc/modprobe.d/zz-local.conf".

LGTM FWIW :)

Kind regards,
Andrei

[Baptiste BEAUPLAT]

Post by a***@gmail.com

Post by Justin B Rye
Okay, so here's a first rough idea of what the release-notes might
say. It would make sense to put the new section after the existing
_Module configuration for bonding and dummy interfaces_
Administrators who are using channel bonding and/or dummy interfaces
(for instance to configure a machine as a router), and who have set
up module parameters in a file under /etc/modprobe.d, may need to
change the name of the local configuration file used, to avoid these
parameters being ignored after the upgrade to buster.
In buster, udev ships a file /lib/modprobe.d/systemd.conf designed to
make it easier to configure such interfaces using systemd-networkd.[?]
This contains the lines
options bonding max_bonds=0
options dummy numdummies=0
A file in /lib/modprobe.d can be overridden by one with the same name
under /etc/modprobe.d, but the names are processed in alphabetical
order, so /lib/modprobe.d/systemd.conf follows and overrides (for
instance) /etc/modprobe.d/dummy.conf. Make sure that any local
configuration file has a name that sorts after "systemd.conf", such
as "/etc/modprobe.d/zz-local.conf".

LGTM FWIW :)
Kind regards,
Andrei

Looks very nice indeed. The problem is described precisely and the
solution is clear.

Thanks a lot Justin.

Best,

[Justin B Rye]

Post by Justin B Rye
Okay, so here's a first rough idea of what the release-notes might
say.

Properly formatted patch attached.

[Michael Biebl]

Post by a***@gmail.com
Control: tags -1 moreinfo

Post by Baptiste BEAUPLAT
Package: release-notes
Dear maintainers,
Systemd, in buster, will ship the file /lib/modprobe.d/systemd.conf,
options bonding max_bonds=0
# Do the same for dummy0.
options dummy numdummies=0
This breaks any configuration that an administrator could have added to
/etc/modprobe.d regarding the dummy and bonding modules.

Would you have background information on why this is done?
What would be the impact on local configuration (networking unusable,
degraded, etc.)?

Assuming this was targeted at pkg-systemd-maintainers, I assume xnox can
tell you more. See
https://github.com/systemd/systemd/pull/6448

[Debian Bug Tracking System]

Post by a***@gmail.com
tags -1 moreinfo

Bug #931130 [release-notes] Document that local configuration for dummy and bonding modules are getting overwritten by systemd
Added tag(s) moreinfo.

[Debian Bug Tracking System]

Your message dated Thu, 27 Jun 2019 20:50:06 +0200
with message-id <72daf5cb-f2b1-4581-dad3-***@debian.org>
and subject line Re: Bug#931130: Document that local configuration for dummy and bonding modules are getting overwritten by systemd
has caused the Debian Bug report #931130,
regarding Document that local configuration for dummy and bonding modules are getting overwritten by systemd
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)

[Justin B Rye]

Afaiu, the kernel default is to create one dummy device by default when
the module is loaded.
I assume there might be cases where users rely on that default behaviour
without having explicitly configured anything.

Do we have any idea what their use case would be? Apparently a
different one from Baptiste's, if it's true that he needed to
explicitly configure "numdummies=1"...

Atm, the release note entry reads like only admins who have a modprobe.d
config are possibly affected.

That's true. I can't immediately see any way of rephrasing it without
making it much more complicated.

[Baptiste BEAUPLAT]

Post by Justin B Rye

Afaiu, the kernel default is to create one dummy device by default when
the module is loaded.
I assume there might be cases where users rely on that default behaviour
without having explicitly configured anything.

Do we have any idea what their use case would be? Apparently a
different one from Baptiste's, if it's true that he needed to
explicitly configure "numdummies=1"...

In my case, I didn't need to specify the "numdummies=1" as it is the
default (when systemd override is not installed).

Someone that needed only one interface with bonding or dummy would have
to just load the module while someone that required more interfaces
would have to explicitly set numdummies.

[Justin B Rye]

Post by Baptiste BEAUPLAT

Post by Justin B Rye
Do we have any idea what their use case would be? Apparently a
different one from Baptiste's, if it's true that he needed to
explicitly configure "numdummies=1"...

In my case, I didn't need to specify the "numdummies=1" as it is the
default (when systemd override is not installed).
Someone that needed only one interface with bonding or dummy would have
to just load the module while someone that required more interfaces
would have to explicitly set numdummies.

Okay, back to the drawing board:

_Module configuration for bonding and dummy interfaces_

Systems using channel bonding and/or dummy interfaces (for instance to
configure a machine as a router) may encounter problems upgrading to buster
due to the file /lib/modprobe.d/systemd.conf now shipped by udev (intended
to simplify configuration via systemd-networkd). This file contains the lines

options bonding max_bonds=0
options dummy numdummies=0

Admins who were depending on different values will need to ensure they are
set in the correct way to take precedence. A file in /lib/modprobe.d can be
overridden by one with the same name under /etc/modprobe.d, but the names
are processed in alphabetical order, so /lib/modprobe.d/systemd.conf follows
and overrides (for instance) /etc/modprobe.d/dummy.conf. Make sure that any
local configuration file has a name that sorts after "systemd.conf", such
as "/etc/modprobe.d/zz-local.conf".

It sounds as if we should also be adding something to the effect that

With systemd it may be simpler to configure virtual network devices
using a .netdev file - see
"https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#[Bond] Section Options".

but (a) is that odd #fragment-id going to choke docbook? and (b) does
it need any hedging ("using an appropriately de-frobularised version
of systemd-networkd and a .netdev file")?

[Justin B Rye]

I attach a slightly rephrased version with markup.

Post by Justin B Rye
[...]
It sounds as if we should also be adding something to the effect that
With systemd it may be simpler to configure virtual network devices
using a .netdev file - see
"https://www.freedesktop.org/software/systemd/man/systemd.netdev.html#[Bond] Section Options".

I left this out.

[Baptiste BEAUPLAT]

Post by Justin B Rye

Afaiu, the kernel default is to create one dummy device by default when
the module is loaded.

.. or bond device, for that matter.

Post by Justin B Rye

I assume there might be cases where users rely on that default behaviour
without having explicitly configured anything.

Do we have any idea what their use case would be? Apparently a
different one from Baptiste's, if it's true that he needed to
explicitly configure "numdummies=1"...

Dunno why Baptiste had to do that.

I ended up creating the file to explicitly configure numdummies=1 when I
saw that my interfaces disappeared after the upgrade to buster.

This "customization" is only necessary when multiple interfaces are
required to show on boot.

I just verified that by removing /lib/modprobe.d/systemd.conf, I got the
6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN
group default qlen 1000
link/ether 82:e5:b3:41:67:17 brd ff:ff:ff:ff:ff:ff
7: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 5e:eb:30:64:69:ef brd ff:ff:ff:ff:ff:ff
With the modprobe.d config file installed, there obviously was neither a
dummy0 nor bond0 device.
Fwiw, I know too little about those dummy and bonding devices and how
they are usually used.
E.g. network-manager and networkd seem to create them on the fly as
needed. So I'm not sure, which users will actually be affected by that
change.

All users not using networkd and network-manager. Usually servers, just
with ifup/ifdown.

Would be good to know how Baptiste is setting up those devices. If he is
doing it manually via some scripting of low level tool or uses a higher
level network management tool
Fwiw, with "ip link add dummy0 type dummy" (or "ip link add bond0 type
bond") I was e.g. able to create such a device manually as well.
I wonder whether such an approach isn't better then statically setting
the number of devices via a kernel module option.

My original setup was:

echo "dummy" > /etc/modules

cat << EOF >> /etc/network/interfaces
auto dummy0
iface dummy0 inet static
address 192.168.64.1
netmask 24
EOF

The interface would pop up configured at boot time (by ifup). Then, I
had services binding on 192.168.64.1.

I hope this info helps.

Best,

[a***@gmail.com]

Post by Baptiste BEAUPLAT

Would be good to know how Baptiste is setting up those devices. If he is
doing it manually via some scripting of low level tool or uses a higher
level network management tool
Fwiw, with "ip link add dummy0 type dummy" (or "ip link add bond0 type
bond") I was e.g. able to create such a device manually as well.
I wonder whether such an approach isn't better then statically setting
the number of devices via a kernel module option.

echo "dummy" > /etc/modules
cat << EOF >> /etc/network/interfaces
auto dummy0
iface dummy0 inet static
address 192.168.64.1
netmask 24
EOF
The interface would pop up configured at boot time (by ifup). Then, I
had services binding on 192.168.64.1.

A (quick) look at interfaces(5) doesn't reveal any specific method to
create dummy or bond interfaces.

In order to keep things within one configuration file it would make
sense to use a pre-up with ip commands as suggested above.

Would this work *without* overriding the systemd.conf file?

Kind regards,
Andrei

[Justin B Rye]

+ configure a machine as a router, may encounter problems upgrading to buster.
+ New versions of <systemitem role="package">udev</systemitem> install a

That modprobe config file is shipped by the systemd package, not by udev.

Ah, thanks.

A file in
+ <filename>/lib/modprobe.d</filename> can be overridden by one with the
+ same name under <filename>/etc/modprobe.d</filename>, but the names are
+ processed in alphabetical order, so

That sentence is a bit confusing: files with the same name are
overridden. period. there is no but.

If there wasn't a "but", people wouldn't have been caught out! It's
true that if you create a file in /etc then you don't need to worry
about the one it masks in /lib, but you do still need to worry about
others that might turn up in /lib with names that sort later.

Are you trying to say that files can be overridden as a whole but
usually its better to only override specific settings in a local
configuration file which doesn not have the same name?

No, I *could* have added a paragraph explaining the drawbacks of using
the filename /etc/modprobe.d/systemd.conf, but it seemed simpler just
to not mention the suboptimal solution. After all, who knows what
might turn up in bullseye's /etc/modprobe.d/udev.conf?

(which is indeed a useful recommendation because future updates of
systemd.conf might be missed if you name the file
/etc/modprobe.d/systemd.conf).

The simplest way of looking at this bug is that the old approach
(as recommended in legacy HOWTOs) of using filenames like
"/etc/modprobe.d/dummy.conf" needs to be updated to using something
like "/etc/modprobe.d/zz-local.conf". My explanation covers things in
what I hoped was just enough detail to make it clear why that change
is needed.


I've stared at it quite a bit and can't find a clearer way of saying
it. Here's a patch with just the s/udev/systemd/ change; I'll go for
a walk and then try looking at it again.

[Justin B Rye]

Back from my walk in the lovely warm pouring rain...

[...] A file in
+ <filename>/lib/modprobe.d</filename> can be overridden by one with the
+ same name under <filename>/etc/modprobe.d</filename>, but the names are
+ processed in alphabetical order, so

That sentence is a bit confusing: files with the same name are
overridden. period. there is no but.

Okay, coming back to it, it can't be helping that this is a useless
use of passive voice. Swapping it around:

[...] A file in
<filename>/etc/modprobe.d</filename> will override one with the same
name under <filename>/lib/modprobe.d</filename>, but the names are
processed in alphabetical order, so [...]

This way we start off with the focus in the right place: on the
existing dummy.conf in /etc. It's not a problem if an upstream
/lib/modprobe.d/dummy.conf turns up, *but*...

[Andrei POPESCU]

- <quote><filename>systemd.conf</filename></quote>, such as
- <quote><filename>/etc/modprobe.d/zz-local.conf</filename></quote>.
+ Admins who were depending on different values will need to ensure they are
+ set in the correct way to take precedence. A file in
+ <filename>/etc/modprobe.d</filename> will override one with the same name
+ under <filename>/lib/modprobe.d</filename>, but the names are

Somehow /etc and /lib ended up reversed, fixed in 32d6bf4c.

Kind regards,
Andrei

[Petr Tichý]

Just stumbled upon this on upgrade to Buster on a IPsec gateway, where the dummy interface is as tunnel source IP.

My solution was to let systemd-networkd manage the device by adding /etc/systemd/network/00-dummy.netdev with

[NetDev]
Name=dummy0
Kind=dummy

and then the standard /etc/network/interfaces stanza for dummy0, followed by `systemctl restart systemd-networkd` and `ifup dummy0`. Everything is back without overriding /lib/modprobe.d/systemd.conf

I thing the note should suggest the systemd-networkd solution first, as Debian adopted systemd.

Thanks Petr