Bug 787314 - assorted issues with remote filesystem dependency tree
Summary: assorted issues with remote filesystem dependency tree
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: NetworkManager
Version: 19
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Dan Williams
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 812039 (view as bug list)
Depends On:
Blocks: 829449 WhyWeBootSoSlow
TreeView+ depends on / blocked
 
Reported: 2012-02-03 22:20 UTC by Bill Nottingham
Modified: 2015-02-18 11:07 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 829449 (view as bug list)
Environment:
Last Closed: 2015-02-18 11:07:42 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Bill Nottingham 2012-02-03 22:20:16 UTC
I was looking at the netfs script in initscripts. It appears to be redundant with systemd.

However, in testing what could be done out of the box with it removed, I ran into issues with systemd's network mount support.

1) Waiting for the network with NetworkManager

The documented way to handle this is to enable NetworkManager-wait-online.service, which makes it WantedBy network.target.

This is overkill - if you enable it this way so that it can be used out of the box for network mounts, you enable it for all network-using services, which is overkill.

For this usage, NM-wait-online should be required (if NM is enabled) by remote-fs-pre.target, not by network.target. This avoids the common case slowdown.

This should be enabled always as part of enabling NetworkManager.service - there is no point to conceptually enabling an automatic remote filesystem mount that operates before the network is up.

2) Dependencies between remote filesystems and remote-fs-pre.target

remote-fs-pre.target isn't actually pulled in by anything - it needs to be enabled (for example, in remote-fs.target.wants)

This seems wrong. If there is a special remote-fs-pre.target, it should automatically be pulled in if there's a remote filesystem being mounted.

One possible way:

diff --git a/src/mount.c b/src/mount.c
index 3411b73..16da385 100644
--- a/src/mount.c
+++ b/src/mount.c
@@ -320,7 +320,7 @@ static bool needs_quota(MountParameters *p) {
 }
 
 static int mount_add_fstab_links(Mount *m) {
-        const char *target, *after = NULL, *after2 = NULL;
+        const char *target, *after = NULL, *after2 = NULL, *wants = NULL;
         MountParameters *p;
         Unit *tu;
         int r;
@@ -351,6 +351,7 @@ static int mount_add_fstab_links(Mount *m) {
         if (mount_is_network(p)) {
                 target = SPECIAL_REMOTE_FS_TARGET;
                 after = SPECIAL_REMOTE_FS_PRE_TARGET;
+                wants = SPECIAL_REMOTE_FS_PRE_TARGET;
                 after2 = SPECIAL_NETWORK_TARGET;
         } else {
                 target = SPECIAL_LOCAL_FS_TARGET;
@@ -368,6 +369,10 @@ static int mount_add_fstab_links(Mount *m) {
                 if ((r = unit_add_dependency_by_name(UNIT(m), UNIT_AFTER, after2, NULL, true)) < 0)
                         return r;
 
+        if (wants)
+                if ((r = unit_add_dependency_by_name(UNIT(m), UNIT_WANTS, wants, NULL, true)) < 0)
+                        return r;
+
         if (automount) {
                 Unit *am;
 

Another way would be to add it as WantedBy remote-fs.target IFF remote-fs.target has any Requires (i.e., mounts). Not sure if that's simpler.

Version-Release number of selected component (if applicable):

systemd-37-11, but I'm assuming 39 is similar.

Comment 1 Michal Schmidt 2012-02-06 15:41:04 UTC
(In reply to comment #0)
> I was looking at the netfs script in initscripts. It appears to be redundant
> with systemd.

It is redundant as a service to run on boot, yes. But it is also called from /etc/NetworkManager/dispatcher.d/05-netfs where it perhaps serves a useful purpose.

> However, in testing what could be done out of the box with it removed, I ran
> into issues with systemd's network mount support.
> 
> 1) Waiting for the network with NetworkManager
> 
> The documented way to handle this is to enable
> NetworkManager-wait-online.service, which makes it WantedBy network.target.
> 
> This is overkill - if you enable it this way so that it can be used out of the
> box for network mounts, you enable it for all network-using services, which is
> overkill.

I don't think it's overkill. I see two main reasons why a service may want to declare "After=network.target": The service wants to:
 - bind() to an IP address without using IP_FREEBIND; or
 - connect() to a network host
In both cases the service benefits from waiting on NM-wait-online.

> For this usage, NM-wait-online should be required (if NM is enabled) by
> remote-fs-pre.target, not by network.target. This avoids the common case
> slowdown.

Can you give me a specific example of what is slowed down needlessly?

> This should be enabled always as part of enabling NetworkManager.service -
> there is no point to conceptually enabling an automatic remote filesystem
> mount that operates before the network is up.

Agreed. We really should pull in NM-wait-online automatically in at least
some of the cases where we currently depend on the administrator to enable it
manually. Having any remote filesystems to mount is definitely such a case.

> 2) Dependencies between remote filesystems and remote-fs-pre.target
> 
> remote-fs-pre.target isn't actually pulled in by anything - it needs to be
> enabled (for example, in remote-fs.target.wants)
> 
> This seems wrong. If there is a special remote-fs-pre.target, it should
> automatically be pulled in if there's a remote filesystem being mounted.
> 
> One possible way:
> [...]
> Another way would be to add it as WantedBy remote-fs.target IFF
> remote-fs.target has any Requires (i.e., mounts). Not sure if that's simpler.

It would be almost as simple. It would result in fewer edges in the graph of
requirement dependencies, so I'd find it preferable.

So these are the changes we could do:
1. for remote-fs.target: auto-add "Wants=remote-fs-pre.target" if we're going
   to mount any remote fss.
2. stop auto-adding "After=network.target" to each and every remote fs mount
   unit because this ordering is already guaranteed transitively
   via remote-fs-pre.target.
3. for remote-fs-pre.target: NM will ship a symlink /lib/systemd/system/remote-fs-pre.target.wants/NetworkManager-wait-online.service
   Note that the symlink will be static, not to be enabled/disabled.
     (NM-wait-online.service will simply refuse to start if
      NetworkManager.service is not enabled, due to a Requisite dependency.
      It won't even show up as a failed service when NM is disabled.)
   Enabling/disabling of NM-wait-online.service will affect network.target
   the same way it does today.

These changes should satisfy the requirements of remote filesystems.

Comment 2 Bill Nottingham 2012-02-07 03:19:31 UTC
(In reply to comment #1)
> (In reply to comment #0)
> > I was looking at the netfs script in initscripts. It appears to be redundant
> > with systemd.
> 
> It is redundant as a service to run on boot, yes. But it is also called from
> /etc/NetworkManager/dispatcher.d/05-netfs where it perhaps serves a useful
> purpose.

I'd rather handle this with something systemd-native than keep it around merely for the dispatcher script. 

> Can you give me a specific example of what is slowed down needlessly?

By enabling NM-wait-online as a dep of network.target, you're forcing all network.target-dependent services to synchronize on network startup, even if you only need it for your network filesystems - that's why I'd want to split it out for network filesystems to be covered by remote-fs-pre.target instead. (In any case, the changes you mention later won't run into this.)

> So these are the changes we could do:
> 1. for remote-fs.target: auto-add "Wants=remote-fs-pre.target" if we're going
>    to mount any remote fss.
> 2. stop auto-adding "After=network.target" to each and every remote fs mount
>    unit because this ordering is already guaranteed transitively
>    via remote-fs-pre.target.
> 3. for remote-fs-pre.target: NM will ship a symlink
> /lib/systemd/system/remote-fs-pre.target.wants/NetworkManager-wait-online.service
>    Note that the symlink will be static, not to be enabled/disabled.
>      (NM-wait-online.service will simply refuse to start if
>       NetworkManager.service is not enabled, due to a Requisite dependency.
>       It won't even show up as a failed service when NM is disabled.)
>    Enabling/disabling of NM-wait-online.service will affect network.target
>    the same way it does today.
> 
> These changes should satisfy the requirements of remote filesystems.

Sounds good to me.

Comment 3 Michal Schmidt 2012-02-07 08:44:17 UTC
(In reply to comment #2)
> By enabling NM-wait-online as a dep of network.target, you're forcing all
> network.target-dependent services to synchronize on network startup, even if
> you only need it for your network filesystems

My point is that services that declare their dependency on network.target really do want to synchronize on network startup. What other meaning could the dependency have?
We probably have services that are ordered after network.target for no good reason. We should fix them.
(But as you correctly point out, we don't have to deal with this subproblem right now.)

>> These changes should satisfy the requirements of remote filesystems.
> Sounds good to me.

OK, I'll make changes 1. and 2. Then we'll test 3. and ask Dan W. to add the symlink.

Comment 4 Michal Schmidt 2012-02-07 12:04:08 UTC
systemd changes done upstream:

http://cgit.freedesktop.org/systemd/systemd/commit/?id=db1355b1c181a4b1ac277064918d7c794dfb6edd

Comment 5 Bill Nottingham 2012-02-07 15:02:45 UTC
(In reply to comment #3)
> My point is that services that declare their dependency on network.target
> really do want to synchronize on network startup. What other meaning could the
> dependency have?
> We probably have services that are ordered after network.target for no good
> reason. We should fix them.

The problem is that network.target is so underspecified in terms of the LSB ('basic networking support is available. Example: a server program could listen on a socket.'), that you have to make these sorts of assumptions - by the strictest reading of the standard, network.target is always available in systemd, as localhost is always available for listening.

Comment 6 Bill Nottingham 2012-02-07 21:39:43 UTC
In any case, moving back to the point of this bug (dropping netfs).

A test build with the patch on F-16 works with the appropriate NM-wait-online symlink, so the basic startup/shutdown works.

Replacing the netfs dispatcher script is trickier - while the 'start' case can be handled easily (systemctl start remote-fs.target), the stop case can't, as stopping remote-fs.target doesn't do anything - there is no reverse mapping that tears down all remote mounts.

Comment 7 Michal Schmidt 2012-02-08 10:36:04 UTC
(In reply to comment #6)
> Replacing the netfs dispatcher script is trickier - while the 'start' case can
> be handled easily (systemctl start remote-fs.target), the stop case can't, as
> stopping remote-fs.target doesn't do anything - there is no reverse mapping
> that tears down all remote mounts.

If stopping the target should stop the mount units, the mount units would have to Require the target. But the mutual requirements between the mount units and the target would mean that all the remote mount units would become a bundle - starting/stopping one of them would start/stop all of them.

A better way to achieve what you want could be to introduce a new target unit, say remote-fs-requirement.target. The remote mount units would Require it. Stopping this target would thus stop all the remote mounts. The only disadvantage is that the usage would be asymmetric:
  # to mount all remote filesystems:
  systemctl start remote-fs.target
  # to unmount all remote filesystems:
  systemctl stop remote-fs-requirement.target

[ after some more thinking ]
Maybe we can substitute remote-fs-pre.target for remote-fs-requirement.target. In order to do this, the remote mounts will have to Require it, which they currently don't. And as the NM-wait-online proposal currently stands, the remote mount units would then indirectly pull NM-wait-online in by themselves, which is not optimal.

A third way could be a manual way: get all the Wants and Requires dependencies of remote-fs.target manually and stop them explicitly.

Comment 8 Bill Nottingham 2012-02-08 15:59:20 UTC
One question is how critical this functionality is. While I know people used the NM dispatcher script to mount network filesystems after NM connected, I'm not sure how many used the lazy umount feature for when the network dropped; it's not as if can cleanly unmount in this case.

Comment 9 Bill Nottingham 2012-03-09 18:22:00 UTC
Moving to NetworkManager for the last bit of implementation here.

We need something like:

diff --git a/NetworkManager.spec b/NetworkManager.spec
index 300a24e..7860688 100644
--- a/NetworkManager.spec
+++ b/NetworkManager.spec
@@ -303,6 +303,9 @@ echo 'NotShowIn=KDE;' >>$RPM_BUILD_ROOT%{_sysconfdir}/xdg/autostart/nm-applet.de
 # validate the autostart .desktop file
 desktop-file-validate $RPM_BUILD_ROOT%{_sysconfdir}/xdg/autostart/nm-applet.desktop
 
+mkdir -p $RPM_BUILD_ROOT%{systemd_dir}/remote-fs-pre.target.wants
+ln -s ../NetworkManager-wait-online.service $RPM_BUILD_ROOT%{systemd_dir}/remote-fs-pre.target.wants
+
 
 %clean
 %{__rm} -rf $RPM_BUILD_ROOT
@@ -416,6 +419,7 @@ gtk-update-icon-cache %{_datadir}/icons/hicolor &>/dev/null || :
 # systemd stuff
 %{systemd_dir}/NetworkManager.service
 %{systemd_dir}/NetworkManager-wait-online.service
+%{systemd_dir}/remote-fs-pre.target.wants/NetworkManager-wait-online.service
 %{_datadir}/dbus-1/system-services/org.freedesktop.NetworkManager.service
 
 %ifnarch s390 s390x

in NM, to ensure that if remote filesystems are configured, NM-wait-online is automatically pulled in.

Comment 10 Bill Nottingham 2012-04-12 16:04:45 UTC
*** Bug 812039 has been marked as a duplicate of this bug. ***

Comment 11 Lennart Poettering 2013-03-26 16:28:47 UTC
Hmm, so, this can't work this way really.

remote-fs-pre.target is a target that is "passive", i.e. that is pulled in by the provider of the feature, rather than the user. That means it shouldn't and cannot be used to pull in services from remote mount units.

I have now documented this in more detail in the docs:

http://www.freedesktop.org/software/systemd/man/systemd.special.html

There's now a new section about these "passive" targets, please have a look.

To solve the issue at hand we have now introduced a new target "remote-fs-setup.target" that can be used for pull in services from remote mounts. remote-fs-setup.target is hence for pulling things in, remote-fs-pre.target is for ordering things.

The rationale for this is explained in the commit:

http://cgit.freedesktop.org/systemd/systemd/commit/?id=e8d2f6cde0af86eece9118718ad0a8a19e1cffec

The new target is also documented in the man page linked above.

Also, nm-w-o.service is *not* something that should be linked in statically via symlinks in /usr. Static links like that are stuff for really low-level components, i.e. systemd, dracut, plymouth and suchlike, but not NM. It should be possible to disable NM-w-o.service, and currently it is not.

So, to fix the issue please alter NetworkManager-wait-online.service to do the following:

In the [Unit] section of NM-w-o.service add this:
Wants=remote-fs-pre.target
Before=remote-fs-pre.target

In the [Install] section of NM-w-o.service add this:
WantedBy=remote-fs-setup.target

And then make sure to "systemctl enable" this unit from the postinst scriptlets.

Comment 12 Lennart Poettering 2013-03-26 16:30:04 UTC
And of course, please remove these lines from the .spec file:

mkdir -p $RPM_BUILD_ROOT%{systemd_dir}/remote-fs-pre.target.wants
ln -s ../NetworkManager-wait-online.service $RPM_BUILD_ROOT%{systemd_dir}/remote-fs-pre.target.wants

Comment 13 Lennart Poettering 2013-03-26 16:40:53 UTC
By "systemctl enable" I actually meant the RPM macros for that, i.e.

%post
%systemd_post NetworkManager.service NetworkManager-wait-online.service

%preun
%systemd_preun NetworkManager.service NetworkManager-wait-online.service

%postun
%systemd_postun

As described on:

https://fedoraproject.org/wiki/Packaging:ScriptletSnippets#Systemd

Also, please add a "Conflicts: systemd < 199" to the .spec file header, to make sure remote-fs-setup.target is actually available.

Comment 14 Rahul Sundaram 2013-03-26 16:45:09 UTC
Do you really need Conflicts there?

https://fedoraproject.org/wiki/Packaging:Conflicts

Comment 15 Harald Hoyer 2013-03-26 16:47:40 UTC
(In reply to comment #14)
> Do you really need Conflicts there?
> 
> https://fedoraproject.org/wiki/Packaging:Conflicts

Well, it could be Requires: systemd >= 199

Comment 16 Jóhann B. Guðmundsson 2013-03-26 17:54:31 UTC
(In reply to comment #11)
> Hmm, so, this can't work this way really.
> 
> remote-fs-pre.target is a target that is "passive", i.e. that is pulled in
> by the provider of the feature, rather than the user. That means it
> shouldn't and cannot be used to pull in services from remote mount units.
> 
> I have now documented this in more detail in the docs:
> 
> http://www.freedesktop.org/software/systemd/man/systemd.special.html
> 
> There's now a new section about these "passive" targets, please have a look.
> 
> To solve the issue at hand we have now introduced a new target
> "remote-fs-setup.target" that can be used for pull in services from remote
> mounts. remote-fs-setup.target is hence for pulling things in,
> remote-fs-pre.target is for ordering things.
> 
> The rationale for this is explained in the commit:
> 
> http://cgit.freedesktop.org/systemd/systemd/commit/
> ?id=e8d2f6cde0af86eece9118718ad0a8a19e1cffec
> 
> The new target is also documented in the man page linked above.
> 
> Also, nm-w-o.service is *not* something that should be linked in statically
> via symlinks in /usr. Static links like that are stuff for really low-level
> components, i.e. systemd, dracut, plymouth and suchlike, but not NM. It
> should be possible to disable NM-w-o.service, and currently it is not.
> 
> So, to fix the issue please alter NetworkManager-wait-online.service to do
> the following:
> 
> In the [Unit] section of NM-w-o.service add this:
> Wants=remote-fs-pre.target
> Before=remote-fs-pre.target
> 
> In the [Install] section of NM-w-o.service add this:
> WantedBy=remote-fs-setup.target
> 
> And then make sure to "systemctl enable" this unit from the postinst
> scriptlets.


I argue you got this backwards for the NetworkManager-wait-online.service.

The hard requirements on nm-w-o be put into remote-fs-pre.target. 

General rule of thumb is never to put requirements into units that the unit it self is not explicated depend on for functionality.

The NetworkManager-wait-online.service does not depend on remote-fs-pre.target thus it should not want or require it. 

The functionality of remote-fs-pre.target however *needs* NetworkManager-wait-online.service for units that require it thus it should want or require it.

In your proposal there users that require NetworkManager-wait-online.service to be enable ( yes remote fs are not the only case the nm-w-o needs to be enabled for ) but at same time have disabled the remote-fs-pre.target wont have the nm-w-o enabled since remote-fs-pre.target has been disabled. 

What you are proposing is wrong unit implementation behaviour dammit! :)

Comment 17 Bill Nottingham 2013-03-26 18:22:04 UTC
I'd agree with Johann here - there are users that need/want NM-wait-online that expect 'simple' enablement of it to work for network.target. (i.e., to make network.target synchronous at some level). The *additional* symlink was to ensure that it was added for any case where there are remote filesystems being pulled in.

Comment 18 Harald Hoyer 2013-03-26 18:36:12 UTC
(In reply to comment #11)
> In the [Install] section of NM-w-o.service add this:
> WantedBy=remote-fs-setup.target

# systemctl show remote-fs-setup.target
...
Wants=NetworkManager-wait-online.service
WantedBy=mnt-nas-home\x2dharald.mount mnt-nas-pub.mount mnt-nas-Public.mount
Before=remote-fs.target

# systemctl show remote-fs-pre.target
...
WantedBy=NetworkManager-wait-online.service
Conflicts=shutdown.target
Before=remote-fs.target mnt-nas-home\x2dharald.mount mnt-nas-pub.mount mnt-nas-Public.mount
After=nfs-lock.service NetworkManager-wait-online.service

# systemctl show remote-fs.target
...
Requires=mnt-nas-Public.automount mnt-nas-pub.automount mnt-nas-home\x2dharald.automount
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=ceph.service systemd-journal-flush.service systemd-user-sessions.service multi-user.target
After=remote-fs-pre.target remote-fs-setup.target mnt-nas-home\x2dharald.automount mnt-nas-pub.automount mnt-nas-Public.automount

What's the problem?

Comment 19 Harald Hoyer 2013-03-26 18:39:13 UTC
# systemctl show mnt-nas-pub.mount|head -10
...
Requires=-.mount
Wants=remote-fs-setup.target
Conflicts=umount.target
Before=umount.target
After=mnt-nas-pub.automount remote-fs-pre.target systemd-journald.socket -.mount
TriggeredBy=mnt-nas-pub.automount

Comment 20 Jóhann B. Guðmundsson 2013-03-26 19:01:08 UTC
(In reply to comment #18)
> (In reply to comment #11)
> > In the [Install] section of NM-w-o.service add this:
> > WantedBy=remote-fs-setup.target
> 
> # systemctl show remote-fs-setup.target
> ...
> Wants=NetworkManager-wait-online.service
> WantedBy=mnt-nas-home\x2dharald.mount mnt-nas-pub.mount mnt-nas-Public.mount
> Before=remote-fs.target
> 
> # systemctl show remote-fs-pre.target
> ...
> WantedBy=NetworkManager-wait-online.service
> Conflicts=shutdown.target
> Before=remote-fs.target mnt-nas-home\x2dharald.mount mnt-nas-pub.mount
> mnt-nas-Public.mount
> After=nfs-lock.service NetworkManager-wait-online.service
> 
> # systemctl show remote-fs.target
> ...
> Requires=mnt-nas-Public.automount mnt-nas-pub.automount
> mnt-nas-home\x2dharald.automount
> WantedBy=multi-user.target
> Conflicts=shutdown.target
> Before=ceph.service systemd-journal-flush.service
> systemd-user-sessions.service multi-user.target
> After=remote-fs-pre.target remote-fs-setup.target
> mnt-nas-home\x2dharald.automount mnt-nas-pub.automount
> mnt-nas-Public.automount
> 
> What's the problem?

The problem are the changes Lennart is proposing should be made to the NetworkManager-wait-online.service . 

Along with the fact that he has enabled it by default in preset with http://pkgs.fedoraproject.org/cgit/systemd.git/commit/?h=f19&id=ea6e552a3643d488536d86d8d508e5d1ecbfb5cc 

That can add up to 30s delayed boot time for *every* fedora users not just administrators that a) would enable it if necessary or b) or by enabling server related service that would potentially pull it in as an want or hard 
requirement before starting. 

The proper implementation from my pov is that by default the NetworkManager-wait-online.service should be left as it is in F18, be disabled by default ( as at one time it was ) and other units that have dependency on it being present and running before starting should pull it in as a soft or hard requirement or user/administrators enabled manually for those corner cases where it's necessary.

Comment 21 Harald Hoyer 2013-03-26 19:03:46 UTC
my dump is with those changes in place

and no, there is no 30s delayed boot time for *every* fedora users

Comment 22 Harald Hoyer 2013-03-26 19:04:41 UTC
$ systemd-analyze 
Startup finished in 6s 2ms 201us (firmware) + 44ms 169us (loader) + 1s 106ms 757us (kernel) + 391ms 946us (initrd) + 1s 451ms 664us (userspace) = 8s 996ms 737us

Comment 23 Bill Nottingham 2013-03-26 19:29:00 UTC
(In reply to comment #18)
> # systemctl show remote-fs-setup.target

<snip>

> What's the problem?

systemctl enable NetworkManager-wait-online.service no longer does what the user expects/what it did in prior releases. It's moved from an optional service that is also enabled automatically when it's needed for one specific case to a service that's enabled OOTB for that one specific case but can't be enabled except by manual symlinking otherwise.

Comment 24 Harald Hoyer 2013-03-26 19:39:55 UTC
Ah, now I understand:

# systemctl enable NetworkManager-wait-online.service

was enabling it for the network.target (remote-fs-pre was hardcoded).

Now:

# systemctl enable NetworkManager-wait-online.service

would enable it for the network.target and remote-fs-setup.target.

Which means, that NetworkManager-wait-online.service is pulled in via the network.target now by default with the preset.

And

# systemctl disable NetworkManager-wait-online.service

would also remove NM-w-o.service remote-fs-setup.target

Comment 25 Lennart Poettering 2013-03-26 19:58:34 UTC
so, let me get this right. There are now three states Fedora wants to cover:

a) nm-w-o.s off
b) nm-w-o.s on, but only pulled in from r-f-s.t
c) nm-w-o.s on, but pulled into all boots (i.e. multi-user.target)

systemctl enable/disable of course only covers I binary choice.

Bill's suggestion made a) impossible, except with masking.
My suggestion made c) impossible, except with manually linking it in.

Is that a correct summary?

Comment 26 Bill Nottingham 2013-03-26 20:00:51 UTC
Think so, yes. I thought a) was fairly pointless in that if you want to mask it in r-f-s.t you're already doing something a little odd.

Comment 27 Jóhann B. Guðmundsson 2013-03-26 21:01:23 UTC
To me it looks like a simply matter of adding hard requirement to the services that is required for the remote mount in the first places. 

All the enterprise storage services on root ( like iscsi,multipath,nfs etc ) should have

Require=NetworkManager-wait-online.service
Before=remote-fs-pre.target
After=network.target networkManager-wait-online.service ...

And probably be [Installed] to something that is available via the rescue.target 
( not multi-user.target you know encase of failures )

Which leaves what left for remote mounts?

Comment 28 Jóhann B. Guðmundsson 2013-03-26 21:03:04 UTC
To me it looks like a simply matter of adding hard requirement to the services that is required for the remote mount in the first places. 

All the enterprise storage services on root ( like iscsi,multipath,nfs etc ) should have

Requires=NetworkManager-wait-online.service
Before=remote-fs-pre.target
After=network.target networkManager-wait-online.service ...

And probably be [Installed] to something that is available via the rescue.target 
( not multi-user.target you know encase of failures )

Which leaves what left for remote mounts?

Comment 29 Jóhann B. Guðmundsson 2013-03-26 21:04:19 UTC
So much for bugzilla committing only my second comment on collision...

Comment 30 Jóhann B. Guðmundsson 2013-03-26 21:21:14 UTC
The problem with the double comment of mine ( which to me is the correct approach ) is that we must not install/enable all those enterprise storages solution by default unless the users has chosen to do so at install time either by choosing the relevant packages or by the chosen or custom partitioning scheme. ( arguably those should not be pulled in as dependency either for anything being shipped on the default *DE either  ) 

Otherwise we risk having the networkManager-wait-online.service enabled by default through the requirement

Comment 31 Bill Nottingham 2013-03-27 14:31:21 UTC
For the rootfs, it's all done in dracut - I'm not sure we should use the same dependencies there that we would on the final system for post-dracut, and I believe having the dependencies come from the mount makes more sense than coming from the services.

Comment 32 Lennart Poettering 2013-03-27 16:50:13 UTC
OK, so let's take a step back here. We don't actually need NM-w-o.s being pulled in by the remote mounts, do we? It would totally suffice if we can make sure those mounts are ordered after network.target -- as long as network.target is ordered after NM itself, and pulled in by that. And we need that because on shutdown we want to make sure that the mounts go away first, and NM only then.

So, here's what I propose now:

a) order all remount mounts not only after remote-fs-pre.target (as we already do) but also explicitly against network.target. [ And network.target continues to be this passive thing that NM.s pulls in and orders itself before ]

b) remove remote-fs-setup.target again

c) introduce instead "network-online.target". This is an active target that pulls in and is ordered after NM-w-o.service (basically, network-online.target shall just be a generic name for this, so that systemd doesn't have to hardcode NM's service names...). This new target is pulled in by all remote mounts.

So let's look at the three interesting cases with this in place:

1) Initial transaction with no remote mounts won't include NM-w-o.s, since nothing pulls it in. WIN!

2) Initial transaction with remote mounts will pull it in, because the mounts pull in n-o.t which pulls in NM-w-o.s. WIN!

3) If an NFS mount suddenly apepars during runtime, we won't pull in n-o.t/NM-w-o.s since systemd won't 'retroactively' activate units. At shutdown things are clean because network.target has been pulled in by NM.s. WIN!

So that already sounds pretty good.

By naming this network-online.target we actually describe what this does, rather then what it is useful for. i.e. NM never has to know anything about remote mounts or anything like that...

Of course, it will be a source of confusion that network.target is passive (i.e. pulled in by the provider of a service, not by the consumer), but network-online.target is active (i.e. pulled in by the consumer of a service, not the provider).

Does this make sense? if so, I'll make the necessary changes upstream, and explain the exact changes necessary in NM.rpm.

Comment 33 Bill Nottingham 2013-03-27 17:36:16 UTC
Your first paragraph of description doesn't make sense to me - the remote mounts need more than a basic network.target (which OOTB just means NM has started), they need the stronger semantics of a network-online.target or NM-wait-online.service. Which is what you then propose, so it ends up making sense.

My concerns would be:
1) this doesn't specifically help solve the problem of what services want from network.target outside of the remote mount case. Of course, I'm not sure that *can* be solved cleanly, as whether a service requires basic network.target (loopback & NM presence) or network-online.target (a specific address) is usually dependent on administrator-specific configuration.
2) For the case where the administrator *does* want a synchronous network startup due to some service, it's not clear what the mechanism is - edit
the service to s/After=network.target/After=network-online.target/? Add network-online.target to network.target.wants? Something else?

Comment 34 Lennart Poettering 2013-03-28 12:45:41 UTC
(In reply to comment #33)

> 1) this doesn't specifically help solve the problem of what services want
> from network.target outside of the remote mount case. Of course, I'm not
> sure that *can* be solved cleanly, as whether a service requires basic
> network.target (loopback & NM presence) or network-online.target (a specific
> address) is usually dependent on administrator-specific configuration.

Actually I think it does. 

We now have a "softer" and weakly defined network.target that people can order themselves against. And while they won't get guaranteed much at startup if they do that they do get something very useful guaranteed at shutdown for it: that they are shutdown themselves *before* the network -- to whichever level it is configured at that point --- goes down.

And we have the "stronger" network-online.target that people can pull in themselves and order themselves against. This will delay the boot, so it comes at a certain price, and hence should only be used where either we really really know we need it (which is the remote mounts case), or where the user really wants it...

> 2) For the case where the administrator *does* want a synchronous network
> startup due to some service, it's not clear what the mechanism is - edit
> the service to s/After=network.target/After=network-online.target/? Add
> network-online.target to network.target.wants? Something else?

Issue "systemctl enable NetworkManager-wait-online", as before. That would hook NM-w-o into network.target, in addition to network-online.target (where we'd hook it in statically).

hence:

network.target: passive unit whose purpose is weakly defined at startup, but very precisely at shutdown.

network-online.target: active unit whose purpose is more strictly defined at startup, and which isn't any different from network.target at shutdown.

NetworkManager.service is to be installed (dynamically) into multi-user.target and pulls in (statically) network.target and orders itself before it.

NetworkManager-wait-online.service is to be installed (statically) into network-online.target and orders itself before it, too. It also should pull in (statically) network.target and install itself before it. It might (dynamically) also be pulled in by network.target too, if the admin chooses so, but this is strictly optional.

Makes sense?

Comment 35 Bill Nottingham 2013-03-28 14:41:29 UTC
If I'm reading you right, the only change outside of systemd is that the static link of NM-w-o.service moves from remote-fs-pre.target to network-online.target, and the [Install] section remains the same? OK.

Comment 36 Kay Sievers 2013-03-28 15:42:43 UTC
(In reply to comment #35)
> the only change outside of systemd is that the
> static link of NM-w-o.service moves from remote-fs-pre.target to
> network-online.target, and the [Install] section remains the same?

Yes, that sounds right, and is the current plan.

Comment 37 Lennart Poettering 2013-03-29 17:28:31 UTC
OK, this is now in systemd 200 and in F19.

Please make sure that NetworkManager-wait-online.service now statically is pulled in from network-online.target, via a symlink:

/usr/lib/systemd/system/network-online.target.wants/NetworkManager-wait-online.service → /usr/lib/systemd/system/NetworkManager-wait-online.service. Or in other words:

<snip>
mkdir -p $RPM_BUILD_ROOT%{systemd_dir}/network-online.target.wants
ln -s ../NetworkManager-wait-online.service $RPM_BUILD_ROOT%{systemd_dir}/network-online.target.wants
</snip>

Then, please change NetworkManager-wait-online.service to:

<snip>
[Unit]
Description=Network Manager Wait Online
Requisite=NetworkManager.service
After=NetworkManager.service
Wants=network.target
Before=network.target network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/nm-online -q --timeout=30

[Install]
WantedBy=multi-user.target
</snip>

and NetworkManager.service to:

<snip>
[Unit]
Description=Network Manager
Wants=network.target
Before=network.target

[Service]
Type=dbus
BusName=org.freedesktop.NetworkManager
ExecStart=/usr/sbin/NetworkManager --no-daemon
# Suppress stderr to eliminate duplicated messages in syslog. NM calls openlog()
# with LOG_PERROR when run in foreground. But systemd redirects stderr to
# syslog by default, which results in logging each message twice.
StandardError=null
# NM doesn't want systemd to kill its children for it
KillMode=process

[Install]
WantedBy=multi-user.target
Alias=dbus-org.freedesktop.NetworkManager.service
</snip>

And then, please update the spec file to use these macro invocations:

<snip>
Requires: systemd >= 200-3

%post
%systemd_post NetworkManager.service NetworkManager-wait-online.service

%preun
%systemd_preun NetworkManager.service NetworkManager-wait-online.service

%postun
%systemd_postun
</snip>

I removed NM-w-o.service from the default preset files again, so this will not have the effect of enabling it by default on installation, however opens the avenue for admins to enable it by default, simply by adding it to their private preset files.

For details see: https://fedoraproject.org/wiki/Packaging:ScriptletSnippets#Systemd

Comment 38 Lennart Poettering 2013-04-04 17:36:33 UTC
When you update NetworkManager.rpm please also consider turning nm_dispatcher into a proper unit at the same time:

https://bugzilla.redhat.com/show_bug.cgi?id=948433

With that in place NM would provide systemd units for all its services, and all hooked up with the preset logic.

Comment 39 Dan Williams 2013-05-15 12:36:57 UTC
NM currently has the After=syslog.target; is that no longer required?

Comment 40 Michal Schmidt 2013-05-15 12:56:35 UTC
Correct. "After=syslog.target" has not been required since F16 or so.

Comment 41 Dan Williams 2013-05-15 15:32:56 UTC
Built for F20; F19 will follow soon. NetworkManager-0.9.9.0-2.git20130515.fc20

Comment 42 Fedora Update System 2013-06-07 22:28:25 UTC
NetworkManager-0.9.8.2-1.fc19,network-manager-applet-0.9.8.2-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.8.2-1.fc19,network-manager-applet-0.9.8.2-1.fc19

Comment 43 Fedora Update System 2013-06-09 03:28:02 UTC
NetworkManager-0.9.8.2-1.fc19, network-manager-applet-0.9.8.2-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 45 Fedora End Of Life 2015-01-09 22:30:19 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 46 Fedora End Of Life 2015-02-18 11:07:42 UTC
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.