do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

classic Classic list List threaded Threaded
83 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Marc Haber-3
Hi,

since cron uses run-parts to execute the scripts from
/etc/cron.{daily,weekly,monthly}, and run-parts is documented to
execute the contents from the directory in lexical sort-order,
packages could theoretically depend on a certain execution order of
cron jobs and fail miserably if the order is violated.

I am wondering whether there are packages that actually do depend on
that and how do find out short of unpacking the entire archive and
inspecting the scripts.

Scripts with such dependencies will probably fail miserably on systems
that are using systemd-cron instead of one of the "classic" cron
packaes since systemd will start all cron jobs in parallel, or if a
feature like RandomizeDelay (see man systemd.timer) is used to reduce
the impact in random order.

Do you people know about packages that have cron jobs that depend on
execution order? Would it probably be a good idea to proactively put
language into the Policy Manual Chapter 9.5 that scripts put into the
/etc/cron.{daily,weekly,monthly} MUST NOT depend on any execution
order and MUST be prepared to run in parallel?

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Ian Jackson-2
Marc Haber writes ("do packages depend on lexical order or {daily,weekly,monthly} cron jobs?"):
> I am wondering whether there are packages that actually do depend on
> that and how do find out short of unpacking the entire archive and
> inspecting the scripts.

My feeling is that this situation is
  - quite unlikely
  - very hard to analyse without trying it
  - easy to remediate by combining the two cron jobs into a single
     (perhaps wrapper) script
  - but there is no benefit for changing the rules, because:

> Do you people know about packages that have cron jobs that depend on
> execution order? Would it probably be a good idea to proactively put
> language into the Policy Manual Chapter 9.5 that scripts put into the
> /etc/cron.{daily,weekly,monthly} MUST NOT depend on any execution
> order and MUST be prepared to run in parallel?

"Does not depend on order" and "can be run in parallel" are not the
same.  I worry about additional concurrency.  Unlike ordering bugs,
concurrency bugs are hard to find by testing.  So running these
scripts in parallel is risky.

And, I think running cron.fooly scripts in parallel is a bad idea.
The objective is to run them "at some point", not to get it done as
soon as possible.  Running them in sequence will save electricity,
may save wear on components, and will reduce overall impact on other
uses of the same system.

Ian.

--
Ian Jackson <[hidden email]>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Marc Haber-3
On Sat, 27 Jul 2019 19:02:16 +0100, Ian Jackson
<[hidden email]> wrote:
>Marc Haber writes ("do packages depend on lexical order or {daily,weekly,monthly} cron jobs?"):
>> Do you people know about packages that have cron jobs that depend on
>> execution order? Would it probably be a good idea to proactively put
>> language into the Policy Manual Chapter 9.5 that scripts put into the
>> /etc/cron.{daily,weekly,monthly} MUST NOT depend on any execution
>> order and MUST be prepared to run in parallel?
>
>"Does not depend on order" and "can be run in parallel" are not the
>same.

Right. I didn't mean to imply that.

>I worry about additional concurrency.  Unlike ordering bugs,
>concurrency bugs are hard to find by testing.  So running these
>scripts in parallel is risky.
>
>And, I think running cron.fooly scripts in parallel is a bad idea.
>The objective is to run them "at some point", not to get it done as
>soon as possible.  Running them in sequence will save electricity,
>may save wear on components, and will reduce overall impact on other
>uses of the same system.

I fully agree with that. However, moving away from "in sequence" thing
would greatly ease the migration to systemd timers, making it easier
to get away without crond on many systems.

I am wondering whether this is something we should think about giving
up in future. We have given up so many things since we moved to
systemd, would it be worth throwing this out of the window as well?

Setting RandomizeDelaySecs sufficiently high on our daily jobs would
probably help to chop off the load pike that especially virtualization
setups running many Debian instances suffer from at 06:25 or 07:35. I
think this could be a net gain worth pursuing.

Grüße
Marc
--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Bernd Zeimetz


On 7/28/19 8:47 AM, Marc Haber wrote:
> Setting RandomizeDelaySecs sufficiently high on our daily jobs would
> probably help to chop off the load pike that especially virtualization
> setups running many Debian instances suffer from at 06:25 or 07:35. I
> think this could be a net gain worth pursuing.

definitely. we are able to see the daily logrotate/.... run in our power
usage statistics...


--
 Bernd Zeimetz                            Debian GNU/Linux Developer
 http://bzed.de                                http://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Ian Jackson-2
In reply to this post by Marc Haber-3
Marc Haber writes ("Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?"):

> On Sat, 27 Jul 2019 19:02:16 +0100, Ian Jackson
> >I worry about additional concurrency.  Unlike ordering bugs,
> >concurrency bugs are hard to find by testing.  So running these
> >scripts in parallel is risky.
> >
> >And, I think running cron.fooly scripts in parallel is a bad idea.
> >The objective is to run them "at some point", not to get it done as
> >soon as possible.  Running them in sequence will save electricity,
> >may save wear on components, and will reduce overall impact on other
> >uses of the same system.
>
> I fully agree with that. However, moving away from "in sequence" thing
> would greatly ease the migration to systemd timers, making it easier
> to get away without crond on many systems.

Why can't systemd run cron.fooly as one big timer job rather than one
timer job for each script ?

That leaves cron.fooly running in parallel with cron.barly but that is
something these jobs have to cope with anyway ?

> I am wondering whether this is something we should think about giving
> up in future. We have given up so many things since we moved to
> systemd, would it be worth throwing this out of the window as well?

Obviously, I don't think it is a good idea to break this for
non-systemd users because of difficulties making it work properly
with systemd.  Perhaps I have misunderstood you ?

Thanks,
Ian.

--
Ian Jackson <[hidden email]>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Philipp Kern-2
On 2019-07-28 15:07, Ian Jackson wrote:

> Marc Haber writes ("Re: do packages depend on lexical order or
> {daily,weekly,monthly} cron jobs?"):
>> On Sat, 27 Jul 2019 19:02:16 +0100, Ian Jackson
>> >I worry about additional concurrency.  Unlike ordering bugs,
>> >concurrency bugs are hard to find by testing.  So running these
>> >scripts in parallel is risky.
>> >
>> >And, I think running cron.fooly scripts in parallel is a bad idea.
>> >The objective is to run them "at some point", not to get it done as
>> >soon as possible.  Running them in sequence will save electricity,
>> >may save wear on components, and will reduce overall impact on other
>> >uses of the same system.
>>
>> I fully agree with that. However, moving away from "in sequence" thing
>> would greatly ease the migration to systemd timers, making it easier
>> to get away without crond on many systems.
>
> Why can't systemd run cron.fooly as one big timer job rather than one
> timer job for each script ?

Of course it could. But it's better for the administrator to export
state as granular as possible. That way the exit codes are exported into
global systemd state and you can see exactly what is failing rather than
a generic "something in cron.$foo failed, good luck finding the right
logs". systemd also gives you the logs per timer unit, rather than in
bulk, so the error is trivially visible rather than filtering a long log
for what went wrong.

Think of this as an alert condition: I want to know if, say,
popularity-contest failed and treat that with lower urgency than, say,
debsums. One is a goodie and one is evidence of potential disk
corruption. If the "cron.daily" script failed, I don't know if this has
a particular urgency until I crawled the logs.

> Obviously, I don't think it is a good idea to break this for
> non-systemd users because of difficulties making it work properly
> with systemd.  Perhaps I have misunderstood you ?

To be honest, that's something that the compatibility/init diversity
folks then need to figure out.

Kind regards
Philipp Kern

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Simon McVittie-7
In reply to this post by Ian Jackson-2
On Sun, 28 Jul 2019 at 14:07:59 +0100, Ian Jackson wrote:
> Why can't systemd run cron.fooly as one big timer job rather than one
> timer job for each script ?

Of course it could, but perhaps it shouldn't.

To avoid ambiguity, this is not a systemd design decision: it's a design
decision of systemd-cron, a separate package that reads cron jobs and
writes pairs of systemd timer and service units that result in systemd
doing approximately the same things (ana)cron would have done under the
same configuration.

systemd itself does not have cron integration, so packages with
important cron jobs must depend on some cron-daemon implementation. In
current Debian that means either Vixie cron, cronie or bcron, which are
independent daemons, or systemd-cron.

As Philipp Kern pointed out, one reason to break them up is to attribute
log messages and failures to the actual task (popularity-contest.service
or whatever), rather than to some larger unit like cron-weekly.service.
systemd-cron writes cron job output to the system log, rather than sending
it by email like traditional cron implementations, but the general
principle is similar: you can prioritize more easily if the report is
"cron.daily/man-db failed" than if the report is "cron.daily failed".

Another reason to break them up is that if the same package installs
both a cron script like /etc/cron.daily/man-db and a systemd unit like
/lib/systemd/system/man-db.timer, systemd-cron deliberately generates its
unit to be named man-db.timer, so that it will be "shadowed" by the native
man-db.timer (which is in a higher-precedence directory). This avoids
log noise (and inefficiency, but the log noise is more important) from
/etc/cron.daily/man-db starting, noticing that this is a systemd system,
and exiting early so that it won't duplicate the timer unit. This is the
same "shadowing" mechanism that ensures that if a package has a systemd
service dbus.service and an init script /etc/init.d/dbus, then the init
script will be skipped. If every script in cron.daily is shadowed in
this way, no daily systemd-cron timers will be scheduled at all.

systemd-cron *could* generate one big timer unit that activates a "target"
(a group of services), generate a separate service per script as part of
that target, chain them together in lexical order by building a linked
list of Before or After directives, and implement the "don't schedule
this one if there's a native timer" logic itself instead of relying
on shadowing - but right now it doesn't, and what it does now is a lot
simpler. Introducing more complexity might be a necessary evil if real
cron jobs rely on it, but it seems better to avoid that if possible.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Simon McVittie-7
On Mon, 29 Jul 2019 at 18:26:22 +0100, Simon McVittie wrote:
> On Sun, 28 Jul 2019 at 14:07:59 +0100, Ian Jackson wrote:
> > Why can't systemd run cron.fooly as one big timer job rather than one
> > timer job for each script ?
>
> Of course it could, but perhaps it shouldn't.
>
> To avoid ambiguity, this is not a systemd design decision: it's a design
> decision of systemd-cron

Correction: at the moment, systemd-cron *does* run cron.daily, etc. as
one big timer/service pair per interval, using essentially the same
run-parts call as for traditional cron.

If scripts rely on lexical order, then it will have to continue to do this,
and maintainers who want to opt-in to having their periodic script run
outside that sequence on systemd systems will have to write native timer
units.

(Conversely, if scripts don't rely on lexical order, then systemd-cron
would be free to schedule them separately if the trade-offs around
race-to-idle, predictable load spikes, etc. make that desirable.)

> Another reason to break them up is that if the same package installs
> both a cron script like /etc/cron.daily/man-db and a systemd unit like
> /lib/systemd/system/man-db.timer, systemd-cron deliberately generates its
> unit to be named man-db.timer, so that it will be "shadowed" by the native
> man-db.timer (which is in a higher-precedence directory).

As a result, systemd-cron doesn't currently do this for daily, etc. cron
jobs (but it does for cron.d jobs).

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Simon McVittie-7
In reply to this post by Marc Haber-3
On Wed, 24 Jul 2019 at 20:14:22 +0200, Marc Haber wrote:
> Scripts with such dependencies will probably fail miserably on systems
> that are using systemd-cron instead of one of the "classic" cron
> packaes

I thought so too, but they don't: systemd-cron uses run-parts for
cron.{hourly,etc.} too. It only generates individual, parallelizable
systemd timers for scripts in cron.d, which in general run on their own
independent schedules anyway.

On Sun, 28 Jul 2019 at 08:47:49 +0200, Marc Haber wrote:
> Setting RandomizeDelaySecs sufficiently high on our daily jobs would
> probably help to chop off the load pike that especially virtualization
> setups running many Debian instances suffer from at 06:25 or 07:35. I
> think this could be a net gain worth pursuing.

If you want this, the way to opt-in to it is currently to add a native
systemd timer, and make the traditional cron script a no-op on systems
running systemd (to avoid doing the periodic task twice). For example,
apt and man-db have both done this.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Colin Watson
In reply to this post by Simon McVittie-7
On Mon, Jul 29, 2019 at 06:36:00PM +0100, Simon McVittie wrote:
> On Mon, 29 Jul 2019 at 18:26:22 +0100, Simon McVittie wrote:
> > Another reason to break them up is that if the same package installs
> > both a cron script like /etc/cron.daily/man-db and a systemd unit like
> > /lib/systemd/system/man-db.timer, systemd-cron deliberately generates its
> > unit to be named man-db.timer, so that it will be "shadowed" by the native
> > man-db.timer (which is in a higher-precedence directory).
>
> As a result, systemd-cron doesn't currently do this for daily, etc. cron
> jobs (but it does for cron.d jobs).

It's also a little hard to see how it could rely on shadowing for this,
since it's permitted for e.g. /etc/cron.daily/foo and
/etc/cron.weekly/foo to both exist and do different things.

(This is the case for man-db.  When adding its systemd unit the
submitter of #858022 decided and I agreed that the differences between
the daily and weekly scripts weren't worth representing, and I might yet
change the traditional cron.* scripts to match that decision, but this
wasn't a decision that an automatic conversion system could have taken.)

--
Colin Watson                                       [[hidden email]]

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Marc Haber-3
In reply to this post by Simon McVittie-7
On Mon, 29 Jul 2019 18:42:34 +0100, Simon McVittie <[hidden email]>
wrote:
>On Wed, 24 Jul 2019 at 20:14:22 +0200, Marc Haber wrote:
>> Scripts with such dependencies will probably fail miserably on systems
>> that are using systemd-cron instead of one of the "classic" cron
>> packaes
>
>I thought so too, but they don't: systemd-cron uses run-parts for
>cron.{hourly,etc.} too.

... this imports many disadvantages of the old scheme into the new
world. Philipp has explained in this thread very well.

Maybe systemd-cron could be extended to be locally configurable
whether to use run-parts, keeping the old semantics, or to generate
individual timers.

systemd itself would need a means to group different timers together
with the constraint that at most X of those can run in parallel.

>On Sun, 28 Jul 2019 at 08:47:49 +0200, Marc Haber wrote:
>> Setting RandomizeDelaySecs sufficiently high on our daily jobs would
>> probably help to chop off the load pike that especially virtualization
>> setups running many Debian instances suffer from at 06:25 or 07:35. I
>> think this could be a net gain worth pursuing.
>
>If you want this, the way to opt-in to it is currently to add a native
>systemd timer, and make the traditional cron script a no-op on systems
>running systemd (to avoid doing the periodic task twice). For example,
>apt and man-db have both done this.

... which takes away flexibility and standardization from the local
admin, adding more things to think about when developing larger
infrastructures. I am not sure whether I like that.

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Ian Jackson-2
Marc Haber writes ("Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?"):

> On Mon, 29 Jul 2019 18:42:34 +0100, Simon McVittie <[hidden email]>
> wrote:
> >On Wed, 24 Jul 2019 at 20:14:22 +0200, Marc Haber wrote:
> >> Scripts with such dependencies will probably fail miserably on systems
> >> that are using systemd-cron instead of one of the "classic" cron
> >> packaes
> >
> >I thought so too, but they don't: systemd-cron uses run-parts for
> >cron.{hourly,etc.} too.
>
> ... this imports many disadvantages of the old scheme into the new
> world. Philipp has explained in this thread very well.

Right.  That makes sense.

So it seems to me that there are the following options for systemd
users:

A. Continue to use run-parts.

   Disadvantages: Bundles the output together.
   Doesn't provide individual status.

   Advantages: No work needed.

B. Run each script as a single systemd timer unit.

   Disadvantages: Runs scripts in parallel (causing load spikes and
   other undesirable performance effects).  Imposes un-auditable and
   undebuggable new concurrency/ordering requirements on scripts (that
   is, scripts must cope with concurrent running and in any order).
   Ie, effectively, exposes systemd users to new bugs.

   Advantages: Unbundled output and individual status.

C. Provide a feature in systemd to do what is needed.
   Advantages: Everything works properly.
   Disadvantage: Requires writing new code to implement the feature.

D. Provide a version of run-parts which does per-script reporting.
   Advantages: Provides the benefits of (c) to sysvinit/cron users
   too.  Disadvantages: Requires design of a new reporting protocol,
   and implementation of that protocol in run-parts; to gain the
   benefits for systemd users, must be implemented in systemd too.
   (Likewise for cron users.)

I was wrong when earlier I wrote that sysvinit users would suffer any
significant trouble, regardless what option above is picked.  The only
negative impact of (B) on sysvinit users is that maybe the cron
scripts they are running become more complex and fragile, in order to
try to satisfy the new concurrency requirement.

I would be interested in (D) if you thought it would be worthwhile.
Maybe the subunit v1 protocol or something :-).

> Maybe systemd-cron could be extended to be locally configurable
> whether to use run-parts, keeping the old semantics, or to generate
> individual timers.

There would be question of the default.

With current code the options are:

A. Things run in series but with concatenated output and no individual
   status.

B. Things run in parallel, giving load spikes and possible concurrency
   bugs; vs.

I can see few people who would choose (B).

People who don't care much about paying attention to broken cron
stuff, or people who wouldn't know how to fix it, are better served by
(A).  It provides a better experience.

Knowledgeable people will not have too much trouble interpreting
combined output, and maybe have external monitoring arrangements
anyway.  Conversely, heisenbugs and load spikes are still undesirable.
So they should also choose (A).

IOW reliability and proper operation is more important than separated
logging and status reporting.

Ian.

--
Ian Jackson <[hidden email]>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Philipp Kern-6
On 2019-08-05 17:34, Ian Jackson wrote:

> With current code the options are:
>
> A. Things run in series but with concatenated output and no individual
>    status.
>
> B. Things run in parallel, giving load spikes and possible concurrency
>    bugs; vs.
>
> I can see few people who would choose (B).
>
> People who don't care much about paying attention to broken cron
> stuff, or people who wouldn't know how to fix it, are better served by
> (A).  It provides a better experience.
>
> Knowledgeable people will not have too much trouble interpreting
> combined output, and maybe have external monitoring arrangements
> anyway.  Conversely, heisenbugs and load spikes are still undesirable.
> So they should also choose (A).
>
> IOW reliability and proper operation is more important than separated
> logging and status reporting.

If we are in agreement that concurrency must happen with proper locking
and not depend on accidental lineralization then identifying those
concurrency bugs is actually a worthwhile goal in order to achieve
reliability, is it not? I thought you would be the first to acknowledge
that bugs are worth fixing rather than sweeping them under the rug. We
already identified that parallelism between the various stages is
undesirable. With a systemd timer you can declare conflicts as well as a
lineralization if so needed.

I also question the "knowledgeable people will not have too much
trouble". Export state as granular as possible and there is no guesswork
required. I have no doubt that my co-workers can do this. But I want
their life to be as easy as possible.

Similarly I wonder what the external monitoring should be apart from
injecting fake jobs around every run-parts unit in this case. Replacing
run-parts with something monitoring-aware? Then why not take the tool
that already exists (systemd)?

And finally, the load spikes: Upthread it was mentioned that
RandomizedDelaySec exists. Generally this should be sufficient to even
out such effects. I understand that there is a case where you run a lot
of unrelated VMs that you cannot control. In other cases, like laptops
and desktops, it is very likely much more efficient to generate the load
spike and complete the task as fast as possible in order to return to
the low-power state of (effectively) waiting for input. I suspect that
there is a conflict between the two that could be dealt with by
encouraging liberal use of DefaultTimerAccuracySec on the system-level.
I understand that Debian inherently does not distinguish between the two
cases. I'd still expect a Cloud/Compute provider to offer default images
in any case that could be preconfigured appropriately.

I apologize that I think of this in terms of systemd primitives. But the
tool was written for a reason and a lot of thought went into it.

Kind regards
Philipp Kern

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Bill Allombert-4
On Mon, Aug 05, 2019 at 10:29:41PM +0200, Philipp Kern wrote:
> And finally, the load spikes: Upthread it was mentioned that
> RandomizedDelaySec exists. Generally this should be sufficient to even out
> such effects. I understand that there is a case where you run a lot of
> unrelated VMs that you cannot control. In other cases, like laptops and
> desktops, it is very likely much more efficient to generate the load spike
> and complete the task as fast as possible in order to return to the
> low-power state of (effectively) waiting for input.

This assumes the system has enough RAM and other ressources to complete
all the jobs in parallel.

Cheers,
--
Bill. <[hidden email]>

Imagine a large red swirl here.

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Philipp Kern-6
On 2019-08-06 13:43, Bill Allombert wrote:

> On Mon, Aug 05, 2019 at 10:29:41PM +0200, Philipp Kern wrote:
>> And finally, the load spikes: Upthread it was mentioned that
>> RandomizedDelaySec exists. Generally this should be sufficient to even
>> out
>> such effects. I understand that there is a case where you run a lot of
>> unrelated VMs that you cannot control. In other cases, like laptops
>> and
>> desktops, it is very likely much more efficient to generate the load
>> spike
>> and complete the task as fast as possible in order to return to the
>> low-power state of (effectively) waiting for input.
>
> This assumes the system has enough RAM and other ressources to complete
> all the jobs in parallel.

We are talking about cron.*. I feel like that's more of a scheduling
problem. If you run a constrained machine, I'm sure you are already
doing customization to accommodate to the shortage and you would not
want the background cron jobs to kill your production workload?

Apart from the fact that systemd actually lets you limit the resource
usage/declare what to prioritize. If you go the route of separate timers
even on a per script basis.

Kind regards
Philipp Kern

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Ian Jackson-2
In reply to this post by Philipp Kern-6
Philipp Kern writes ("Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?"):

> On 2019-08-05 17:34, Ian Jackson wrote:
> > With current code the options are:
> >
> > A. Things run in series but with concatenated output and no individual
> >    status.
> >
> > B. Things run in parallel, giving load spikes and possible concurrency
> >    bugs; vs.
> >
> > I can see few people who would choose (B).
...
> > IOW reliability and proper operation is more important than separated
> > logging and status reporting.
>
> If we are in agreement that concurrency must happen with proper locking
> and not depend on accidental lineralization then identifying those
> concurrency bugs is actually a worthwhile goal in order to achieve
> reliability, is it not?

Given that the concurrency is not actually desirable, it seems foolish
to change the execution model to a concurrent one and thereby impose
a lot of additional requirements on these scripts.

Obviously I agree that bugs, where there are bugs, should be fixed.

However, I do not agree that because bugs should be fixed,
specifications should be changed to declare buggy, things which are
currently not buggy at all.

> I thought you would be the first to acknowledge
> that bugs are worth fixing rather than sweeping them under the rug.

I confess I was quite annoyed when I read this the first time.  It
still irritates me.

I am not suggesting that anything should be "swept under the rug".
The existing scripts, which do not have any locking, are completely
correct.  They are entitled to make the assumptions that they
currently do, because that is what the specification says and because
that is how the current callers work.

Concurrency in particular is a particularly awkward thing to retrofit,
because code which was not previously concurrent suddenly becomes
unauditably buggy, and because concurrency bugs are very hard to find,
analyse and fix.

Sometimes it is necessary to retrofit concurrency for performance
reasons.  We have suffered a lot of buggy software as a result.  But
in this case concurrency is simply harmful and should be avoided.

>  We already identified that parallelism between the various stages
> is undesirable. With a systemd timer you can declare conflicts as
> well as a lineralization if so needed.

ISTM that a simple solution is for systemd to infer the appropriate
linearisation.  I don't understand why this approach is apparently
being rejected.

> And finally, the load spikes: Upthread it was mentioned that
> RandomizedDelaySec exists. Generally this should be sufficient to even
> out such effects.

A predictable scheduler which runs all the jobs in the same order each
time, sequentially, and without gaps, is clearly better than one which
runs them at random times and hopes that the resulting schedule is a
good one.


My bottom line:

The Debian systemd maintainers are of course free to enable
parallelism for systemd users; notwithstanding that everyone
apparently acknowledges that parallelism here is undesirable.

But it would be wrong to change the Debian Policy specification for
cron.fooly script execution.  There is nothing stopping systemd from
implementing the existing specification; and, there is nothing in this
specification which prevents systemd from providing whatever logging
and monitoring features are desired.
        i
And there is nothing wrong with the current specification.  The only
difficulty is that systemd doesn't currently implement it to the
systemd maintainers' satisfaction.

So any concurrency bugs (or other infelicities) resulting from
violations by systemd of the currently defined interface, should be
seen as bugs in that hypothetical concurrent systemd cron.fooly
arrangements, not bugs in the cron.fooly scripts provided by packages.

The solution is to improve systemd, not to change policy to divert the
blame for any bugs caused by anyway-unwanted concurrency.

Ian.

--
Ian Jackson <[hidden email]>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Simon McVittie-7
On Tue, 06 Aug 2019 at 16:58:31 +0100, Ian Jackson wrote:
> The Debian systemd maintainers are of course free to enable
> parallelism for systemd users;

Trying to make this point more clearly because you seem to be confusing
systemd with systemd-cron:

Whether to run cron jobs in lexical order or in parallel is not a systemd
decision, because systemd does not read or understand cron jobs. The
component that reads and interprets cron jobs and has a feature request
involving potentially running {daily,etc.} cron jobs in parallel is
systemd-cron, a third-party project that translates cron jobs into
systemd units.

systemd is entirely capable of running Vixie cron as a daemon (as is done
in default installations of stretch and buster), or running cronie as
a daemon. systemd-cron is an alternative to those, using systemd rather
than a separate daemon as the long-running component that waits for the
right time.

If there are flaws in how systemd-cron translates cron jobs into systemd
units, then those are bugs to be fixed in systemd-cron, not in systemd.

(In other words, this is one of those systems involving composing smaller
modules that people sometimes say systemd opposes or makes impossible.)

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Sam Hartman-3
In reply to this post by Ian Jackson-2
>>>>> "Ian" == Ian Jackson <[hidden email]> writes:


I think this is the big question.

Because of modern thinking about power optimization, if I were designing
this interface today I'd design it to be concurrent.
In particular, you want to use as much concurrency as you can to
maximize utilization of CPU and IO without introducing unnecessary
resource thrashing.

We've generally concluded that short periods of very high activity are
better in terms of power utilization than longer periods of moderate
activity.


So, I think this feature request is coming about exactly because some
people desire the concurrency and are questioning whether the classical
interface design still meets modern needs.

Of course we can continue to provide the same semantics.  That's what
systemd-cron does today.

The interesting question here is whether that's actually desirable.

Starting by presuming that it is sidesteps the entire discussion.

--Sam

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Bill Allombert-4
In reply to this post by Philipp Kern-6
On Tue, Aug 06, 2019 at 05:19:14PM +0200, Philipp Kern wrote:

> On 2019-08-06 13:43, Bill Allombert wrote:
> > On Mon, Aug 05, 2019 at 10:29:41PM +0200, Philipp Kern wrote:
> > > And finally, the load spikes: Upthread it was mentioned that
> > > RandomizedDelaySec exists. Generally this should be sufficient to
> > > even out
> > > such effects. I understand that there is a case where you run a lot of
> > > unrelated VMs that you cannot control. In other cases, like laptops
> > > and
> > > desktops, it is very likely much more efficient to generate the load
> > > spike
> > > and complete the task as fast as possible in order to return to the
> > > low-power state of (effectively) waiting for input.
> >
> > This assumes the system has enough RAM and other ressources to complete
> > all the jobs in parallel.
>
> We are talking about cron.*. I feel like that's more of a scheduling
> problem. If you run a constrained machine, I'm sure you are already doing
> customization to accommodate to the shortage and you would not want the
> background cron jobs to kill your production workload?

One background cron job at once might be OK, all at once less so, even
on a full server.

Some of the cron jobs are IO intensive, some other RAM intensive
(think updatedb, Web indexing software like xapian-omega etc).

Cheers,
--
Bill. <[hidden email]>

Imagine a large red swirl here.

Reply | Threaded
Open this post in threaded view
|

Re: do packages depend on lexical order or {daily,weekly,monthly} cron jobs?

Marc Haber-3
In reply to this post by Ian Jackson-2
On Mon, 5 Aug 2019 16:34:18 +0100, Ian Jackson
<[hidden email]> wrote:

>So it seems to me that there are the following options for systemd
>users:
>
>A. Continue to use run-parts.
>
>   Disadvantages: Bundles the output together.
>   Doesn't provide individual status.
>
>   Advantages: No work needed.
>
>B. Run each script as a single systemd timer unit.
>
>   Disadvantages: Runs scripts in parallel (causing load spikes and
>   other undesirable performance effects).  Imposes un-auditable and
>   undebuggable new concurrency/ordering requirements on scripts (that
>   is, scripts must cope with concurrent running and in any order).
>   Ie, effectively, exposes systemd users to new bugs.
>
>   Advantages: Unbundled output and individual status.

In fact, systemd-cron already has an issue open upstream to change
this behavior, see
https://github.com/systemd-cron/systemd-cron/issues/47

I have taken the liberty of commenting a pointer to this discussion
into the upstream issue and also making my case to make tihs
configurable.

>C. Provide a feature in systemd to do what is needed.
>   Advantages: Everything works properly.
>   Disadvantage: Requires writing new code to implement the feature.

Imo, there should be a possibility in a systemd timer to switch on the
"old" output-to-e-mail behavior. This is probably something that
systemd upstream would never implement, so we'd end up with a wrapper
that is called by the systemd timer unit.

>D. Provide a version of run-parts which does per-script reporting.
>   Advantages: Provides the benefits of (c) to sysvinit/cron users
>   too.  Disadvantages: Requires design of a new reporting protocol,
>   and implementation of that protocol in run-parts; to gain the
>   benefits for systemd users, must be implemented in systemd too.
>   (Likewise for cron users.)

We have already thrown sysvinit away. Things are exactly like I
predicted, the init scripts are rotting away quickly. I think we
should stop wasting personpower for sysvinit support, people not
liking systemd have gone to devuan anyway.

>With current code the options are:
>
>A. Things run in series but with concatenated output and no individual
>   status.
>
>B. Things run in parallel, giving load spikes and possible concurrency
>   bugs; vs.
>
>I can see few people who would choose (B).

>From an individual system's POV, with RandomizeDelay the load would be
higher, granted, but from a virtualization environment's POV the load
spike would greatly be evened out when the VMs don't invoke their
run-parts in the same second. I know virtualization operators who call
the 06:25 peak in their load "The Debian peak".

I would choose Option B in all my use cases, with different
parameters: As a server jockey, I'd use a high RandomizeDelay to even
out the impact on the infrastructure, and on my personal laptop, I'd
probably use a very small RandomizeDelay so that the box can return to
power-save idle faster.

What keeps me from doing this now is the lack of e-mail output.

>People who don't care much about paying attention to broken cron
>stuff, or people who wouldn't know how to fix it, are better served by
>(A).  It provides a better experience.

I disagree.

>Knowledgeable people will not have too much trouble interpreting
>combined output, and maybe have external monitoring arrangements
>anyway.  Conversely, heisenbugs and load spikes are still undesirable.
>So they should also choose (A).

I disagree.

>IOW reliability and proper operation is more important than separated
>logging and status reporting.

I would choose B to help in debugging the scripts. Better force those
errors now to have them ironed out for bullseye. The current behavior
of run-parts doing things in lexical order and serialized is basically
a historically caused coincidence.

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber         |   " Questions are the         | Mailadresse im Header
Mannheim, Germany  |     Beginning of Wisdom "     |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

12345