Heads up: persistent journal has been enabled in systemd

classic Classic list List threaded Threaded
117 messages Options
1234 ... 6
Reply | Threaded
Open this post in threaded view
|

Heads up: persistent journal has been enabled in systemd

Michael Biebl-3
Hi,

with today's upload of systemd 244.1-2 I finally enabled persistent
journal by default [1]. It has been a long requested feature.

The package will create a directory /var/log/journal on upgrades and new
installs, which enables persistent journal in so called auto mode.

If you decide, that you want to disable the persistent journal again,
you can run:
journalctl --relinquish-var; rm -rf /var/log/journal

Future package updates will respect this choice and not re-create the
directory. You can, of course, also configure this explicitly via the
Storage= option in journald.conf.

Depending on how it goes, I might ask the ftp-masters to lower the
priority of rsyslog from important to optional, so it would no longer be
installed by default on new bullseye installations.
This would avoid, that we store log messages twice on disk.
Users that prefer text logs can of course still install rsyslog by
default (or their syslogger of choice).
Alternative init systems might consider adding a Recommends on a syslog
implementation of their choice or creating a task, which would pull in a
syslogger.

Here are some resources that you might find useful:
- man journald.conf
  https://www.freedesktop.org/software/systemd/man/journald.conf.html#
- man journalctl
  https://www.freedesktop.org/software/systemd/man/journalctl.html#
- man systemd-journald.service
  https://www.freedesktop.org/software/systemd/man/systemd-journald.html#

Regards,
Michael


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717388


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Steve McIntyre
Michael Biebl wrote:
>
>with today's upload of systemd 244.1-2 I finally enabled persistent
>journal by default [1]. It has been a long requested feature.
>
>The package will create a directory /var/log/journal on upgrades and new
>installs, which enables persistent journal in so called auto mode.

Fine for new installations, but please *don't* do this for
upgrades. Those people with existing logging setups will be surprised
by this.

--
Steve McIntyre, Cambridge, UK.                                [hidden email]
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Didier 'OdyX' Raboud-5
Le samedi, 1 février 2020, 14.36:20 h CET Steve McIntyre a écrit :

> Michael Biebl wrote:
> >with today's upload of systemd 244.1-2 I finally enabled persistent
> >journal by default [1]. It has been a long requested feature.
> >
> >The package will create a directory /var/log/journal on upgrades and new
> >installs, which enables persistent journal in so called auto mode.
>
> Fine for new installations, but please *don't* do this for
> upgrades. Those people with existing logging setups will be surprised
> by this.
I disagree. Please _do_ this for upgrades (it's great functionality!), but
please do make sure that it is documented in NEWS.Debian (and release notes)
so that it gets mentioned to host admins upon upgrade.

--
    OdyX

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Bernd Zeimetz
In reply to this post by Steve McIntyre


On 2/1/20 2:36 PM, Steve McIntyre wrote:

> Michael Biebl wrote:
>>
>> with today's upload of systemd 244.1-2 I finally enabled persistent
>> journal by default [1]. It has been a long requested feature.
>>
>> The package will create a directory /var/log/journal on upgrades and new
>> installs, which enables persistent journal in so called auto mode.
>
> Fine for new installations, but please *don't* do this for
> upgrades. Those people with existing logging setups will be surprised
> by this.


Yes, Steve is right here, please don't mess with existing
configurations. In most cases the result would not be what people wanted
to have.


--
 Bernd Zeimetz                            Debian GNU/Linux Developer
 http://bzed.de                                http://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Michael Biebl-3
In reply to this post by Steve McIntyre
Hi Steve

Am 01.02.20 um 14:36 schrieb Steve McIntyre:

> Michael Biebl wrote:
>>
>> with today's upload of systemd 244.1-2 I finally enabled persistent
>> journal by default [1]. It has been a long requested feature.
>>
>> The package will create a directory /var/log/journal on upgrades and new
>> installs, which enables persistent journal in so called auto mode.
>
> Fine for new installations, but please *don't* do this for
> upgrades. Those people with existing logging setups will be surprised
> by this.
I honestly don't like packages that behave differently depending on
whether they were upgraded or installed anew (given their default
configuration wasn't changed). I assume there would be equally as many
or even more surprised users that find their upgraded system behave
differently then their freshly installed system.

The change here is basically just an update of the default behaviour of
journald. If you explicitly configured a different journald behavior via
Storage=, this is respected. If you already created a /var/log/journal
in the past, the change will be a nop.

Existing sysloggers will continue to work after this change as before,
they are not directly affected by this change.

Can you eloborate more on this what your concerns are here?

I think Didier's idea of documenting this in the release notes is a good
one, so I filed #950447. The NEWS entry is a good idea as well and I
thought about that, but decided to not add it (yet) but instead ask for
feedback on debian-devel first. Depending on the feedback, the content
of that NEWS entry might change. E.g. I was unsure whether I should
mention that a syslogger like rsyslog can now be uninstalled safely, as
the contents are now available in the journal as well. Feedback on this
welcome.

I would also like to mention that there is prior art. Ubuntu made the
switch to a persistent journal a while back and they did it on upgrades
and new installations [1] as well.
I've CCed Dimitri, just in case he wants to chime in here. Maybe he has
some feedback on this transition in Ubuntu.

Regards,
Michael


[1]
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f94f18d9dbc085b6a9ff33c141a6e542142f85b5



signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Dimitri John Ledkov-2
On Sat, 1 Feb 2020 at 22:01, Michael Biebl <[hidden email]> wrote:

>
> Hi Steve
>
> Am 01.02.20 um 14:36 schrieb Steve McIntyre:
> > Michael Biebl wrote:
> >>
> >> with today's upload of systemd 244.1-2 I finally enabled persistent
> >> journal by default [1]. It has been a long requested feature.
> >>
> >> The package will create a directory /var/log/journal on upgrades and new
> >> installs, which enables persistent journal in so called auto mode.
> >
> > Fine for new installations, but please *don't* do this for
> > upgrades. Those people with existing logging setups will be surprised
> > by this.
>
> I honestly don't like packages that behave differently depending on
> whether they were upgraded or installed anew (given their default
> configuration wasn't changed). I assume there would be equally as many
> or even more surprised users that find their upgraded system behave
> differently then their freshly installed system.
>
> The change here is basically just an update of the default behaviour of
> journald. If you explicitly configured a different journald behavior via
> Storage=, this is respected. If you already created a /var/log/journal
> in the past, the change will be a nop.
>
> Existing sysloggers will continue to work after this change as before,
> they are not directly affected by this change.
>
> Can you eloborate more on this what your concerns are here?
>
> I think Didier's idea of documenting this in the release notes is a good
> one, so I filed #950447. The NEWS entry is a good idea as well and I
> thought about that, but decided to not add it (yet) but instead ask for
> feedback on debian-devel first. Depending on the feedback, the content
> of that NEWS entry might change. E.g. I was unsure whether I should
> mention that a syslogger like rsyslog can now be uninstalled safely, as
> the contents are now available in the journal as well. Feedback on this
> welcome.
>
> I would also like to mention that there is prior art. Ubuntu made the
> switch to a persistent journal a while back and they did it on upgrades
> and new installations [1] as well.
> I've CCed Dimitri, just in case he wants to chime in here. Maybe he has
> some feedback on this transition in Ubuntu.
>

In Ubuntu, we have enabed persistant journal by default on fresh
installs and upgrades, but only once.

# Enable persistent journal, in auto-mode, by default on new installs
installs and upgrades
if dpkg --compare-versions "$2" lt "235-3ubuntu3~"; then
    mkdir -p /var/log/journal
    # create tmpfiles only when running systemd, otherwise %b substitution fails
    if [ -d /run/systemd/system ]; then
        systemd-tmpfiles --create --prefix /var/log/journal
    fi
fi

Meaning, post-install/upgrade if one removes /var/log/journal it gets
disabled and doesn't come back again.

Overall, the feedback was extremely positive. A lot of different
classes of users were expecting journal to be there and were very
unhappy to find out retroactively that it was not there on first boot
/ last boot / broken boot. Especially people with existing logging
setups were pissed off about missing logs that only appeared in the
small ephemeral runtime journal and never made it to the persistent
journal on disk - for example early boot logs; failing to boot;
shutdown logs; and whenever remote logging services failed.

The fact that we leave it "auto" mode and simply removing
/var/log/journal gets rid of the persistent journaling is very
intuitive too, and grumpy people who dislike it manage to
disable/remove it, without any complaints (i.e. there are no bug
reports "why is journal back").

We have a recurring class of "bugs" filed by people who claim "OMG
journal is growing to unlimited size", and every single time so far
that turned out to be not true, and the caps that journald calculates
are actually enforced.
$ journalctl -u systemd-journald -b | head
System Journal (/var/log/journal/1b8df0fa27039f0163586c6756a6d401) is
3.9G, max 4.0G, 23.7M free.

After pointing out that caps are actually enfoced and rotated off, and
that one can set bigger / smaller caps for the persistent journal,
people seemed to agree that it doesn't in fact grow to unlimited
sizes.

This is the experience we have had in Ubuntu, and I do hope that in
Debian too it will be enabled on new installs and upgrades, but only
once and keep it in auto mode - meaning that removal of
/var/log/journal disables persistent journaling.

The change was done in December 2017 and thus systems installed with /
upgraded to 18.04 LTS overwhelmingly have persistent journal enabled
by default across the board.

--
Regards,

Dimitri.

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Simon khng
In reply to this post by Michael Biebl-3
Hi,

Not sure if stated in any documentation, but I will ask here since it is easier to get clarification. 

Why was rsyslog used as the persistent storage instead of journald for previous Debian distribution?

I did not check but is the current journald temp session log storing data in binary format maybe? Since it was stated in the email that users can use other log package if text format is preferable. 

On that topic, why use the network manager package instead of the networkd? This is actually a similar situation for current stable of Debian.

Simon

On Sat, Feb 1, 2020, 11:24 AM Michael Biebl <[hidden email]> wrote:
Hi,

with today's upload of systemd 244.1-2 I finally enabled persistent
journal by default [1]. It has been a long requested feature.

The package will create a directory /var/log/journal on upgrades and new
installs, which enables persistent journal in so called auto mode.

If you decide, that you want to disable the persistent journal again,
you can run:
journalctl --relinquish-var; rm -rf /var/log/journal

Future package updates will respect this choice and not re-create the
directory. You can, of course, also configure this explicitly via the
Storage= option in journald.conf.

Depending on how it goes, I might ask the ftp-masters to lower the
priority of rsyslog from important to optional, so it would no longer be
installed by default on new bullseye installations.
This would avoid, that we store log messages twice on disk.
Users that prefer text logs can of course still install rsyslog by
default (or their syslogger of choice).
Alternative init systems might consider adding a Recommends on a syslog
implementation of their choice or creating a task, which would pull in a
syslogger.

Here are some resources that you might find useful:
- man journald.conf
  https://www.freedesktop.org/software/systemd/man/journald.conf.html#
- man journalctl
  https://www.freedesktop.org/software/systemd/man/journalctl.html#
- man systemd-journald.service
  https://www.freedesktop.org/software/systemd/man/systemd-journald.html#

Regards,
Michael


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717388

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Anthony DeRobertis


On February 2, 2020 12:02:33 PM UTC, Simon khng <[hidden email]> wrote:

>Why was rsyslog used as the persistent storage instead of journald for
>previous Debian distribution?

rsyslog has been the default Debian log storage since before switching to systemd, possibly since before systemd existed (it was syslogd, but would have to check if it was rsyslog or some other implementation).

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Michael Stone-2
On Sun, Feb 02, 2020 at 02:35:19PM +0000, Anthony DeRobertis wrote:
>On February 2, 2020 12:02:33 PM UTC, Simon khng <[hidden email]> wrote:
>>Why was rsyslog used as the persistent storage instead of journald for
>>previous Debian distribution?
>
>rsyslog has been the default Debian log storage since before switching to systemd, possibly since before systemd existed (it was syslogd, but would have to check if it was rsyslog or some other implementation).

Way back at the beginning there was syslogd, that got combined early on
with klogd into the sysklogd package, rsyslogd replaced it on default
installs in lenny (released 2009). So yes, the reason is that it
predates systemd. :)

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Dmitry Smirnov
In reply to this post by Michael Biebl-3
On Saturday, 1 February 2020 2:05:55 PM AEDT Michael Biebl wrote:
> Depending on how it goes, I might ask the ftp-masters to lower the
> priority of rsyslog from important to optional, so it would no longer be
> installed by default on new bullseye installations.

I have a mixed feelings about that. I think that replacing rsyslog with
journald is two steps back in regards to functionality and flexibility.

Rsyslog in unparalleled with its ability to process and filter messages
(rainerscript), to transform messages (liblognorm), to forward messages to
Elasticsearch or centralise Rsyslog instance, _reliably_ (relp), to buffer
message queue on disk in case communication is disrupted, etc.

Am I correct that journald is nowhere near that functionality?
Also maturity/stability -- Rsyslog has been around much longer and it never
let me down...

As a heavy user or Rsyslog features I feel that switching default logging
system yields no benefits to say the least.

--
Best wishes,
 Dmitry Smirnov

---

Men become civilized, not in proportion to their willingness to believe,
but in proportion to their readiness to doubt.
        -- H. L. Mencken

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Russ Allbery-2
Dmitry Smirnov <[hidden email]> writes:
> On Saturday, 1 February 2020 2:05:55 PM AEDT Michael Biebl wrote:

>> Depending on how it goes, I might ask the ftp-masters to lower the
>> priority of rsyslog from important to optional, so it would no longer
>> be installed by default on new bullseye installations.

> I have a mixed feelings about that. I think that replacing rsyslog with
> journald is two steps back in regards to functionality and flexibility.

> Rsyslog in unparalleled with its ability to process and filter messages
> (rainerscript), to transform messages (liblognorm), to forward messages
> to Elasticsearch or centralise Rsyslog instance, _reliably_ (relp), to
> buffer message queue on disk in case communication is disrupted, etc.

I completely agree with this assessment of the quality of rsyslog
features, but I'm not sure that's the right criteria to be considering for
the choice of the *default*.  I'm fairly certain that 95% or more of
installed Debian systems never used any of those features, as nice as they
are.

The goal of the default is not to provide in latent form every excellent
feature that anyone may want to use.  It's to provide a hopefully simple,
reliable, functional, and light-weight implementation of a facility that
serves as a reasonable default for most systems.  Anyone with other needs
or preferences is very likely to replace or supplement that implementation
with something else, similar to how I replace exim4 with postfix on all of
my systems.

> Am I correct that journald is nowhere near that functionality?

Yes.

> As a heavy user or Rsyslog features I feel that switching default
> logging system yields no benefits to say the least.

As a heavy user, perhaps you're not the target audience for a default?
You're going to install rsyslog no matter what, since you know it well and
use it heavily.  The only effect of this change on you will be a one-line
change to whatever you use for configuration management for new systems.

The primary benefit that I can see is one fewer daemon running on a
default installation, one fewer thing to have security vulnerabilities or
some other problems, one fewer thing to keep up to date, and a smaller
base installation.  To be clear, these benefits are fairly minor, but they
do exist.

--
Russ Allbery ([hidden email])              <https://www.eyrie.org/~eagle/>

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Vincent Bernat-3
 ❦  4 février 2020 11:30 -08, Russ Allbery <[hidden email]>:

>> As a heavy user or Rsyslog features I feel that switching default
>> logging system yields no benefits to say the least.
>
> As a heavy user, perhaps you're not the target audience for a default?
> You're going to install rsyslog no matter what, since you know it well and
> use it heavily.  The only effect of this change on you will be a one-line
> change to whatever you use for configuration management for new
> systems.

rsyslog even knows how to directly pull logs from the journal, which
gives you access to stuff not logged to syslog (stdout/stderr of service
files, applications logging directly to journal), as well to structured
logs (comm pid, user, unit and more when the service supports journald
directly).
--
Use the good features of a language; avoid the bad ones.
            - The Elements of Programming Style (Kernighan & Plauger)

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Dmitry Smirnov
In reply to this post by Russ Allbery-2
On Wednesday, 5 February 2020 6:30:03 AM AEDT Russ Allbery wrote:
> The primary benefit that I can see is one fewer daemon running on a
> default installation, one fewer thing to have security vulnerabilities or
> some other problems, one fewer thing to keep up to date, and a smaller
> base installation.  To be clear, these benefits are fairly minor, but they
> do exist.

The question is whether those benefits are enough to justify replacing a very
solid and reliable logging system.

It is probably correct that most users don't use Rsyslog features. But that
doesn't mean that those features should be taken away from default
installation.

For example, if a certain daemon manifested a condition when a message is
logged too often, then with Rsyslog I could suppress noise by something like
the following

~~~~
if ($programname == "noisydaemon") then {
  if ($msg contains "frequently repeated noise") then {
    stop
  }
}
~~~~

This is just an example (probably not the best one) how feature can be handy
when it is needed. The point is that you'll only realise that you can't do
something any more is when you need it the most.

Also the cost of learning. I'm sure that more people are more familiar with
Rsyslog. So there is inconvenience too. Disruptive changing of good default
for weak reasons is not nice.

Rsyslog is not broken to be replaced as default logging system.

Finally there were no attempt to seek consensus, no survey, nothing?
Just a decision of few maintainers to replace good default with their
preferences?

It is difficult to appreciate needless disruptive changes.

--
Best wishes,
 Dmitry Smirnov.

---

It is impossible to imagine Goethe or Beethoven being good at billiards
or golf.
        -- H. L. Mencken

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Scott Kitterman-5
In reply to this post by Vincent Bernat-3
On Tuesday, February 4, 2020 5:22:15 PM EST Vincent Bernat wrote:

>  ❦  4 février 2020 11:30 -08, Russ Allbery <[hidden email]>:
> >> As a heavy user or Rsyslog features I feel that switching default
> >> logging system yields no benefits to say the least.
> >
> > As a heavy user, perhaps you're not the target audience for a default?
> > You're going to install rsyslog no matter what, since you know it well and
> > use it heavily.  The only effect of this change on you will be a one-line
> > change to whatever you use for configuration management for new
> > systems.
>
> rsyslog even knows how to directly pull logs from the journal, which
> gives you access to stuff not logged to syslog (stdout/stderr of service
> files, applications logging directly to journal), as well to structured
> logs (comm pid, user, unit and more when the service supports journald
> directly).
For those of us who aren't customizers of Debian's logging function, it'd be
nice to have a clearer understanding of what this changes means.

Today, when, for example, I want to investigate something email related, I
look in /var/log/mail.log.  Other specialized log files for their special
purposes.  For data not covered by a specialized log, I look in /var/log/
syslog.

Will the specialized log files still be there?  Will the net effect be that I
just need to look in /var/log/journal (or something similar) instead of in /
var/log/syslog?  Is the persistent journal a text file or will I need
specialized tools to interact with it?

Scott K

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Scott Kitterman-5
In reply to this post by Dmitry Smirnov
On Tuesday, February 4, 2020 5:39:19 PM EST Dmitry Smirnov wrote:

> On Wednesday, 5 February 2020 6:30:03 AM AEDT Russ Allbery wrote:
> > The primary benefit that I can see is one fewer daemon running on a
> > default installation, one fewer thing to have security vulnerabilities or
> > some other problems, one fewer thing to keep up to date, and a smaller
> > base installation.  To be clear, these benefits are fairly minor, but they
> > do exist.
>
> The question is whether those benefits are enough to justify replacing a
> very solid and reliable logging system.
>
> It is probably correct that most users don't use Rsyslog features. But that
> doesn't mean that those features should be taken away from default
> installation.
>
> For example, if a certain daemon manifested a condition when a message is
> logged too often, then with Rsyslog I could suppress noise by something like
> the following
>
> ~~~~
> if ($programname == "noisydaemon") then {
>   if ($msg contains "frequently repeated noise") then {
>     stop
>   }
> }
> ~~~~
>
> This is just an example (probably not the best one) how feature can be handy
> when it is needed. The point is that you'll only realise that you can't do
> something any more is when you need it the most.
>
> Also the cost of learning. I'm sure that more people are more familiar with
> Rsyslog. So there is inconvenience too. Disruptive changing of good default
> for weak reasons is not nice.
>
> Rsyslog is not broken to be replaced as default logging system.
>
> Finally there were no attempt to seek consensus, no survey, nothing?
> Just a decision of few maintainers to replace good default with their
> preferences?
>
> It is difficult to appreciate needless disruptive changes.
We just had a GR where the project voted it was just fine to systemd all the
things, so this sort of thing is to be expected.

Scott K

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Matt Zagrabelny
In reply to this post by Scott Kitterman-5
On Tue, Feb 4, 2020 at 5:15 PM Scott Kitterman <[hidden email]> wrote:

>
> On Tuesday, February 4, 2020 5:22:15 PM EST Vincent Bernat wrote:
> >  ❦  4 février 2020 11:30 -08, Russ Allbery <[hidden email]>:
> > >> As a heavy user or Rsyslog features I feel that switching default
> > >> logging system yields no benefits to say the least.
> > >
> > > As a heavy user, perhaps you're not the target audience for a default?
> > > You're going to install rsyslog no matter what, since you know it well and
> > > use it heavily.  The only effect of this change on you will be a one-line
> > > change to whatever you use for configuration management for new
> > > systems.
> >
> > rsyslog even knows how to directly pull logs from the journal, which
> > gives you access to stuff not logged to syslog (stdout/stderr of service
> > files, applications logging directly to journal), as well to structured
> > logs (comm pid, user, unit and more when the service supports journald
> > directly).
>
> For those of us who aren't customizers of Debian's logging function, it'd be
> nice to have a clearer understanding of what this changes means.
>
> Today, when, for example, I want to investigate something email related, I
> look in /var/log/mail.log.

Random email related journal commands:

journalctl -u postfix
journalctl -f -u postfix
journalctl -b -u postfix

the -u is for the unit name. the -b is for since boot. man journalctl
for details.

  Other specialized log files for their special
> purposes.  For data not covered by a specialized log, I look in /var/log/
> syslog.
>
> Will the specialized log files still be there?

I suppose it depends. Do the processes write to /dev/log or manually
write to files?

  Will the net effect be that I
> just need to look in /var/log/journal (or something similar) instead of in /
> var/log/syslog?

The contents of /var/log/journal will be binary files that journalctl
will read. IIRC.

  Is the persistent journal a text file or will I need
> specialized tools to interact with it?

specialized: journalctl.

-m




>
> Scott K

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Russ Allbery-2
In reply to this post by Dmitry Smirnov
Dmitry Smirnov <[hidden email]> writes:
> On Wednesday, 5 February 2020 6:30:03 AM AEDT Russ Allbery wrote:

>> The primary benefit that I can see is one fewer daemon running on a
>> default installation, one fewer thing to have security vulnerabilities
>> or some other problems, one fewer thing to keep up to date, and a
>> smaller base installation.  To be clear, these benefits are fairly
>> minor, but they do exist.

> The question is whether those benefits are enough to justify replacing a
> very solid and reliable logging system.

I'm confused.  Currently, we have both the systemd journal and, in
addition, rsyslog.  The proposal is that we would stop running the
separate rsyslog daemon by default and have only the systemd journal.

I believe the journal is already upstream of all traffic that rsyslog sees
in a default installation given that systemd controls /dev/syslog.  We
currently are running two logging infrastructures in the default
installation, and the proposal is to run only one.  What's being replaced?
Am I misunderstanding something?

I think the strongest argument against running rsyslog by default is that
currently rsyslog is responsible for receiving forwarded messages from the
journal and writing a second copy of them in the conventional text log
files.  I believe without rsyslog one would have to use journalctl,
systemctl, or similar programs to read messages, which might be a
confusing change.  I'm personally not sure whether avoiding that change is
worth double-writing log messages and running another daemon in the
default install.

It does take a bit of retraining to use journalctl instead (and I'm
personally not horribly fond of its UI, although that's probably because
I'm using it wrong), but it's a lot better at effectively narrowing log
messages to the things of interest once you get used to it.  (And anyone
who doesn't like that can just apt-get install rsyslog and be done.)

In a lot of environments, I'd probably install rsyslog anyway for various
reasons, but that's of course trivial to do.

--
Russ Allbery ([hidden email])              <https://www.eyrie.org/~eagle/>

Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Russell Stuart
On Tue, 2020-02-04 at 18:10 -0800, Russ Allbery wrote:
> It does take a bit of retraining to use journalctl instead (and I'm
> personally not horribly fond of its UI, although that's probably
> because I'm using it wrong), but it's a lot better at effectively
> narrowing log messages to the things of interest once you get used to
> it.

journald has nits I mention below, but I was prepared to put up with
them and drop rsyslog until one day a server stopped in a nasty way and
journalctl refused to display what lead up to the crash because it's
binary logs were corrupt.  As far as I was concerned this made journald
unfit for use on production servers.  (rsyslog's logs also get 4k lumps
of nulls and other garbage in them in similar situations, but they
remain usable.)

That was a long time ago, and it may well be fixed now.  But if it
isn't IMO turning off rsyslog by default is a bad idea.  My view is the
main reason Debian exists is to serve as a reliable base for production
machines.  Debian Desktop is what I use on my personal machine and yes,
dropping rsyslog hardly matters there, but I wouldn't be using Debian
Desktop if I wasn't using Debian in production.  

Another journald anti-feature (which is probably an unfair attribution
as it is almost certainly a consequence of systemd's design), is a
manually started service doesn't print the reason it refused to start
to stderr.  Having to fire up journald and wade through it's crappy UI
to get something sysV used to put under my nose is a step backwards.

Finally, it may be I just don't know how to use it well, but looking
for a needle in a haystack of logs is slower with journalctl that it is
with grep, and not by a small margin.  Journald making the thing you
spend most time doing with logs slower doesn't help it in the
slightest.  But I don't spend a lot of time searching logs, so it
wouldn't stop me from dropping rsyslog.

Get rid of those problems, and dropping rsyslog becomes a no-brainer
for me.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Scott Kitterman-5
In reply to this post by Matt Zagrabelny
On Tuesday, February 4, 2020 9:01:55 PM EST Matt Zagrabelny wrote:

> On Tue, Feb 4, 2020 at 5:15 PM Scott Kitterman <[hidden email]> wrote:
> > On Tuesday, February 4, 2020 5:22:15 PM EST Vincent Bernat wrote:
> > >  ❦  4 février 2020 11:30 -08, Russ Allbery <[hidden email]>:
> > > >> As a heavy user or Rsyslog features I feel that switching default
> > > >> logging system yields no benefits to say the least.
> > > >
> > > > As a heavy user, perhaps you're not the target audience for a default?
> > > > You're going to install rsyslog no matter what, since you know it well
> > > > and
> > > > use it heavily.  The only effect of this change on you will be a
> > > > one-line
> > > > change to whatever you use for configuration management for new
> > > > systems.
> > >
> > > rsyslog even knows how to directly pull logs from the journal, which
> > > gives you access to stuff not logged to syslog (stdout/stderr of service
> > > files, applications logging directly to journal), as well to structured
> > > logs (comm pid, user, unit and more when the service supports journald
> > > directly).
> >
> > For those of us who aren't customizers of Debian's logging function, it'd
> > be nice to have a clearer understanding of what this changes means.
> >
> > Today, when, for example, I want to investigate something email related, I
> > look in /var/log/mail.log.
>
> Random email related journal commands:
>
> journalctl -u postfix
> journalctl -f -u postfix
> journalctl -b -u postfix
>
> the -u is for the unit name. the -b is for since boot. man journalctl
> for details.
Not particularly useful IMO.  In /var/log/mail.log I can see log entries from
all the programs configured to log to the mail facility.  That way I can see
the interaction between them.  On a typical server that is for sending mail I
often need to see log entries from postfix, clamsmtp, and dkimpy-milter
together to understand how a message is (or isn't) making it through the
system.

Of course the fact that I can't use all the tools available to manipulate text
files to follow or analyze logs is problematic.  If I'm using journalctl, how
do I replicate 'tail -f /var/log/mail.loog'?

Note that I'm not asking about some specialized configuration that I've set up.  
All I want to know is how to make it work like Debian works out of the box
today.

Scott K

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Heads up: persistent journal has been enabled in systemd

Vincent Bernat-3
 ❦  5 février 2020 01:01 -05, Scott Kitterman <[hidden email]>:

> Not particularly useful IMO.  In /var/log/mail.log I can see log entries from
> all the programs configured to log to the mail facility.  That way I can see
> the interaction between them.  On a typical server that is for sending mail I
> often need to see log entries from postfix, clamsmtp, and dkimpy-milter
> together to understand how a message is (or isn't) making it through the
> system.
>
> Of course the fact that I can't use all the tools available to manipulate text
> files to follow or analyze logs is problematic.  If I'm using journalctl, how
> do I replicate 'tail -f /var/log/mail.loog'?
You can use:

journalctl -f SYSLOG_FACILITY=2
--
Its name is Public Opinion.  It is held in reverence.  It settles everything.
Some think it is the voice of God.
                -- Mark Twain

signature.asc (847 bytes) Download Attachment
1234 ... 6