nosh version 1.30

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

nosh version 1.30

Jonathan de Boyne Pollard
The nosh package is now up to version 1.30 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

service bundles
---------------

As usual, there are more service bundles, including for the UWSGI
"Emperor" and the new services in FreeBSD/TrueOS 11 such as ypldap.
There are now services to run Sendmail in the same manner as the
services that run exim.  Note that this is slightly different to the old
FreeBSD division of labour.  There are individually controllable
services for SMTP Submission, SMTP Relay, the Submission queue runner,
and the Relay queue runner.

doco
----

The Guide has been extended with several new chapters, including a
gazetteer of interesting directories, a chapter on log file
post-processing, a chapter on logging security, a chapter on per-user
service management, and some notes for individual services.  The
commands list has moved from the blurb into the Guide, too, as it seems
like something that an administrator might find handy to have available
when there's no Internet connection.

* http://jdebp.eu./Softwares/nosh/guide.html

service management
------------------

There's now a hardlimit chain-loading command, analogous to softlimit.  
The convert-systemd-services utility now makes use of this and permits
setting separate hard and soft limits, or only one or the other, with
settings like LimitOFILE=32:128 and LimitNPROC=:infinity .

There's now a local-reaper chain-loading command, that can turn "local
reaper" status for the current process on or off.  Have a care when
using this, per the note on the manual page.  There is a
LocalReaper=true extension to systemd service units for this.

netlink-datagram-socket-listen is now available on the BSDs for script
compatibility.  It always aborts with an address family error.

There's a new hangup subcommand of system-control, equivalent to the
existing -H option to svc .

enhancements to system-control stop/start/reset and single-shot services
------------------------------------------------------------------------

This is the first big item for 1.30 :

The start and stop subcommands of system-control now operate more
quickly.  Instead of polling once per second, they monitor the
supervise/status files of each service that is in the process of being
started and stopped, with kevent().

In addition, system-control now supports the notion of services that
become ready when their main process has exited, marked with a new flag
file in the service directory.  convert-systemd-units has been modified
to convert "oneshot" services to this, instead of to services that put
all of the run code into the start program.  Thus "oneshot" services
that are running their actual main programs are reported as "running" by
svstat, rather than as "starting".

This takes advantage of the extended status information that
service-manager has been writing to the status file since version 1.28.  
The sharp-eyed may have noticed that in version 1.28 the output of
"svstat"/"system-control status" gained information about the exit
statuses of the start, run, restart, and stop programs. This is what
system-control now uses to detect whether ready-after-run services ran
before they stopped.  (Detection of ready-after-run services that are
running with no processes, because they are "remain" services, can be
and is done with just the daemontools-encore-compatible status information.)

Old-style "oneshot"s will continue to work as before, as of course they
become ready as soon as the run process is spawned, which is after they
have run their programs as part of start.

The benefit of this new style, apart from reporting a running service as
actually "running", which should help with nagios monitoring and the
like, is that "oneshot" services converted from systemd no longer have
to be marked as RemainAfterExit=true in order to avoid a dummy "pause"
process hanging around.  This is the case for old-style "oneshot"
services.  They have to run something in run, after all, and that
something has to keep running in order for the service to be considered
ready and services ordered after it to be unblocked.  A ready-after-run
service, however, unblocks ordered-after services if it has reached the
stopped state via a run, thus puts its programs in run, thus doesn't
have to have a dummy pause process, and can be RemainAfterExit=false
without adding to the process list.

log file management
-------------------

export-to-rsyslog had a bug that caused it to skip old log files (the
@nnnnnnnnnnnnnnnnnnnnnnnn.s ones) in catch-up mode.  This has been
corrected.  There is now a follow-log-directories command that can
substitute for tail -F .  It knows the actual structure of log
directories, operates using one or more cursors like export-to-rsyslog
does, and copes correctly with cyclog/multilog log rotation (which GNU
tail, at least, apparently has problems with when the timing is
particularly wrong on a loaded system).

See also http://jdebp.eu./FGA/do-not-use-logrotate.html

build
-----

More warnings are now turned on with clang++ during the build, and a lot
of the resultant warnings have been eliminated where appropriate.  The
check for eg++ in preference to g++ is now limited to OpenBSD, where (at
least on OpenBSD 5.9) eg++ is still ahead of g++ by a wide margin.

Per-user service management
---------------------------

Changes in per-user service management are the second big item for 1.30 :

The per-user service manager instances are now invoked via userenv, so
all per-user services that you run under nosh service management, D-BUS
servers or otherwise, will have your own HOME, SHELL, and USER set.  
Several per-user daemon softwares were expecting HOME to be set.

To match what the Desktop Bus people are doing, the dbus socket path for
the per-user D-BUS broker has changed from
"/run/user/$USER/dbus/user_bus_socket" to "/run/user/$USER/bus".  In
theory, this is addressable (in D-BUS speak) as "unix:runtime=yes". In
practice, there is no version of D-BUS available on stable/release
FreeBSD, TrueOS, or Debian that understands this address syntax.  So one
still has to use "unix:path=/run/user/$USER/bus".

The Desktop Bus people and the desktop environments people are also
switching from per-login D-BUS brokers to per-user D-BUS brokers. The
nosh toolset has already had this for over a year, since the middle of
2015.  Each real-person user account has an optional per-user service
management service (e.g. user-services@fred).  What is new is that
per-user service bundle areas are now populated with a whole load of
service bundles for real services, many relating to GUI desktop
environments, and the per-user D-BUS broker has moved to there, from
being a system-level service bundle.

The configuration import subsystem creates these new per-user service
bundles in the home directories of individual real users, under
~fred/.config/service-bundles/services/ and
~fred/.config/service-bundles/targets/ (for user fred).  These run
per-user services for a whole load of things, from GNOME editor and
emacs through dconf and KDE Notify to urxvtd and GNOME Terminal.

The configuration import subsystem also sets up a bypass for D-BUS's
broken "bus activation" mechanism, so that instead of attempting to run
these D-BUS servers directly, the D-BUS broker instead tells the nosh
per-user service manager to run them.  This takes the form of a
replacement dbus-daemon-launch-helper, and the per-user D-BUS brokers
now employ a modified configuration file that invokes it.

There's a full explanation of how this all works in the new chapter on
demand-starting user-level Desktop Bus services in the nosh Guide.

Notes:
   * For emacs as a per-user service, you must have a very recent emacs
with its very-late-to-the-party --new-daemon option.
   * GNOME Weather and its interaction with GeoClue2 are only partly
tested, because the versions of them available for the test platforms
were attempting to contact a weather service that the U.S. Government
discontinued in June 2016; and this was hardwired into their code.

Reply | Threaded
Open this post in threaded view
|

nosh version 1.31

Jonathan de Boyne Pollard
The nosh package is now up to version 1.31 .

* http://jdebp.eu./Softwares/nosh/

*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

* http://jdebp.info./Softwares/nosh/

This release fixes a problem with emergency mode that was introduced by
accident in 1.29 .  The emergency-login@console service was not properly
enabled by package installation.  Now it once again is.

There are a number of bug fixes in this release, such as rare corner
cases in how convert-systemd-units generates arguments to pass to sh,
what port the nginx server part of Appcafe binds to when not the
default, the use of setuidgid-fromenv to set more than 1 supplementary
group ID, and making the Makefile in tinydns@* services work with both
BSD and GNU make.  Various service bundles that perform
clean-up-directories actions at bootstrap have been made more difficult
to accidentally re-trigger after bootstrap.

There is also a fair amount of new features:

* The automatically-generated data for tinydns@* services now
encompasses all of the reverse lookup domain names for private/local IP
addresses, so none of the DNS traffic involving such lookups will leak
out of your machine/organization to the rest of Internet.

* The userenv command has gained the ability to (optionally) set a whole
lot more environment variables from the capabilities in /etc/login.conf
and ~/.login_conf .  It now can be used as the
setup-the-user-environment part of a command chain that is designed to
perform the setup of an interactive login session. This is particularly
useful for fixing PCDM, the display manager in TrueOS.

* The pipe command can now arrange to clean up the child process in one
of two ways.  This is made use of in the dnscache service bundles, and
dnscache services no longer contain the perpetual zombie process that
they had in version 1.30 .

* Presets now support wildmat-style character set wildcards. e.g. one
can now write "ttylogin@vc[0-9]-tty" as a service name pattern.

* If you have been using the --verbose option to the start/stop/reset
subcommands of system-control, you'll notice that it now colourizes its
output.  Its output has also been adjusted to more clearly indicate
blocked services and what they are blocked by.

The big item is that there is now a complete set of simple control
groups manipulation commands, the pre-supplied service bundles all make
use of it, and all service bundles created by convert-systemd-units make
use of it.  (All of this is a no-op on FreeBSD/TrueOS and OpenBSD, of
course.)

If you've read the Linux doco, you'll know that control groups do not
require any sort of centralized gatekeeper process, and are a
decentralized system that can be driven with just the echo command.  In
practice, using echo is non-trivial.  The move-to-control-group,
delegate-control-group-to, and set-control-group-knob commands take the
hassle out of working out exactly what to echo where.  They do all of
the hard work of determining what the directory name of the current
control group under /sys/fs/cgroup is, and present a simple system
allowing one to create and navigate to another control group, delegate
control over the current control group (and its subgroups) to an
unprivileged user, and set control group knobs.

The set-control-group-knob utility further illustrates the convenience
functionality over and above a simple echo command. It can calculate a
knob setting as a percentage of another number, handle SI and IEEE/IEC
multiplier suffixes, and translate the device file names that are
(comparatively) convenient for humans into the literal major and minor
device numbers that the Linux control groups API actually operates in
terms of.

There are new chapters in the Guide covering the automatic import of
FreeBSD 9 and PC-BSD Warden jails, how jailing services on
FreeBSD/TrueOS works, and limiting services.  The limiting services
chapter covers both the original Unix resource limits system and Linux
control groups.

Reply | Threaded
Open this post in threaded view
|

nosh version 1.32

Jonathan de Boyne Pollard
The nosh package is now up to version 1.32 .

* http://jdebp.eu./Softwares/nosh/

*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

* http://jdebp.info./Softwares/nosh/

This release fixes two problems with Gentoo Linux (control group version
detection and a problem with mounting API filesystems) that we hashed
out on the Supervision mailing list.  It furthermore contains a change
to the way that convert-systemd-units generates service bundles that
fixes problems with control group setup when the service unit defines a
"slice" for the service or when the service unit is a template. In
furtherance of that there's a new create-control-group command.

Other things in this release include improvements to the (unpackaged) Z
Shell command-line completions, which now display option completion
menus properly; some improvements to the Terminals chapter in the Guide;
fixes to various service bundles that were using shell reserved words
and operators such as "for" and "&&" without explicitly invoking the
shell; additions to userenv for setting DBus and XDG Runtime variables;
and a fix that prevents "system-control reset" from looping indefinitely
when run by an unprivileged user such as "messagebus" that lacks access
to the control/status API.

The major improvement in this release, though, is to console-fb-realizer
on TrueOS.

FreeBSD gives console-fb-realizer uhid device files to use for input
devices, which speak the USB HID report protocol and which
console-fb-realizer has been happy with for a long time.  TrueOS
provides either ums/ukbd devices, which lack various features because
they speak the old sysmouse and atkbd protocols, or ugen devices.  There
are no uhid devices available. console-fb-realizer can now use the ugen
devices.  Moreover, it will detach the ums/ukbd drivers from the ugen
devices using the new detach-kernel-usb-driver command, so that there
aren't two things both attempting to read HID reports.

console-fb-realizer also now correctly sets the keyboard LEDs on both
FreeBSD and TrueOS.

There have been several minor adjustments to the kernel VT sharing parts
of console-fb-realizer, preparatory to splitting the program up into
separate parts for input and output devices, permitting things such as
multiple keyboards each with its own keyboard map and numlock semantics,
in a future release.

Reply | Threaded
Open this post in threaded view
|

nosh version 1.33

Jonathan de Boyne Pollard
The nosh package is now up to version 1.33 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

This has been held back because of work being done by someone else.  I don't
want to steal xyr thunder, so I'll leave the announcement of that work to xem.
 Suffice it to say that it will interest a new group of people.

There are several major improvements in 1.33 .

Packaging
---------

In the version 1.29 announcement I said that the Debian packaging system was
going to be brought into line with the system used for FreeBSD/TrueOS and
OpenBSD.  This is now done.  Debian and the BSDs all now use a similar system
for generating each package manager's package maintenance instructions from an
abstract package description.

==============================================================
=========== IMPORTANT UPGRADE NOTE FOR Debian: ===============
==============================================================

An important consequence of the aforementioned is that the semantics of the
nosh-bundles package have changed. In earlier versions, the various nosh-run-*
packages were how one set services running, except for a small rump set of
services that were set up by the nosh-bundles package.

This is now no longer the case. The nosh-bundles package now presets and starts
no services at all. *All* running of services must be achieved with the
nosh-run-* packages or some other sets of scripts and presets.

To this end, there are now two new packages, nosh-run-debian-desktop-base and
nosh-run-debian-server-base. These parallel the
nosh-run-{freebsd,trueos}-{desktop,server}-base packages already available since
1.29 for FreeBSD/TrueOS. You must install, for a working fully-nosh-managed
system, exactly one of the nosh-run-debian-{desktop,server}-base packages.

If you are running nosh service management under systemd, you can of course run
as many or as few services under the nosh service manager as you care to switch
over from systemd. But if you are running a fully-nosh-managed system these
packages will arrange to run the various fundamentals that one pretty much
cannot do without, such as mounting/unmounting volumes, running
udev/eudev/vdev/mdev, binfmt loading, and initializing the PRNG.

Log service account names
-------------------------

The naming scheme used for the user accounts for dedicated log service users has
changed.  Installing the new nosh-bundles package should automatically rename
all existing log service accounts to use the new scheme.

The new naming scheme is slightly more compact, and copes better with services
that have things like underscores and plus characters (e.g. powerd++) in their
names.

As an ancillary to this, system-control now has an "escape" subcommand which can
be (and indeed is) used in scripts to perform the escaping transformations.

More packages
-------------

There are now four more -shims packages, for commands whose names conflict with
commands from other packages: nosh-kbd-shims, nosh-bsd-shims, nosh-core-shims,
and nosh-execline-shims.

nosh-kbd-shims, for example, contains a chvt shim that is an alias for the (also
new) console-multiplexor-control command; with it, and suitable privileges to
access the virtual terminal's input queue, one can switch between multiplexed
user-space virtual terminals in much the same way as the old chvt command does
with kernel virtual terminals.

The Z Shell command-line completion for the various commands in the toolset
(system-control, svcadm, shutdown, svstat, and so forth), which has been
available to the people building from source for a while, is now also available
as a binary package.

Configuration import
--------------------

ldconfig on TrueOS is now properly handled.  In particular, the external
configuration import subsystem now correctly pulls in and converts all of the
ldconfig directories.  (TrueOS has a lot more things that require ldconfig
support than stock FreeBSD does.)

The configuration import subsystem also now handles instances of Percona server,
alongside MySQL and MariaDB.  Moreover, these are now handled by the same set of
service bundles, which always produce service bundles named mysql@*.  MySQL
version 5.7 or later is now assumed.

The configuration import subsystem now automatically generates OpenVPN service
bundles based upon the current OpenVPN configuration.

=======================
==== CAVE: OpenVPN ====
=======================

The upgrade process attempts to remove the old hardwired openvpn@server and
openvpn@client service bundles.  However, you might encounter remnants of these
service bundles lying around in /var/sv that you will find that you need to
clean up by hand.

GOPHER
------

To accompany the new gopherd server in djbwares 5, there is a gopher6d service
bundle that runs it, serving up the same static files area as http6d, https6d,
and ftp4d do.

The FreeBSD, OpenBSD, and Debian package repositories can now be browsed with
GOPHER.  This is gopherd in action.  On the server side, generating the
index.gopher files is a fairly humdrum exercise in the use of redo (to
regenerate the indexes only when the directory contents change) and printf (to
construct the GOPHER format menus).

UCSPI-UNIX
----------

Two new UCSPI tools have been added to enable UCSPI-UNIX servers to listen on
and accept connections on AF_UNIX sequential packet sockets.  udevd is one such
server, and it is now handed its listening socket at startup rather than
expected to open its own.

Reply | Threaded
Open this post in threaded view
|

nosh version 1.34

Jonathan de Boyne Pollard
The nosh package is now up to version 1.34 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

Once again, there are a few more service bundles.  The most interesting ones in
this version are perhaps the finish-update and finish-install targets, designed
to be invoked the first bootstrap after an update or install has been done, and
the users target, which is used to auto-start per-user subsystems at bootstrap.
 Several NFS service bundles are now common across operating systems.  And the
OpenVPN service bundles are now split into separate client and server services.

Several minor bugs have been fixed here and there: a duplicated newline in
line-banner that was throwing off publicfile FTP service; a problem with
recordio on FreeBSD/TrueOS; and a problem with attempts to use slashes in
environment variables in service bundle environment directories.

The user-space virtual terminal emulator now implements the Xterm extensions to
DECSCUSR, and the framebuffer realizer can display the resultant cursor shapes.
 This can be made use of by programs such as Neovim.

There are now separate service bundles and nosh-run- packages for running eudev
and systemd-udev, because the two are now significantly divergent.

The various utilities for changing the process environment no longer use the GNU
C library/BSD C library functions for doing so, and so no longer suffer from the
concomitant memory leaks that their manual pages used to warn about.

The convert-systemd-units tool has been slightly enhanced, for the benefit of a
fix that has been made to the per-user gpg-agent service.

The external configuration imports system has been extended.  It now deals with
importing the hostname configuration value, taking that responsibility away from
and simplifying the set-dynamic-hostname utility.  It now imports various Debian
and other kernel virtual terminal settings, from /etc/kbd/config,
/etc/default/console-setup, and /etc/vconsole.conf .  And network configuration
import now can set up services for both dhcpcd and dhclient.
 /etc/system-control/convert/rc.conf now contains more settings on Linux
operating systems as a result, including dhclient_program.

Reply | Threaded
Open this post in threaded view
|

nosh version 1.35

Jonathan de Boyne Pollard

The nosh package is now up to version 1.35 .

Networking

As I mentioned a week or so ago, the external configuration import subsystem now converts a Debian-style /etc/network/interfaces configuration file, via rc.conf settings, into the native networking subsystem.

There is also a whole new Networking chapter in the nosh Guide, which explains this and several other things, including how Plug and Play integration interoperates with the networking services and what the native networking subsystem encompasses, to the level of what service does what and to what purpose.

Work on the Plug and Play integration is on-going, and I hope to have yet more for this, and indeed for other parts of the networking subsystem, in version 1.36.

Packages

There are some Debian packages that declare that they need the logrotate package, even though they do not when run under nosh service management.  For their benefit there is now a nosh-logrotate-shims Debian package that is simply a dummy package that satisfies this need without setting up a spurious and unnecessary logrotate system.

Service bundles

There are a few more service bundles, including ones for sysstat and elasticsearch.  The existing service bundles for things such as unbound, clamav, and freshclam have been augmented and fixed in response to user feedback.  And a bug that incorrectly resulted in the ldconfig service being disabled has been fixed.

The dbus services, the system-wide one and the per-user one(s), have been renamed to dbus-daemon.  This is because of the existence of a dbus-broker service bundle.  This is a placeholder for if the dbus-broker people ever fix it so that it works.  dbus-broker does not provide a working system right now.  It is currently not possible to substitute dbus-broker for dbus-daemon on non-systemd systems, because dbus-broker is very tightly tied in to systemd's idiosyncratic D-Bus control interface.  It only speaks the systemd-specific protocol, and knows no other way of stopping and starting services, not even the service command.  (In contrast dbus-daemon can still be configured to demand-start services using simple service management commands.)

Reply | Threaded
Open this post in threaded view
|

nosh version 1.36

Jonathan de Boyne Pollard

The nosh package is now up to version 1.36 .

More Java tools

This release comes with the find-default-jvm and find-matching-jvm tools, which will set up the JAVA_HOME environment variable to point to a default/matching JVM directory, using the FreeBSD/TrueOS and Debian conventions for locating JVM directories.  To match these, convert-systemd-units now recognizes JVMDefault, JVMVersion, JVMOperatingSystem, and JVMManufacturer extensions to the systemd unit file format.

Tool improvements

convert-systemd-units now recognizes a MachineEnvironment extension to the systemd unit file format, which controls the generation of an invocation of machineenv.  It also now recognizes and translates RDMAHCAHandlesMax and RDMAHCAObjectsMax settings.

The unshare command now has flags for specifying process ID and user ID namespaces on Linux.

The setup-machine-id command now correctly falls back to the old D-Bus files on FreeBSD, which it had not been doing because of a bug.

New system management features

In support of an initiative by Warner Losh, there is support for power cycling via hardware and a kernel that support it.

The system manager treats SIGRTMIN+6, unused in the systemd system, as a request to invoke a new powercycle service bundle; and SIGRTMIN+16, similarly unused, as the underlying actual powercycle request, which it translates to either RB_POWERCYCLE if it is present in the C library headers, or RB_AUTOBOOT if it is not. There is a new system-control powercycle subcommand, which defaults to sending these signals.

Note that the binary packages are currently built on a system that lacks RB_POWERCYCLE in the C library.

The compatibility shutdown, reboot, halt, and poweroff commands all now sport a new -c/--powercycle option.  There are new fastpowercycle and powercycle commands.  The system-control init subcommand now sports a new c/C argument, by analogy to h/H. And this is of course thus reflected automatically in the compatibility telinit command and the initctl-read server.

Service bundles

Fixing an oversight in 1.35, the per-user dbus services are now renamed to dbus-daemon too.

There are a few more service bundles, including ones for jenkins, apacheds, udisks2, and ndppd.

The linux-utmp service bundle has been retired, in favour of a unified utx service bundle, which was previously FreeBSD-only, that operates across platforms.  In support of this, there is a new login-update-utmpx command, and a new freebsd-shims package that aliases that to the utx command on non-FreeBSD platforms.

Reply | Threaded
Open this post in threaded view
|

nosh version 1.37

Jonathan de Boyne Pollard
In reply to this post by Jonathan de Boyne Pollard

The nosh package is now up to version 1.37 .

Some of the changes in this release are works in progress, that you will see fully realized in version 1.38 or later.

Changes include:

  • There is a new chapter in the nosh Guide for those wishing to make packages and ports of other softwares, or add service bundle support to existing packages and ports.

  • The external formats configuration import subsystem has been reorganized a bit.

    • Nothing uses the JAVA_HOME import system any more, where service bundles explicitly have their JAVA_HOME variables set by configuration import, although it is retained.  All service bundles instead use the find-matching-jvm mechanism to auto-detect a JVM matching their chosen criteria at start time.

    • The per-user services import is now in two parts.  System-wide import sets up a  $HOME/.config/service-bundles/convert/ subdirectory for each (real user) user account; and each user can then use that, which contains a subordinate per-user configuration import mechanism, to set up imported per-user service bundles for things.

    • Per-user service source files for Desktop Bus and other services are now in their own subdirectory, as are converted keyboard maps for the userspace virtual terminals.
  • static-networking external format configuration import has been enhanced to set up snort@interface services and to handle ipv6_cpe_wanif and ipv6_activate_all_interfaces from /etc/rc.conf.

  • There is a new make-read-only-fs chain loading tool that is a placeholder for now.  It is used in some service bundles generated by the convert-systemd-units tool, which now recognizes and converts CPUAffinity, ProtectHome, ProtectSystem, ReadWriteDirectories, ReadOnlyPaths, and InaccessiblePaths settings.

  • Per-user management has been augmented, finally fixing the problem of system-control locating the per-user manager by giving the per-user manager an optional listening FIFO open file descriptor, which it uses to listen for user-wide state change commands.  system-control --user halt/normal/sysinit/&c. now send commands via this FIFO, and each user's user-services@username service bundle now uses fifo-listen to set up the FIFO and creates the per-user-manager/ subdirectory in /run/user.

  • There are some more service bundles in the collection that comes with the toolset: clickhouse-server, hue, udhcpc-log, minissdpd, rtkit-daemon, accounts-daemon, gdm3, speech-dispatcher, gdomap, blueman-mechanism, and sysvipc.

  • The per-user configuration import now recognizes and sets up per-user service bundles for a whole lot more per-user services.

  • On FreeBSD/TrueOS systems setup-machine-id now writes /usr/local/etc/machine-id.

  • The userspace virtual terminal services, the multiplexor and the terminal emulators, no longer run under the aegis of the daemon system account.  Rather, they now have their own dedicated accounts under whose aegides they run.  To go with that, there is now a user-vt-realizer group to which users can be added to grant them realizer (i.e. front-end I/O) access to the system-wide userspace virtual terminals.

  • A common build problem across several toolsets that occurs if one has set a CDPATH, has been fixed.  Various tweaks have also been made to make life easier for Archnosh and ports to other operating systems.

Reply | Threaded
Open this post in threaded view
|

nosh version 1.39

Jonathan de Boyne Pollard
The nosh package is now up to version 1.39 .

* http://jdebp.eu./Softwares/nosh/

*
https://www.freebsd.org/news/status/report-2017-07-2017-09.html#The-nosh-Project 


* http://jdebp.info./Softwares/nosh/

I missed announcing 1.38, so this announcement will cover both versions.

These versions see a major addition to the user-space virtual terminal
subsystem, various other changes in several areas, the completion of
some items mentioned as placeholders in version 1.37, and some bug fixes.


Completed placeholders
======================

make-read-only-fs is now fully implemented, and is no longer a placeholder.


More service bundles
====================

There are several more additions to the set of service bundles supplied
with the toolset: connman, ofono, dundee, cntlm, minidlna, powertop,
alsa-state, alsa-restore, unattended-upgrade-shutdown, apt-daily-update,
apt-daily-upgrade, LCDd, phpsessionclean, tinysshd, watchman, rngd,
isnsd, isnsdd, usbmux, and VBoxBalloonCtrl.  atd is now a Linux-only
service, with the BSDs now having an atrun service.


More packages
=============

The new nosh-run-bcron, nosh-bcron-as-cron-shims, nosh-debian-crontab,
and nosh-debian-crontab-anacron packages deal in running the services
and providing the data files for various cron toolsets.  The former two
deal in bcron, running its services and providing the crontab command as
an alias for bcrontab; and the latter two (only available for Linux
operating systems) deal in Debian's /etc/crontab file.

The new nosh-openrc-shims package contains shims for OpenRC's rc-service
and rc-update commands.  And the new nosh-run-via-open-rc package
contains OpenRC scripts for running the service manager.

The new nosh-linux-shims package contains shims for commands to be found
in the non-portable util-linux toolset, such as setterm (more on which
later).

The Debian desktop and server base -run packages no longer preset ntpd
and openntpd, on the grounds that a range of such services exist and
these are not necessarily the installed softwares.


More tools
==========

New commands include getuidgid, userenv-fromenv, setgid-fromenv, envgid,
printenv, setlogin, console-decode-ecma48, console-control-sequence,
console-flat-table-viewer, console-input-method, and
local-stream-socket-connect.

The userenv command is now a combination of two of these new commands,
getuidgid and userenv-fromenv.  It has also gained options for not
setting SHELL and USER/LOGNAME.

setlogin sets the login account that is associated with a kernel
session, as returned by the logname command.

printenv is roughly equivalent to the conventional tool of the same
name, except that it is a nosh/exec built-in command and that it
supports several forms of output (including properly quoted rc.conf
form, NUL-terminated form, and envdir form) in addition to the
conventional human-readable form.  This built-in command makes a common
idiom easier.  When combining clearenv, read-conf/envdir, and printenv
to read a configuration setting, before the advent of the built-in
command one had to employ `command -v printenv` (because clearenv unsets
PATH).  Now one can invoke it as simply printenv.

One common use of this idiom is by the toolset's own build system and by
the external configuration import subsystem, to read things like the
amalgamated /etc/system-control/convert/rc.conf and an os_version file.  
Further to this, the amalgamated rc.conf now has an os_version setting
on Linux operating systems, consolidating the code for obtaining that in
one place.

console-flat-table-viewer is a full-screen TUI viewer for various sorts
of common flat database tables.  It decodes the vis(3) encoding that is
employed in various FreeBSD system tables.  It also handles tables that
use the standard ASCII US, RS, GS, and FS characters.  File separators
permit a form of continual update and redisplay if used in combination
with pipes.

local-stream-socket-connect is the AF_LOCAL socket equivalent of
tcp-socket-connect.


Improvements to existing tools and bug fixes
============================================

The Z shell completions now function better, and now cover a lot more of
the commands in the toolset.

systemd service unit conversion has been modified to make use of the new
environment commands.  The conversion tool in particular makes use of
these when converting per-user Desktop Bus services.  The
EnvironmentUser extension has been replaced by an EnviromnentUserOnly
extension, so that User and Group are consistently the sources of the
user account and primary group. Additional settings now supported by
convert-systemd-units include RuntimeDirectoryGroup,
RuntimeDirectoryPreserve, WantsMountsFor, AfterMountsFor, and
RequiresMountsFor.

convert-systemd-units now also supports %T, %V, and %E expansions and
snippets files.

By analogy to \S, the login-banner command now also recognizes the \N
sequence.

The external configuration import subsystem now generates per-user
service bundles that import user-wide environment variables from a
${HOME}/config/service-bundles/common pseudo-bundle, allowing users to
maintain a single environment directory that affects all per-user
services.  One can thus use rcctl or system-control set-service-env
against this common pseudo-bundle to set environment variables in all
per-user services.

Other improvements to external configuration import include the
automatic generation of service bundles for dbus services, at both the
system-wide and per-user levels.

The N and P actions of console-multiplexor-control when applied to
kernel virtual terminals now work properly.

The external configuration input subsystem now imports defaultrouter
properly from the amalgamated rc.conf.

The cleanX service no longer generates incorrect symbolic links in /tmp.

ttylogin services on user-space virtual terminals were sometimes
(depending from exactly what order things ran in) causing a loop because
of vhangup().  This has been fixed.

And a spelling error in the names of the UNIXREMOTEEUID and
UNIXREMOTEEGID UCSPI-UNIX environment variables has been fixed.


Input methods
=============

* http://jdebp.eu./Softwares/nosh/japanese-input-methods.html

Providing a "front-end processor" for running "input methods" has been a
to-do item on the roadmap for a long time.  It has always been a goal to
provide more than just U.S.-centric mechanisms in user-space virtual
terminals, and this already ranges from allowing one to use fonts with
large Unicode glyph repertoires through to providing the full ISO 9995-3
common secondary keyboard group.  The problems with input method support
were that all of the existing systems that I could tie into required X11
servers, X11 libraries, or direct low-level access to the framebuffer.

Then I discovered OpenVanilla.

The .cin file mechanism is table-based, does not involve plug-ins
needing direct drawing access to the framebuffer, and is common across
that and at least 8 different other tools.  It is now common to
user-space virtual terminals, too.  Just as one can take a SCO Unix or
FreeBSD kbdmap file and (after conversion) use it with user-space
virtual terminals, one can take other people's existing .cin files and
use them.

Other people have written quite a lot of them, moreover.  The blurb page
on Japanese input methods hyperlinked earlier focuses on Japanese, but
there is a wide range of .cin files available from plenty of sources,
from Hangeul Jamo through Array40 to Esperanto.

A new console-input-method command is at the heart of new input method
services, which plumb in to user-space virtual terminals in between the
terminal emulator(s) and the realizer(s).  The pre-supplied "head0"
user-space virtual terminal now has an input method service plumbed in.  
The nosh Guide documents input method front-end processors.  And a new
console-input-method-control command can be used to control them in a
few basic ways.

In line with the philosophy of adopting and adapting existing ideas,
visible from the aforementioned keyboard map and .cin files as well as
the whole building upon the architecture of daemontools notion, the
input method front-end processor provides various control key chords and
function keys that are roughly compatible with other systems, such as
Microsoft's Japanese IME in Windows and the OSF/1 IMLIB.  One can, of
course, drive it with the extra keys that are dedicated to the purpose
on a JIS 106/109-key keyboard or a Korean 103/106-key keyboard.  There
should be not too much change required to existing typing habits.


Other terminal improvements
===========================

In addition to the aforementioned documentation of front-end processors,
the Guide also now documents how "head0" works (although this is just
one way in which one can set up virtual terminals) and has a new chapter
on some of the choices of fonts, keyboard maps, and input methods that
are available from various sources for user-space virtual terminals.

(In a separate project, I have been working on fixes to FreeBSD's
vtfontcvt that permit it to convert Ubuntu Monospace, which as the Guide
documents it currently cannot handle.)

* http://jdebp.eu./Proposals/linux-kvt-manual-pages.html

There are new manual entries for TERM(7), linux-vt(4), linux-console(4),
and TerminalCapabilities(3).

No utility other than console-ncurses-realizer uses NCurses and terminfo
any more.  They have flaws in how they model real terminals and terminal
emulators.  All other TUI tools are built around a different library,
that decodes terminal input using a proper full UTF-8 ECMA-48 state
machine rather than by limited and faulty pattern matching, and that is
geared towards primarily ECMA-48 and ITU T.416 output (with various
common DEC private extensions).  This enables better handling of cursor,
editing, and calculator keypad keys with modifiers; full recognition of
"application" and "normal" modes on the cursor and calculator keypads;
full use of 24-bit colour where the terminal supports it; and use of the
DEC Locator or XTerm mouse.

One such is a new realizer, console-termios-realizer, another
realize-a-terminal-on-a-terminal realizer that is more capable than
console-ncurses-realizer.

console-fb-realizer now has better fallback behaviour in the event of
not being supplied a keyboard map, which has been a fairly common
configuration error.  It now falls back to the U.S. English
International keyboard map, rather than to an empty one where no keys do
anything.

Other improvements lie in extended keys, on the cursor and calculator
keypads.  console-terminal-emulator now fully supports individually
switching these two keypads between "application" and "normal" modes,
and fully supports separate control sequences sent by keys for each
mode.  To align with this, the "fkey49" to "fkey61" actions in SCO
Unix/FreeBSD keyboard maps have changed in console-convert-kbdmap to
distinguish the cursor keypad from the calculator keypad, and
console-fb-realizer collaborates in ensuring that the terminal
emulator(s) receive as much modifier state information accompanying such
keys as is appropriate.

The keyboard map files themselves now follow a new naming convention.  
The external configuration import subsystem will not delete any old
keyboard map files that you might still be using in a running realizer
service, so it is left to you to clean them up. The new naming
convention allows for, say, different "Japanese" maps to be used with a
101/104-key keyboard and a 106/109-key keyboard.

The user-space virtual terminal subsystem now also has the notion of
accelerator key input messages, i.e. alphanumeric key input with the ALT
key modifier.  These are recognized by the realizers and the terminal
emulator generates escape sequences for them.  Note that these escape
sequences, which are neither ECMA-48 nor DEC VT/SCO standard, conflict
with ECMA-48 7-bit control aliases.

console-decode-ecma48 is a debugging aid for text sent to/from
terminals.  It decodes a sequence of ECMA-48 characters, control
characters, escape sequences, and control sequences into a
human-readable representation.  It can handle both ECMA-48 output (as
sent to terminals) and ECMA-48 input (as received from terminals), and
recognizes as an extension various common additional control sequences,
such as the control sequences for function keys generated by the SCO
Unix console, and a bunch of DEC VT private control sequences.

* https://unix.stackexchange.com/a/491883/5132

console-control-sequence provides a human-readable way of emitting
common ECMA-48, DEC VT, AIXterm, and XTerm control sequences.  It is
aliased as setterm and is to a large extent a workalike for the setterm
from the util-linux package, except that it is portable where util-linux
is not.  It is both portable to other operating systems, and portable to
other terminals.  It does not implement the few things that util-linux
setterm does that are specific to Linux and to Linux's built-in teminal
emulator, but conversely it implements quite a number of standard
ECMA-48 and DEC VT control sequences that the util-linux setterm does
not.  These include turning the calculator keypad to/from application
mode; switching to/from the XTerm alternate screen buffer; turning DEC
Locator reports on/off; turning XTerm mouse reports on/off; setting the
cursor shape; changing the mappings of the Delete and Backspace keys;
using the strikethrough, italic, and invisible attributes; turning
background colour erase on/off; setting the underline type; indirect
8-bit colour; direct 24-bit colour; and DEC VT soft reset.

vc-reset-tty and console-resize are now implemented by invoking
console-control-sequence, removing the hardwiring of control sequences
from both.

console-terminal-emulator now recognizes a few additional control
sequences, such as DECST8C (used by console-control-sequence to set
regular tabstops if the interval length is 8).  It also has tabstops at
8 column intervals in its reset state.  Both the FreeBSD and Linux
kernel virtual terminals do this, even though real DEC VTs do not, so
user-space virtual terminals do too.

The --keep-term option to clearenv now retains a few more
terminal-related environment variables.


===========
This way up

And finally, a feature that you will not want.

* https://unix.stackexchange.com/a/465166/5132

Over many years, people have regularly asked for a terminal emulator
where the line progression goes from bottom to top.  The idea that they
have is that they will find this easier than top to bottom.  In response
to a recent question in the same vein on Stack Exchange, I have given
console-fb-realizer, console-ncurses-realizer, and
console-termio-realizer the ability to realize displays with the line
progression reversed.

Having used this, it seems to me that people only want this because they
have not tried it.  Trying it makes a convincing case for not using it.  
So I have retained this feature in order that, at long last, there is an
actual terminal system with this feature that people can try, and come
to the same realization.  (-:

Reply | Threaded
Open this post in threaded view
|

nosh version 1.40

Jonathan de Boyne Pollard

The nosh package is now up to version 1.40 .

This version sees changes to the doco, improvements to network configuration, and a change to machine ID generation.


FreeBSD binaries

================

I plan for this to be the last release with binaries built on FreeBSD 10.  I am going to upgrade the build machine.


ifconfig

========

There is now an ifconfig command in the toolset, with a command-line interface and output similar to the FreeBSD ifconfig.  It is primarily intended for use on non-FreeBSD systems, to provide a FreeBSD-like ifconfig where one does not have the actual FreeBSD tool.  The ifconfig@* services generated by the external configuration import subsystem make use of it on Linux operating systems, allowing the services themselves to be pretty much the same across platforms.

It handles multiple (unlabelled) addresses per interface and both IP version 6 and IP version 4 addressing, which are two of the long-standing complaints against the old ifconfig programs from GNU inetutils and NET-3 net-tools.  It has no notion of overwriting a single "primary" address.  It has a broadcast1 flag for calculating the broadcast address from the prefix length and address.  It prefers the new (since 1993) notation for IP version 4 network masks.  It can do the FreeBSD style of EUI-64 address assignment for IP version 6 with an eui64 flag.

And it colours its output if writing to a terminal.  (-:


Machine ID generation

=====================

FreeBSD from 2007 onwards used the SMBIOS system UUID from the machine firmware as a fallback source for a machine ID.  setup-machine-id prior to this release of the toolset would do the same for compatibility.  This has now been removed from setup-machine-id.  The privacy problems that it entails have turned out to outweigh what little utility it had.

Systems that would have fallen back upon the SMBIOS system UUID will now fall back to creating UUIDs using the C library.  Note that the FreeBSD C library still uses MAC addresses to create UUIDs.  The OpenBSD and GNU C libraries use CSPRNGs.

There is also now an erase-machine-id command that resets all of the machine ID storage locations set by setup-machine-id to a nil UUID.  The machine-id service now calls erase-machine-id at shutdown.

Thus: Machine IDs (when using the supplied service bundles) now have a lifetime from bootstrap to shutdown, will not persist across reboots, do not reveal the SMBIOS system UUID and are not constant and correlatable because of it even when explicitly wiped, and can still reveal MAC addresses on FreeBSD.

The new machine-id(7) manual page lists some of the known users of machine IDs, explains where machine IDs are stored, and gives some of the history of machine IDs.


Square mode

===========

Square mode is now switchable in console-terminal-emulator, using DEC Private Mode 1369.  console-control-sequence has a --square option for changing it.


Other tools

===========

ucspi-socket-rules-check has gained the ability to check uid/self/ and gid/self/ subdirectories when handling UCSPI-UNIX connections.


Doco

====

The Guide now includes the original command manuals, written in DocBook XML.  These are directly readable using a GUI WWW browser and the supplied stylesheet.  The conversions to HTML are still supplied, but reading the original DocBook XML format is better.

TUI WWW browsers such as lynx cannot read DocBook XML.  Their deficiency has inspired a new console-docbook-xml-viewer tool that parses and displays the manual pages with a simple full-screen interface on a terminal.  This can of course display other DocBook XML manual pages as well.


External configuration import improvements

==========================================

The external configuration import subsystem now allows various extensions in a /etc/network/interfaces file, including ipv4ll stanzas (which will cause avahi-autoipd services to be set up) and eui64 stanzas.  A "broadcast +" setting is now transformed into the aforementioned broadcast1 flag for ifconfig.

It also now once more treats false for the login shell as signifying a non-personal user account.


More service bundles

====================

There are a few more service bundles in this release, including ones for Laurent Bercot's mdev, for two-ftp, and for NcFTPd.