Attention: /usr/share/file/magic{,.mime} removal

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Attention: /usr/share/file/magic{,.mime} removal

Daniel Baumann-4
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

file or rather libmagic1 ships its magic files in /usr/share/file/,
namely this used to be:

/usr/share/file/magic
/usr/share/file/magic.mgc
/usr/share/file/magic.mime
/usr/share/file/magic.mime.mgc

where as *.mgc are the binary files which are used by file/libmagic, and
the others are the conlgomerated source files *for informational
purposes only*. The sources have never been used by file for anything,
and nobody shall do this either[0].

However, as of file version 4.24, the format of the sources has changed
in order to compile the mime files from the magics automatically[1].
This means, that if you are using the magic or magic.mime files
directly, your package will break anyway.

Additionally since file 4.24, those packages that need/want to introduce
new magics unknown to file (and for strange reasons are not considering
it to include in the debian file package or upstream), can now do it by
storing their magic snippeds in /usr/share/file/, and call file
- --compile to produce the binary magics file (more on that at a later
point).

In unstable (and testing, soon), this has been avoided by an extra step
of dumping a plain magic file in the old format and including the legacy
copy of magic.mime from file version 4.23[2]. As soon as lenny is
released, these will disappear and you're supposed to eventually convert
your packages.

Probably, I will also fill bug reports before lenny release to affected
packages.

Regards,
Daniel

[0] they *could* change format suddenly, only the library and its
    bindings are safe.

[1] which is a big improvement and means, that the mime entries are
    no longer endlessly lacking behind to catch up with the magics.

[2] this won't be updated to 4.24 though, so really don't use it anymore
    if you want to have recent magics.

- --
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          [hidden email]
Internet:       http://people.panthera-systems.net/~daniel-baumann/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIHF/h+C5cwEsrK54RAiCsAKCgINMg/u9PaufJ2+KHrEoW73qejgCcCG9C
MmMzwiblFWKjQ9nyZbCtJXw=
=UsQm
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Attention: /usr/share/file/magic{,.mime} removal

Bernhard R. Link-2
* Daniel Baumann <[hidden email]> [080503 14:52]:

> file or rather libmagic1 ships its magic files in /usr/share/file/,
> namely this used to be:
>
> /usr/share/file/magic
> /usr/share/file/magic.mgc
> /usr/share/file/magic.mime
> /usr/share/file/magic.mime.mgc
>
> where as *.mgc are the binary files which are used by file/libmagic, and
> the others are the conlgomerated source files *for informational
> purposes only*. The sources have never been used by file for anything,
> and nobody shall do this either[0].

> [0] they *could* change format suddenly, only the library and its
>     bindings are safe.

And where is this documented? (It has a documentation for the format, so
I guess if that was supposed to be "writing" only, that was a canonical
place to state it).

> In unstable (and testing, soon), this has been avoided by an extra step
> of dumping a plain magic file in the old format and including the legacy
> copy of magic.mime from file version 4.23[2]. As soon as lenny is
> released, these will disappear and you're supposed to eventually convert
> your packages.

Thanks for the warning and the compaitibility wrapper. But please also
advertise that a bit wider. Just a mail to debian-devel is hardly enough
when breaking an advertised interface.

> Probably, I will also fill bug reports before lenny release to affected
> packages.

xfm is one of them. Perhaps I will find the time to file a list of
whishlist bugs to extend libmagic so it can be used instead of parsing
the files itself.

Hochachtungsvoll,
        Bernhard R. Link


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Attention: /usr/share/file/magic{,.mime} removal

Bernd Eckenfels
In reply to this post by Daniel Baumann-4
In article <[hidden email]> you wrote:
> where as *.mgc are the binary files which are used by file/libmagic, and
> the others are the conlgomerated source files *for informational
> purposes only*. The sources have never been used by file for anything,
> and nobody shall do this either[0].

So how can a sysadmin add rules?

> However, as of file version 4.24, the format of the sources has changed
> in order to compile the mime files from the magics automatically[1].
> This means, that if you are using the magic or magic.mime files
> directly, your package will break anyway.

So it looks to me that file is recreating the (cached) binary versions from
the "information purpose only" source files, right?

Gruss
Bernd


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Attention: /usr/share/file/magic{,.mime} removal

Daniel Baumann-4
In reply to this post by Bernhard R. Link-2
Bernhard R. Link wrote:
>> [0] they *could* change format suddenly, only the library and its
>>     bindings are safe.
>
> And where is this documented? (It has a documentation for the format, so
> I guess if that was supposed to be "writing" only, that was a canonical
> place to state it).

what do you mean, documentation about the format, documentation about
the recent change, or documentation about to not modify
/usr/share/file/magic* directly?

> Thanks for the warning and the compaitibility wrapper. But please also
> advertise that a bit wider. Just a mail to debian-devel is hardly enough
> when breaking an advertised interface.

err, this is a first attention mail approximately 4 to 6 month before
lenny release..

--
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          [hidden email]
Internet:       http://people.panthera-systems.net/~daniel-baumann/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Attention: /usr/share/file/magic{,.mime} removal

Daniel Baumann-4
In reply to this post by Bernd Eckenfels
Bernd Eckenfels wrote:
> So how can a sysadmin add rules?

just like before, /etc/magic and /etc/magic.mime. this change is only
about packages wrongly using /usr/share/file/magic{,.mime}.

>> However, as of file version 4.24, the format of the sources has changed
>> in order to compile the mime files from the magics automatically[1].
>> This means, that if you are using the magic or magic.mime files
>> directly, your package will break anyway.
>
> So it looks to me that file is recreating the (cached) binary versions from
> the "information purpose only" source files, right?

no, /usr/share/file/magic.mgc and magic.mime.mgc are files own magics,
they are not supposed to be recompiled or touched by anyone after they
have been installed.

however, what i was writing about is that since version 4.24, file is
able to use other binary magic files *in addition*. if you are a package
wanting to ship your own magics or mimes, you compile them at
installation time into /usr/share/file/$package.mgc, which then will be
used by file automatically.

--
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          [hidden email]
Internet:       http://people.panthera-systems.net/~daniel-baumann/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Attention: /usr/share/file/magic{,.mime} removal

Daniel Baumann-4
Daniel Baumann wrote:

>> So it looks to me that file is recreating the (cached) binary versions from
>> the "information purpose only" source files, right?
>
> no, /usr/share/file/magic.mgc and magic.mime.mgc are files own magics,
> they are not supposed to be recompiled or touched by anyone after they
> have been installed.
>
> however, what i was writing about is that since version 4.24, file is
> able to use other binary magic files *in addition*. if you are a package
> wanting to ship your own magics or mimes, you compile them at
> installation time into /usr/share/file/$package.mgc, which then will be
> used by file automatically.

the 'automatic' was refering to the file sources. magics were created
from magic/Magdir/* files; and completely independent of that, there was
another list containing mime entries. Those should have been kept in
sync, but oviously, the were not and constantly lacking behind. Now, the
mime informations are in the same files in magic/Magdir/* as an addition
of the file magic syntax, so mime entries can be created from the same
sources as the magics automatically.

--
Address:        Daniel Baumann, Burgunderstrasse 3, CH-4562 Biberist
Email:          [hidden email]
Internet:       http://people.panthera-systems.net/~daniel-baumann/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Attention: /usr/share/file/magic{,.mime} removal

Bernhard R. Link-2
In reply to this post by Daniel Baumann-4
* Daniel Baumann <[hidden email]> [080503 18:39]:

> Bernhard R. Link wrote:
> >> [0] they *could* change format suddenly, only the library and its
> >>     bindings are safe.
> >
> > And where is this documented? (It has a documentation for the format, so
> > I guess if that was supposed to be "writing" only, that was a canonical
> > place to state it).
>
> what do you mean, documentation about the format, documentation about
> the recent change, or documentation about to not modify
> /usr/share/file/magic* directly?

I mean there is no documentation that the file is not to be used directly,
rather documentation about this file in a way suggesting it is supposed
to be a public interface. (comparing this to the documentation of the
libmagic library, the library looks more like a private interface in
comparision)

> > Thanks for the warning and the compaitibility wrapper. But please also
> > advertise that a bit wider. Just a mail to debian-devel is hardly enough
> > when breaking an advertised interface.
>
> err, this is a first attention mail approximately 4 to 6 month before
> lenny release..

For upstream changes 4 to 6 months is a very short time (especially if
it is not simple changes, but you are requesting to replace whole
subsystems with some library, which will most likely mean to also have
to get upstream changes in some prior unused library).

Hochachtungsvoll,
        Bernhard R. Link


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]