Bug#921840: metar: Parsing metar record fails due to changes in NOAA API

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#921840: metar: Parsing metar record fails due to changes in NOAA API

Sven Paulus
Package: metar
Version: 20061030.1-2.2+b1
Severity: important
Tags: patch

Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

* What led up to the situation?

On February 4th the "metar" tool stopped working.
At this point NOAA switched responding on their HTTP URL with metar data
to returning a redirect to the HTTPS variante of the URL.

This can be seen when stracing the command:

sendto(3, "GET /data/observations/metar/stations/EDDH.TXT HTTP/1.1\r\nHost: tgftp.nws.noaa.gov\r\nAccept: */*\r\n\r\n", 98, MSG_NOSIGNAL, NULL, 0) = 98
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=3, events=POLLIN}], 1, 103)   = 1 ([{fd=3, revents=POLLIN}])
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 1 ([{fd=3, revents=POLLIN|POLLRDNORM}])
recvfrom(3, "HTTP/1.1 302 Object Moved\r\nLocation: https://tgftp.nws.noaa.gov/data/observations/metar/stations/EDDH.TXT\r\nContent-Type: text/html\r\nCache-Control: private\r\nConnection: close\r\n\r\n<head><body> This object may be found <a HREF=\"https://tgftp.nws.noaa.gov/data/observations/metar/stations/EDDH.TXT\">here</a> </body>", 16384, 0, NULL, NULL) = 310

The resulting output is currently - independent of the given ICAO code:

$ metar KJFK
METAR pattern not found in NOAA data.

Since metar is already using the HTTPS capable curl library, a fix is
easy:

--- src/metar.h.orig 2019-02-09 11:42:21.266330785 +0100
+++ src/metar.h 2019-02-09 11:39:30.051454153 +0100
@@ -24,7 +24,7 @@
 #define  METAR_MAXSIZE 512

 /* where to fetch reports */
-#define  METARURL "http://tgftp.nws.noaa.gov/data/observations/metar/stations"
+#define  METARURL "https://tgftp.nws.noaa.gov/data/observations/metar/stations"

 /* clouds */
 typedef struct {

Or - for the debian package - by patching the patch:

--- debian/patches/fix-metarurl.orig 2019-02-09 11:44:46.832712464 +0100
+++ debian/patches/fix-metarurl 2019-02-09 11:44:59.796579902 +0100
@@ -18,7 +18,7 @@

  /* where to fetch reports */
 -#define  METARURL "http://weather.noaa.gov/pub/data/observations/metar/stations"
-+#define  METARURL "http://tgftp.nws.noaa.gov/data/observations/metar/stations"
++#define  METARURL "https://tgftp.nws.noaa.gov/data/observations/metar/stations"

  /* clouds */
  typedef struct {

In debian/control the library dependencies have to be upgraded (at least
for Debian testing):

--- debian/control.orig 2019-02-09 11:47:43.335228481 +0100
+++ debian/control 2019-02-09 11:47:55.223124936 +0100
@@ -3,7 +3,7 @@
 Priority: optional
 Maintainer: Kees Leune <[hidden email]>
 Standards-Version: 3.9.8
-Build-Depends: libcurl3-gnutls-dev, dh-autoreconf, debhelper (>= 9)
+Build-Depends: libcurl4-openssl-dev, dh-autoreconf, debhelper (>= 9)

 Package: metar
 Architecture: any


Regards,

Sven


*** End of the template - remove these template lines ***


-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (650, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.18.0-3-amd64 (SMP w/12 CPU cores)
Locale: LANG=C, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages metar depends on:
ii  libc6            2.28-5
ii  libcurl3-gnutls  7.63.0-1

metar recommends no packages.

metar suggests no packages.

-- debconf-show failed

Reply | Threaded
Open this post in threaded view
|

Bug#921840: metar: Parsing metar record fails due to changes in NOAA API

Peter Palfrader
Kees, ping!

On Sat, 09 Feb 2019, Sven Paulus wrote:

> * What led up to the situation?
>
> On February 4th the "metar" tool stopped working.
> At this point NOAA switched responding on their HTTP URL with metar data
> to returning a redirect to the HTTPS variante of the URL.


weasel@orinoco:~$ metar lows
METAR pattern not found in NOAA data.
weasel@orinoco:~$ METARURL=https://tgftp.nws.noaa.gov/data/observations/metar/stations/ metar lows
LOWS 271220Z VRB03KT CAVOK 15/M00 Q1027 NOSIG

Cheers,
--
                            |  .''`.       ** Debian **
      Peter Palfrader       | : :' :      The  universal
 https://www.palfrader.org/ | `. `'      Operating System
                            |   `-    https://www.debian.org/

Reply | Threaded
Open this post in threaded view
|

Bug#921840: Upload necessary before March 2nd (Buster hard freeze), Bug #921840 NOAA URL -> https

Daniel Lange-2
In reply to this post by Sven Paulus
Hoi Kees,

could you please upload a new version of Metar before March 2nd as the
Buster hard freeze is coming up March 12 and the current delay is 10
days for packages to transition to testing.

Bug #921840 is the related Debian bug.

If you currently don't have the time, I can NMU but you have everything
ready in your Github repo already.

Vriendelijke groet,
Daniel

Reply | Threaded
Open this post in threaded view
|

Bug#921840: Fwd: Re: Upload necessary before March 2nd (Buster hard freeze), Bug #921840 NOAA URL -> https

Joost van Baal-Ilić-236
In reply to this post by Sven Paulus
On Thu, Feb 28, 2019 at 12:17:52PM +0100, Daniel Lange wrote:
> Am 28.02.19 um 12:13 schrieb Joost van Baal-Ilić:
> >Yes, I could invest one hour in getting this solved.
>
> Dank je wel!

For the record, upstream git is at https://github.com/keesL/metar .  It has a
debian/ directory too, updated.

Kees: could you prepare a signed .dsc + .debian.tar.xz + .orig.tar.xz ?

BTW, version number should be 20190227.1-2, not 20190227.1-1.1 as in:

 metar (20190227.1-1.1) unstable; urgency=medium
  * Updated default URL to NOAAs new https URL
 -- Kees Leune <[hidden email]> Wed, 27 Feb 2019 19:07:51 -0500

Dank je!  Groeten,

Joost

Reply | Threaded
Open this post in threaded view
|

Bug#921840: Fwd: Re: Upload necessary before March 2nd (Buster hard freeze), Bug #921840 NOAA URL -> https

Joost van Baal-Ilić-77
In reply to this post by Sven Paulus
Hoi Kees,

( @ bugs.debian.org: sorry for speaking dutch here )

Of ben je anders van plan een .tar.gz release te maken?
Volgens je eigen instructies gaat dat met"

0. Assign a new version number in VERSION.m4, commit, push
1. Clone the repository from github
2. Remove .git/
3. rename the repository directory as metar-newversion-1
4. create a new tarball
        tar c --exclude debian -zf metar_newversion-1.origin.tar.gz


Kun je die .tar.gz dan op github in "releases" zetten?  Of zet hem
anders ergens op https://leune.org/ ter download, dat kan ook.

Dan kan ikzelf wel voor de debian packaging zorgen.

Groeten,

Joost


On Thu, Feb 28, 2019 at 08:05:28PM +0100, Joost van Baal-Ilić wrote:

> Hoi hoi,
>
> OK, ik wacht ongeveer 24 uur af; daarna ga ikzelf wel aan de slag.  Verwacht
> dat we op die manier de deadline wel gaan halen :)
>
> Heel hartelijk bedankt iig!
>
> Groeten,
>
> Joost
>
> On Thu, Feb 28, 2019 at 01:51:43PM -0500, Kees Leune wrote:
> > Hallo!
> >
> > Ik ben gisteren begonnen aan het package; kennelijk zijn de packaging rules
> > weer veranderd en moet ik het hele ding naar nieuwe standaarden brengen
> > voordat ik een schone linitian krijg. Dat kost (misschien) meer tijd dan ik
> > heb, maar ik zal het proberen.
> >
> >
> > On Thu, Feb 28, 2019 at 6:13 AM Joost van Baal-Ilić <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > Yes, I could invest one hour in getting this solved.
> > >
> > > Kees: laat me maar weten wat je nodig hebt.  (En hoe is t verder?)
> > >
> > > Groeten, Bye, Tschüß,
> > >
> > > Joost
> > >
> > >
> > > On Thu, Feb 28, 2019 at 11:43:04AM +0100, Daniel Lange wrote:
> > > > Hi Joost,
> > > >
> > > > we need an upload of a fixed metar before March 2nd to fix bug #921840.
> > > > Kees will prepare that and I would have uploaded it today but my internet
> > > > connection has been cut earlier today and will need 1-2 days for fixing.
> > > So
> > > > I'm using the phone to tether. No builds that way.
> > > >
> > > > So could you kindly jump in and co-ordinate with Kees to get metar
> > > uploaded?
> > > >
> > > > Kind regards,
> > > > Daniel
> > > >
> > > >
> > > > -------- Weitergeleitete Nachricht --------
> > > > Betreff:      Re: Upload necessary before March 2nd (Buster hard
> > > freeze), Bug
> > > > #921840 NOAA URL -> https
> > > > Datum:        Wed, 27 Feb 2019 09:51:41 -0500
> > > > Von:  Kees Leune <[hidden email]>
> > > > An:   Daniel Lange <[hidden email]>
> > > >
> > > >
> > > >
> > > > The process was very different back then ;) Joost van Baal did my initial
> > > > uploads.
> > > >
> > > > I'll take care of this later today!
> > > >
> > > > On Wed, Feb 27, 2019 at 9:16 AM Daniel Lange <[hidden email]
> > > > <mailto:[hidden email]>> wrote:
> > > >
> > > >     Hi Kees,
> > > >
> > > >     Am 27.02.19 um 15:07 schrieb Kees Leune:
> > > >      > No problem. How do I upload?
> > > >
> > > >     You seem to have been able to upload yourself in the past (at least
> > > >     until 2007):
> > > >     https://tracker.debian.org/pkg/metar
> > > >     Or may be the process was vastly different back then from today.
> > > >
> > > >     In any case you can upload to https://mentors.debian.net/
> > > >     and I or somebody else can grab it and upload for you.
> > > >
> > > >     Kind regards,
> > > >     Daniel
> > > >
> > > > --
> > > > https://www.leune.org/about
> > >
> >
> >
> > --
> > https://www.leune.org/about

Reply | Threaded
Open this post in threaded view
|

Bug#921840: Fwd: Re: Upload necessary before March 2nd (Buster hard freeze), Bug #921840 NOAA URL -> https

Kees Leune
Ok!

On Fri, Mar 1, 2019 at 2:49 PM Joost van Baal-Ilić <[hidden email]> wrote:
Hoi Kees,

( @ bugs.debian.org: sorry for speaking dutch here )

Of ben je anders van plan een .tar.gz release te maken?
Volgens je eigen instructies gaat dat met"

0. Assign a new version number in VERSION.m4, commit, push
1. Clone the repository from github
2. Remove .git/
3. rename the repository directory as metar-newversion-1
4. create a new tarball
        tar c --exclude debian -zf metar_newversion-1.origin.tar.gz


Kun je die .tar.gz dan op github in "releases" zetten?  Of zet hem
anders ergens op https://leune.org/ ter download, dat kan ook.

Dan kan ikzelf wel voor de debian packaging zorgen.

Groeten,

Joost


On Thu, Feb 28, 2019 at 08:05:28PM +0100, Joost van Baal-Ilić wrote:
> Hoi hoi,
>
> OK, ik wacht ongeveer 24 uur af; daarna ga ikzelf wel aan de slag.  Verwacht
> dat we op die manier de deadline wel gaan halen :)
>
> Heel hartelijk bedankt iig!
>
> Groeten,
>
> Joost
>
> On Thu, Feb 28, 2019 at 01:51:43PM -0500, Kees Leune wrote:
> > Hallo!
> >
> > Ik ben gisteren begonnen aan het package; kennelijk zijn de packaging rules
> > weer veranderd en moet ik het hele ding naar nieuwe standaarden brengen
> > voordat ik een schone linitian krijg. Dat kost (misschien) meer tijd dan ik
> > heb, maar ik zal het proberen.
> >
> >
> > On Thu, Feb 28, 2019 at 6:13 AM Joost van Baal-Ilić <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > Yes, I could invest one hour in getting this solved.
> > >
> > > Kees: laat me maar weten wat je nodig hebt.  (En hoe is t verder?)
> > >
> > > Groeten, Bye, Tschüß,
> > >
> > > Joost
> > >
> > >
> > > On Thu, Feb 28, 2019 at 11:43:04AM +0100, Daniel Lange wrote:
> > > > Hi Joost,
> > > >
> > > > we need an upload of a fixed metar before March 2nd to fix bug #921840.
> > > > Kees will prepare that and I would have uploaded it today but my internet
> > > > connection has been cut earlier today and will need 1-2 days for fixing.
> > > So
> > > > I'm using the phone to tether. No builds that way.
> > > >
> > > > So could you kindly jump in and co-ordinate with Kees to get metar
> > > uploaded?
> > > >
> > > > Kind regards,
> > > > Daniel
> > > >
> > > >
> > > > -------- Weitergeleitete Nachricht --------
> > > > Betreff:      Re: Upload necessary before March 2nd (Buster hard
> > > freeze), Bug
> > > > #921840 NOAA URL -> https
> > > > Datum:        Wed, 27 Feb 2019 09:51:41 -0500
> > > > Von:  Kees Leune <[hidden email]>
> > > > An:   Daniel Lange <[hidden email]>
> > > >
> > > >
> > > >
> > > > The process was very different back then ;) Joost van Baal did my initial
> > > > uploads.
> > > >
> > > > I'll take care of this later today!
> > > >
> > > > On Wed, Feb 27, 2019 at 9:16 AM Daniel Lange <[hidden email]
> > > > <mailto:[hidden email]>> wrote:
> > > >
> > > >     Hi Kees,
> > > >
> > > >     Am 27.02.19 um 15:07 schrieb Kees Leune:
> > > >      > No problem. How do I upload?
> > > >
> > > >     You seem to have been able to upload yourself in the past (at least
> > > >     until 2007):
> > > >     https://tracker.debian.org/pkg/metar
> > > >     Or may be the process was vastly different back then from today.
> > > >
> > > >     In any case you can upload to https://mentors.debian.net/
> > > >     and I or somebody else can grab it and upload for you.
> > > >
> > > >     Kind regards,
> > > >     Daniel
> > > >
> > > > --
> > > > https://www.leune.org/about
> > >
> >
> >
> > --
> > https://www.leune.org/about
--
Reply | Threaded
Open this post in threaded view
|

Bug#921840: Fwd: Re: Upload necessary before March 2nd (Buster hard freeze), Bug #921840 NOAA URL -> https

Joost van Baal-Ilić-77
Hi Kees,

I took the liberty to upload metar_20190227.1-1, based upon what's currently
in the github repository.  Because of the time contrains I took some
shortcuts.

I'll mail you the relevant patch soonish: it'd be helpful
if you could keep it in your repository.

Thanks, Bye,

Joost

On Fri, Mar 01, 2019 at 02:54:58PM -0500, Kees Leune wrote:

> Ok!
>
> On Fri, Mar 1, 2019 at 2:49 PM Joost van Baal-Ilić <[hidden email]>
> wrote:
>
> > Hoi Kees,
> >
> > ( @ bugs.debian.org: sorry for speaking dutch here )
> >
> > Of ben je anders van plan een .tar.gz release te maken?
> > Volgens je eigen instructies gaat dat met"
> >
> > 0. Assign a new version number in VERSION.m4, commit, push
> > 1. Clone the repository from github
> > 2. Remove .git/
> > 3. rename the repository directory as metar-newversion-1
> > 4. create a new tarball
> >         tar c --exclude debian -zf metar_newversion-1.origin.tar.gz
> >
> >
> > Kun je die .tar.gz dan op github in "releases" zetten?  Of zet hem
> > anders ergens op https://leune.org/ ter download, dat kan ook.
> >
> > Dan kan ikzelf wel voor de debian packaging zorgen.
> >
> > Groeten,
> >
> > Joost
> >
> >
> > On Thu, Feb 28, 2019 at 08:05:28PM +0100, Joost van Baal-Ilić wrote:
> > > Hoi hoi,
> > >
> > > OK, ik wacht ongeveer 24 uur af; daarna ga ikzelf wel aan de slag.
> > Verwacht
> > > dat we op die manier de deadline wel gaan halen :)
> > >
> > > Heel hartelijk bedankt iig!
> > >
> > > Groeten,
> > >
> > > Joost
> > >
> > > On Thu, Feb 28, 2019 at 01:51:43PM -0500, Kees Leune wrote:
> > > > Hallo!
> > > >
> > > > Ik ben gisteren begonnen aan het package; kennelijk zijn de packaging
> > rules
> > > > weer veranderd en moet ik het hele ding naar nieuwe standaarden brengen
> > > > voordat ik een schone linitian krijg. Dat kost (misschien) meer tijd
> > dan ik
> > > > heb, maar ik zal het proberen.
> > > >
> > > >
> > > > On Thu, Feb 28, 2019 at 6:13 AM Joost van Baal-Ilić <
> > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Yes, I could invest one hour in getting this solved.
> > > > >
> > > > > Kees: laat me maar weten wat je nodig hebt.  (En hoe is t verder?)
> > > > >
> > > > > Groeten, Bye, Tschüß,
> > > > >
> > > > > Joost
> > > > >
> > > > >
> > > > > On Thu, Feb 28, 2019 at 11:43:04AM +0100, Daniel Lange wrote:
> > > > > > Hi Joost,
> > > > > >
> > > > > > we need an upload of a fixed metar before March 2nd to fix bug
> > #921840.
> > > > > > Kees will prepare that and I would have uploaded it today but my
> > internet
> > > > > > connection has been cut earlier today and will need 1-2 days for
> > fixing.
> > > > > So
> > > > > > I'm using the phone to tether. No builds that way.
> > > > > >
> > > > > > So could you kindly jump in and co-ordinate with Kees to get metar
> > > > > uploaded?
> > > > > >
> > > > > > Kind regards,
> > > > > > Daniel
> > > > > >
> > > > > >
> > > > > > -------- Weitergeleitete Nachricht --------
> > > > > > Betreff:      Re: Upload necessary before March 2nd (Buster hard
> > > > > freeze), Bug
> > > > > > #921840 NOAA URL -> https
> > > > > > Datum:        Wed, 27 Feb 2019 09:51:41 -0500
> > > > > > Von:  Kees Leune <[hidden email]>
> > > > > > An:   Daniel Lange <[hidden email]>
> > > > > >
> > > > > >
> > > > > >
> > > > > > The process was very different back then ;) Joost van Baal did my
> > initial
> > > > > > uploads.
> > > > > >
> > > > > > I'll take care of this later today!
> > > > > >
> > > > > > On Wed, Feb 27, 2019 at 9:16 AM Daniel Lange <[hidden email]
> > > > > > <mailto:[hidden email]>> wrote:
> > > > > >
> > > > > >     Hi Kees,
> > > > > >
> > > > > >     Am 27.02.19 um 15:07 schrieb Kees Leune:
> > > > > >      > No problem. How do I upload?
> > > > > >
> > > > > >     You seem to have been able to upload yourself in the past (at
> > least
> > > > > >     until 2007):
> > > > > >     https://tracker.debian.org/pkg/metar
> > > > > >     Or may be the process was vastly different back then from
> > today.
> > > > > >
> > > > > >     In any case you can upload to https://mentors.debian.net/
> > > > > >     and I or somebody else can grab it and upload for you.
> > > > > >
> > > > > >     Kind regards,
> > > > > >     Daniel
> > > > > >
> > > > > > --
> > > > > > https://www.leune.org/about
> > > > >
> > > >
> > > >
> > > > --
> > > > https://www.leune.org/about
> >
> --
> https://www.leune.org/about

Reply | Threaded
Open this post in threaded view
|

Bug#921840: the patch / Re: Fwd: Re: Upload necessary before March 2nd (Buster hard freeze), Bug #921840 NOAA URL -> https

Joost van Baal-Ilić-77
Hi Kees,

Attached is the work I did to create metar_20190227.1-1 : the diff with
what's in https://github.com/keesL/metar/tree/master/debian now.

Bye,

Joost




On Sat, Mar 02, 2019 at 03:34:59PM +0100, Joost van Baal-Ilić wrote:

> Hi Kees,
>
> I took the liberty to upload metar_20190227.1-1, based upon what's currently
> in the github repository.  Because of the time contrains I took some
> shortcuts.
>
> I'll mail you the relevant patch soonish: it'd be helpful
> if you could keep it in your repository.
>
> Thanks, Bye,
>
> Joost
<snip>



metar-debian.patch (3K) Download Attachment
signature.asc (836 bytes) Download Attachment