Migrating debbugs' SOAP interface to JSON

Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Migrating debbugs' SOAP interface to JSON

Bastian Venthur-3
Dear debbugs Developers,

I'm Bastian the developer of python-debianbts, the library that is used
by reportbug to query the BTS.

First of all, I'm really happy that we have an API that allows to query
the BTS, and the current interface is *a lot* better than the HTML
parsing that we did 10 years ago. However, while SOAP was state of the
art a while ago, it has it's limitations. The most severe one for us:
it's getting harder and harder to find libraries that parse SOAP
properly in Python and trying to parse SOAP replies "by hand" seems
daunting.

Also there are a lot of bugs related to parsing the SOAP replies from
the BTS and it is not always clear if the problem is the BTS or the
library that tries to parse the SOAP reply. My feeling having worked
with SOAP, it is unpleasant and messy.

Since our API is already nicely structured, I wonder if we couldn't
switch on the long run to something like JSON? JSON can be easily
created and parsed in many languages and makes debugging a lot easier.

Unfortunately I don't speak Perl, so I can't really help you with the
server side implementation, but I'd be willing to help as much as
possible testing anything that helps to get away from SOAP.

If there's a general willingness to move that direction, I propose an
incremental approach where we create a second API with JSON replies for
each endpoint, one by one and I implement the corresponding parser for
each new endpoint. We could run both APIs in parallel for a while and
turn of the old one once we're sufficiently confident that reportbug
works properly.


What do you think?


Cheers,

Bastian

--
Dr. Bastian Venthur                                  http://venthur.de
Debian Developer                                 venthur at debian org

Reply | Threaded
Open this post in threaded view
|

Re: Migrating debbugs' SOAP interface to JSON

Don Armstrong
On Thu, 06 Dec 2018, Bastian Venthur wrote:
> I'm Bastian the developer of python-debianbts, the library that is used by
> reportbug to query the BTS.
>
> First of all, I'm really happy that we have an API that allows to query the
> BTS, and the current interface is *a lot* better than the HTML parsing that
> we did 10 years ago. However, while SOAP was state of the art a while ago,
> it has it's limitations. The most severe one for us: it's getting harder and
> harder to find libraries that parse SOAP properly in Python and trying to
> parse SOAP replies "by hand" seems daunting.

Yeah, I totally agree that we should move to replace SOAP.

My long-standing plan is to have it replaced with a REST API that
returns json or html (maybe even xml?) depending on the Accepts: header.
I'd certainly love to see someone write a good specification for that
API using openAPI (swagger) which I (or someone else) could then work to
write the perl backend to the BTS to spit out that API.

--
Don Armstrong                      https://www.donarmstrong.com

Those who begin coercive elimination of dissent soon find themselves
exterminating dissenters. Compulsory unification of opinion achieves
only the unanimity of the graveyard.
 -- Justice Roberts in 319 U.S. 624 (1943)

Reply | Threaded
Open this post in threaded view
|

Re: Migrating debbugs' SOAP interface to JSON

Bastian Venthur-3
On 07.12.18 05:23, Don Armstrong wrote:

> On Thu, 06 Dec 2018, Bastian Venthur wrote:
>> I'm Bastian the developer of python-debianbts, the library that is used by
>> reportbug to query the BTS.
>>
>> First of all, I'm really happy that we have an API that allows to query the
>> BTS, and the current interface is *a lot* better than the HTML parsing that
>> we did 10 years ago. However, while SOAP was state of the art a while ago,
>> it has it's limitations. The most severe one for us: it's getting harder and
>> harder to find libraries that parse SOAP properly in Python and trying to
>> parse SOAP replies "by hand" seems daunting.
>
> Yeah, I totally agree that we should move to replace SOAP.
>
> My long-standing plan is to have it replaced with a REST API that
> returns json or html (maybe even xml?) depending on the Accepts: header.
> I'd certainly love to see someone write a good specification for that
> API using openAPI (swagger) which I (or someone else) could then work to
> write the perl backend to the BTS to spit out that API.
>

Good to hear! To start it off, I've created a swagger spec that is
faithful to the current setup:

https://github.com/venthur/debbugs-proposal/blob/master/yaml-unresolved/swagger.yaml

The only changes so far:

* renamed `get_foo` methods to `foo`, the "get_" part is implied by the
   HTTP verb `GET`
* complex parameters are passed as JSON
* Return values are JSON

There is another branch in this repo `SWAGGERHUB` that is connected to
swaggerhub, which I use to make changes. The changes are pushed to the
branch automatically but not the other way round. This makes it tricky
to collaborate, since only I can edit in swaggerhub.

However you can see the current status of the `SWAGGERHUB` branch here:
https://app.swaggerhub.com/apis-docs/venthur/debbugs/1.0.0

If you want to take over the repo as owner, that's fine with me. I can
also manually merge the current best version to `master` and others make
PRs against that branch, I don't really mind.

As next steps I propose to improve the `Status` data type (i.e. the
return value of the former `get_status`:

* do we really need all fields?
* the `done` field is currently a maintainer or empty if not closed,
   maybe we can introduce `status` with the possible states: `open`,
   `pending`, `closed`, `archived` and introduce an optional `closed_by`:
   <maintaine>, this way we could also get rid of `archived`,
   `pending` and `unarchived`


Cheers,

Bastian