wanted: recommendation for webhook queueing library/thing

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

wanted: recommendation for webhook queueing library/thing

Ian Jackson-2
Sean Whitton writes ("git & Debian packaging sprint report"):

> Main achievement
> ----------------
>
> We designed and implemented a system to make it possible for DDs to
> upload new versions of packages by simply pushing a specially formatted
> git tag to salsa.
>
> Please see this blog post to learn about how it works:
> https://spwhitton.name/blog/entry/tag2upload/
>
> While the cloud service part of this system has not yet been deployed,
> and so you can't just tag to upload yet, the blog post explains how you
> can run the cloud service in an ad-hoc mode on your laptop, and thereby
> get a feel for how it works.
>
> You can also read git-debpush(1) in sid.[1]

The one technical piece missing for deployment is webhook queue /
dispatcher.  Can someone familiar with this space recommend one ?

In more detail, here is what I think the problem is:

 * The webhook client expects a quick response. But the bot's
   processing for a tag addressed to the bot takes some time (because
   it needs to fetch a fair amount of data and build a source
   package).  So webhook calls must be queued.

 * Sometimes it will be necessary to retry a failed bot invocation.
   (The bot already knows whether a particular invocation should be
   retried, and can feed this back to the queue.)  The retry interval
   and number of retries must be limited.

 * The bot expects to be forked/exec'd.

 * Significant protection against DoS is not needed because the
   web hook receiver's webserver will have a firewall protecting it
   from anything other than salsa.

 * The webserver and web hook receiver/queue (including the JSON
   parser which will be needed) must be high quality software, because
   we really want to avoid any vulnerabilities.

 * I can easily change the bot's fork/exec API to make it conveniently
   fit into whatever queue system.

If necessary I can write my own but it seems like this is a problem
that there should already be a good solution to.

Thanks,
Ian.

Reply | Threaded
Open this post in threaded view
|

Re: wanted: recommendation for webhook queueing library/thing

Antonio Terceiro-3
On Wed, Jul 10, 2019 at 04:09:11PM +0100, Ian Jackson wrote:

> Sean Whitton writes ("git & Debian packaging sprint report"):
> > Main achievement
> > ----------------
> >
> > We designed and implemented a system to make it possible for DDs to
> > upload new versions of packages by simply pushing a specially formatted
> > git tag to salsa.
> >
> > Please see this blog post to learn about how it works:
> > https://spwhitton.name/blog/entry/tag2upload/
> >
> > While the cloud service part of this system has not yet been deployed,
> > and so you can't just tag to upload yet, the blog post explains how you
> > can run the cloud service in an ad-hoc mode on your laptop, and thereby
> > get a feel for how it works.
> >
> > You can also read git-debpush(1) in sid.[1]
>
> The one technical piece missing for deployment is webhook queue /
> dispatcher.  Can someone familiar with this space recommend one ?
>
> In more detail, here is what I think the problem is:
>
>  * The webhook client expects a quick response. But the bot's
>    processing for a tag addressed to the bot takes some time (because
>    it needs to fetch a fair amount of data and build a source
>    package).  So webhook calls must be queued.
>
>  * Sometimes it will be necessary to retry a failed bot invocation.
>    (The bot already knows whether a particular invocation should be
>    retried, and can feed this back to the queue.)  The retry interval
>    and number of retries must be limited.
>
>  * The bot expects to be forked/exec'd.
>
>  * Significant protection against DoS is not needed because the
>    web hook receiver's webserver will have a firewall protecting it
>    from anything other than salsa.
>
>  * The webserver and web hook receiver/queue (including the JSON
>    parser which will be needed) must be high quality software, because
>    we really want to avoid any vulnerabilities.
>
>  * I can easily change the bot's fork/exec API to make it conveniently
>    fit into whatever queue system.
>
> If necessary I can write my own but it seems like this is a problem
> that there should already be a good solution to.
Enrico has been producing some content that is related:

https://www.enricozini.org/blog/2018/debian/automatic-deploy-from-gitlab-salsa-ci/
https://debconf18.debconf.org/talks/71-autodeploy-from-salsa/

It's based on systemd .path units. the advantage is that you don't need
a new component, you only put the config in place and systemd takes care
of it for you.

another option is rabbitmq; that's what debci uses. and it has clients
for several languages. one advantage is that you can have multiple
worker processes consuming from the queue in parallel, but OTOH you have
do manage those extra daemons, care about not exposing it to the
internet, etc.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: wanted: recommendation for webhook queueing library/thing

Sam Hartman-3
I also use systemd.path units for this.
Since it was Ian asking I didn't really feel that was worth suggesting
though.

Reply | Threaded
Open this post in threaded view
|

Re: wanted: recommendation for webhook queueing library/thing

Antonio Terceiro-3
On Wed, Jul 10, 2019 at 05:05:53PM -0400, Sam Hartman wrote:
> I also use systemd.path units for this.
> Since it was Ian asking I didn't really feel that was worth suggesting
> though.

yes, I also though about that but decided to cite prior art anyway. one
can always use inotify to achieve the same effect.

signature.asc (849 bytes) Download Attachment