Bug#924786: ITP: golang-github-michaeltjones-walk -- Fast parallel version of golang filepath.Walk()

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#924786: ITP: golang-github-michaeltjones-walk -- Fast parallel version of golang filepath.Walk()

Dawid Dziurla
Package: wnpp
Severity: wishlist
Owner: Dawid Dziurla <[hidden email]>

* Package name    : golang-github-michaeltjones-walk
  Version         : 0.0~git20161122.4748e29-1
  Upstream Author : Michael T Jones
* URL             : https://github.com/MichaelTJones/walk
* License         : Expat
  Programming Lang: Go
  Description     : Fast parallel version of golang filepath.Walk()

 Performs traversals in parallel so set GOMAXPROCS appropriately. Vaues
 of 8 to 16 seem to work best on my 4-CPU plus 4 SMT pseudo-CPU
 MacBookPro. The result is about 4x-6x the traversal rate of the standard
 Walk().  The two are not identical since we are walking the file system
 in a tumult of asynchronous walkFunc calls by a number of goroutines. So,
 take note of the following:
 • This walk honors all of the walkFunc
 error semantics but as multiple user-supplied walkFuncs may simultaneously
 encounter a traversal error or generate one to stop traversal, only the
 FIRST of these will be returned as the Walk() result.
 • Further, since
 there may be a few files in flight at the instant of  error discovery, a
 few more walkFunc calls may happen after the first error-generating call
 has signaled its desire to stop. In general this is a non-issue but it
 could matter so pay attention when designing your walkFunc. (For example,
 if you accumulate results then you need to have your own means to know
 to stop accumulating once you signal an error.)
 • Because the walkFunc
 is called concurrently in multiple goroutines, it needs to be careful
 about what it does with external data to avoid collisions. Results may
 be printed using fmt, but generally the best plan is to send results
 over a channel or accumulate counts using a locked mutex.
 .
 These issues
 are illustrated/handled in the simple traversal programs supplied with
 walk. There is also a test file that is just the tests from filepath
 in the Go language's standard library. Walk passes these tests when
 run in single process mode, and passes most of them in concurrent mode
 (GOMAXPROCS > 1). The problem is not a real problem, but one of the
 test expecting a specific number of errors to be found based on presumed
 sequential traversals.

This package is in the dependency tree of Lazygit (#908894)

Reply | Threaded
Open this post in threaded view
|

Bug#924786: RFS: golang-github-michaeltjones-walk/0.0~git20161122.4748e29-1

Dawid Dziurla
Dear Go team,

I am looking for a sponsor for the package "golang-github-michaeltjones-walk".
This package is a prerequisite for upcoming package "lazygit" (#908894).

I pushed to our team's Salsa:

  https://salsa.debian.org/go-team/packages/golang-github-michaeltjones-walk

Could you please reviewing/sponsoring this?
Any kind of reviews and suggestions are appreciated.

Reply | Threaded
Open this post in threaded view
|

Bug#924786: RFS: golang-github-michaeltjones-walk/0.0~git20161122.4748e29-1

Shengjing Zhu-3
Hi Dawid,


On Mon, Mar 18, 2019 at 1:13 AM Dawid Dziurla <[hidden email]> wrote:
>
> Dear Go team,
>
> I am looking for a sponsor for the package "golang-github-michaeltjones-walk".
> This package is a prerequisite for upcoming package "lazygit" (#908894).

I just have a quick look at lazygit. Both
golang-github-michaeltjones-walk and golang-github-anmitsu-go-shlex
are not in lazygit's dependency tree[1]

[1] https://salsa.debian.org/go-team/packages/lazygit/blob/debian/sid/Gopkg.lock

Maybe you can reduce your work burden by checking the dependency tree
more carefully? Or maybe I'm wrong if there's complicated dependency
hell here :(

PS. I don't plan to sponsor this since I don't have much time recently.

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#924786: RFS: golang-github-michaeltjones-walk/0.0~git20161122.4748e29-1

Dawid Dziurla
Hello,

yes, actually there are dependencies of dependencies of dependencies that need to be packaged in order for lazygit to be buildable in Debian.

Here is the dependency tree (made by Jongmin Kim):
https://raw.githubusercontent.com/jesseduffield/lazygit-debian/master/deps-graph/exported/20190311.png


On March 17, 2019 6:43:04 PM GMT+01:00, Shengjing Zhu <[hidden email]> wrote:
Hi Dawid,


On Mon, Mar 18, 2019 at 1:13 AM Dawid Dziurla <[hidden email]> wrote:

Dear Go team,

I am looking for a sponsor for the package "golang-github-michaeltjones-walk".
This package is a prerequisite for upcoming package "lazygit" (#908894).

I just have a quick look at lazygit. Both
golang-github-michaeltjones-walk and golang-github-anmitsu-go-shlex
are not in lazygit's dependency tree[1]

[1] https://salsa.debian.org/go-team/packages/lazygit/blob/debian/sid/Gopkg.lock

Maybe you can reduce your work burden by checking the dependency tree
more carefully? Or maybe I'm wrong if there's complicated dependency
hell here :(

PS. I don't plan to sponsor this since I don't have much time recently.

--
Shengjing Zhu
Reply | Threaded
Open this post in threaded view
|

Bug#924786: RFS: golang-github-michaeltjones-walk/0.0~git20161122.4748e29-1

Shengjing Zhu-3
On Mon, Mar 18, 2019 at 1:51 AM Dawid Dziurla <[hidden email]> wrote:
>
> Hello,
>
> yes, actually there are dependencies of dependencies of dependencies that need to be packaged in order for lazygit to be buildable in Debian.
>
> Here is the dependency tree (made by Jongmin Kim):
> https://raw.githubusercontent.com/jesseduffield/lazygit-debian/master/deps-graph/exported/20190311.png
>

I looks... headache. One tip is that, if you don't need the
build-depends' binary excutables, you can just skip them. Maybe it can
save some time.


--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#924786: RFS: golang-github-michaeltjones-walk/0.0~git20161122.4748e29-1

Thorsten Alteholz-4
In reply to this post by Dawid Dziurla
Hi Dawid,

On Sun, 17 Mar 2019, Dawid Dziurla wrote:
>  https://salsa.debian.org/go-team/packages/golang-github-michaeltjones-walk

is it me or are your watch files broken:
  gbp:info: Running Postclone hook
  W: Paket golang-github-michaeltjones-walk kann nicht gefunden werden.
  Trying uscan --download --download-current-version ...
  uscan warn: In debian/watch no matching hrefs for version 0.0~git20161122.4748e29 in watch line
    https://github.com/MichaelTJones/walk/tags .*/v?(\d\S*)\.tar\.gz
  Could not find any location for golang-github-michaeltjones-walk_0.0~git20161122.4748e29.orig.tar.*
  gbp:error: Postclone-hook 'origtargz' failed: it exited with 1


Thorsten

Reply | Threaded
Open this post in threaded view
|

Bug#924786: RFS: golang-github-michaeltjones-walk/0.0~git20161122.4748e29-1

Dawid Dziurla
Hello Thorsten,

That is correct, watch file was looking for tags, which upstream does
not provide.
It should be fixed now.

https://salsa.debian.org/go-team/packages/golang-github-michaeltjones-walk/commit/a7e03df3e00dce925303ec661a79d1947ba12bab

niedz., 14 kwi 2019 o 12:30 Thorsten Alteholz <[hidden email]> napisał(a):

>
> Hi Dawid,
>
> On Sun, 17 Mar 2019, Dawid Dziurla wrote:
> >  https://salsa.debian.org/go-team/packages/golang-github-michaeltjones-walk
>
> is it me or are your watch files broken:
>   gbp:info: Running Postclone hook
>   W: Paket golang-github-michaeltjones-walk kann nicht gefunden werden.
>   Trying uscan --download --download-current-version ...
>   uscan warn: In debian/watch no matching hrefs for version 0.0~git20161122.4748e29 in watch line
>     https://github.com/MichaelTJones/walk/tags .*/v?(\d\S*)\.tar\.gz
>   Could not find any location for golang-github-michaeltjones-walk_0.0~git20161122.4748e29.orig.tar.*
>   gbp:error: Postclone-hook 'origtargz' failed: it exited with 1
>
>
> Thorsten
>

Reply | Threaded
Open this post in threaded view
|

Bug#924786: RFS: golang-github-michaeltjones-walk/0.0~git20161122.4748e29-1

Thorsten Alteholz-4
Hi Dawid

On Sun, 14 Apr 2019, Dawid Dziurla wrote:
> It should be fixed now.

yes, great, but please take care of:
  I: golang-github-michaeltjones-walk-dev: using-first-person-in-description line 2: my
  I: golang-github-michaeltjones-walk-dev: using-first-person-in-description line 4: we
as well.

   Thorsten

Reply | Threaded
Open this post in threaded view
|

Bug#924786: RFS: golang-github-michaeltjones-walk/0.0~git20161122.4748e29-1

Dawid Dziurla
Done.

niedz., 14 kwi 2019 o 20:05 Thorsten Alteholz <[hidden email]> napisał(a):

>
> Hi Dawid
>
> On Sun, 14 Apr 2019, Dawid Dziurla wrote:
> > It should be fixed now.
>
> yes, great, but please take care of:
>   I: golang-github-michaeltjones-walk-dev: using-first-person-in-description line 2: my
>   I: golang-github-michaeltjones-walk-dev: using-first-person-in-description line 4: we
> as well.
>
>    Thorsten
>