2Removal: handling circular dependencies / dropping tests before the binary?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

2Removal: handling circular dependencies / dropping tests before the binary?

Rebecca N. Palmer-2
Some Python 2 packages have circular (build-)dependencies, making it
impossible to remove them one at a time without breaking anything.

One such cycle is seaborn, sphinx-gallery and matplotlib2 (#940873); I
have not attempted to systematically check for them.

How should these be dealt with?

--- Cut the cycle by removing one of the dependencies.

This may mean skipping (some of) the tests, as Python build dependencies
are often actually test dependencies.  There might be cases where it
isn't reasonably possible at all.

(Related question: if a package with a large test suite is being
uploaded anyway, should we consider removing the Python 2 tests just to
save buildd/debci resources?)

--- Prepare all the required packages in salsa / experimental, then
remove the entire cycle at once.

This is easy if the cycle is small and otherwise ready to remove, but if
one member has a big dependency tree, the whole cycle might have to stay
for some time.

(Related question: does it make sense to create no-Python2 salsa
branches of one's regular modules (particularly any that need nontrivial
changes) and mark them "please upload when the reverse dependencies have
been cleared", even if they aren't part of a cycle?)

Reply | Threaded
Open this post in threaded view
|

Re: 2Removal: handling circular dependencies / dropping tests before the binary?

Andrey Rahmatullin-3
On Sat, Sep 21, 2019 at 10:18:16AM +0100, Rebecca N. Palmer wrote:
> Some Python 2 packages have circular (build-)dependencies, making it
> impossible to remove them one at a time without breaking anything.
Unless I'm missing something, this shouldn't be a problem, just go away
and remove them all and upload them in roughly the same time?

> --- Cut the cycle by removing one of the dependencies.
>
> This may mean skipping (some of) the tests, as Python build dependencies are
> often actually test dependencies.  There might be cases where it isn't
> reasonably possible at all.
Unless some of those don't have Python 3 packages in the archive yet, why
removing py2 from one package would break building another, if that
another has py2 parts removed too?
Note that having an unbuildable package in sid for several days, until
it's updated to a no-py2 version, is perfectly acceptable.

--
WBR, wRAR

signature.asc (911 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 2Removal: handling circular dependencies / dropping tests before the binary?

Sandro Tosi-6
> Note that having an unbuildable package in sid for several days, until
> it's updated to a no-py2 version, is perfectly acceptable.

please note that may not be fine with every maintainer. dont just drop
packages without checking rdeps and/or if you know that will make
packages uninstallable/unbuildable (unless you check with the people
taking care of such packages first).

btw sphinx-gallery d-b on pyhton-seaborh has been removed.

--
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
G+: https://plus.google.com/u/0/+SandroTosi

Reply | Threaded
Open this post in threaded view
|

Re: 2Removal: handling circular dependencies

Rebecca N. Palmer-2
In reply to this post by Rebecca N. Palmer-2
[Summary of previous messages: I noted that packages with circular
dependencies can't be removed one at a time without breakage.  Replies
were to remove multiple packages at once if necessary, but ask first if
other maintainers' packages are involved.]

I have now checked what cycles we have:

- 13 small sets (one of 13, others 2-4 each): small enough that "remove
the whole set at once" is manageable.

- One big tangle (159 packages).  This probably needs breaking up:
--- Some of it involves documentation tools (e.g. sphinx).  These cycles
can be broken by using the Python 3 version of the tool.
--- Some of it is "A Suggests (or Recommends) A-extension, A-extension
Depends on A" cycles (e.g. pandas<->statsmodels).  If A-extension is
otherwise ready to remove but A is not, these can be broken by removing
the Suggests from A.  (Assuming we're still using "broken Suggests are
not allowed": this has previously been discussed, I forget where.)

Full listing:

159 pyrex lxml html5lib sphinx mako python-scipy pandas patsy
statsmodels seaborn sphinx-gallery matplotlib2 mpmath sympy
texlive-extra nbconvert jupyter-notebook ipywidgets texlive-base dot2tex
matplotlib numpydoc python-cycler ipykernel python-numpy mayavi2
python-chaco python-envisage python-enable joblib pyzmq jupyter-client
python-hypothesis chardet pygments python-pyface python-traitsui
python-apptools python-traits ipython jupyter-core nbformat
prompt-toolkit mercurial setuptools-scm python-py pytest-xdist pytest
python-packaging python-importlib-metadata python-pluggy pyopenssl
python-urllib3 python-future pyglet python-lz4 fdb sqlalchemy
sphinxcontrib-websupport requests python-click incremental twisted
automat python-service-identity python-gevent entrypoints python-flake8
xcffib cairocffi python-keyring wheel python-pip keyrings.alt execnet
sphinx-rtd-theme python-prometheus-client pytest-expect pytest-forked
pytest-runner python-bleach python-mccabe python-atomicwrites
python-attrs python-babel tap.py dbus-python python-qt4 pyqt5
python-characteristic apipkg python-cryptography python-dateutil
freezegun traitlets python-typing openpyxl python-tz python-et-xmlfile
python-whoosh python-flaky wcwidth pexpect pillow python-docutils pymacs
ropemacs python-mode enum34 sip4 configobj python-iso8601 pycairo
pygobject-2 pygobject send2trash pygtk simplejson six unittest2
python-mock python-psutil contextlib2 python-zipp python-traceback2
python-debian python-apt python-linecache2 python-funcsigs
python-concurrent.futures python-pathlib2 pickleshare testpath
python-genty xlwt more-itertools backports.functools-lru-cache rope
ropemode colorspacious cython pyyaml cvxopt sphinx-paramlinks
python-pysqlite2 alabaster python-setupdocs nose mistune terminado
python-webencodings ipython-genutils pep8 autopep8 pyxdg
python-nose-exclude pyflakes python-greenlet xapian-bindings

2 rdflib sparql-wrapper-python
2 mgltools-pmv autodocktools
2 salutatoi sat-templates
2 cclib cclib-data
3 pastescript paste pastedeploy
2 sugar sugar-pippy-activity
13 trac-mercurial trac trac-customfieldadmin trac-graphviz
trac-mastertickets trac-spamfilter trac-wikiprint trac-wysiwyg
trac-xmlrpc email2trac trac-accountmanager trac-authopenid trac-bitten
4 hachoir-urwid hachoir-core hachoir-parser hachoir-metadata
4 python-pysnmp4 python-pysnmp4-mibs python-pysnmp4-apps pysmi
3 laditools ladish laditools # two 2Removal bugs on the same package
2 rpmlint rpm
2 crossfire crossfire-maps
3 python2.7 python-defaults python-stdlib-extensions

(Based on recent [hidden email] messages; packages not listed
are not in a cycle)

Reply | Threaded
Open this post in threaded view
|

Re: 2Removal: handling circular dependencies

Simon McVittie-7
On Wed, 23 Oct 2019 at 23:05:39 +0100, Rebecca N. Palmer wrote:
> - One big tangle (159 packages).  This probably needs breaking up:
> --- Some of it involves documentation tools (e.g. sphinx).  These cycles can
> be broken by using the Python 3 version of the tool.

You've listed dbus-python as being part of this cycle, but since version
1.2.10-1 it only Build-Depends on sphinx-common, python3-sphinx and
python3-sphinx-rtd-theme. Is that enough to remove it from this cycle?

I removed the block when I made that change, but someone('s script?)
seems to have added it back? I can't remove python-dbus and close the
2removal bug yet, because lots of packages still depend on python-dbus.

I also can't see how -sphinx would depend directly or indirectly on
-dbus. Perhaps some of the blocks have been added the wrong way round?

I wonder whether parts of the 159-package cycle are actually smaller
cycles with non-trivial intersection, such as:

     python-sphinx --D--> sphinx-common
          | ^      <--S--
          D S
          v |
     python-alabaster

I'd tend to think of that as two intersecting 2-package cycles, rather
than one larger cycle.

> (Assuming we're still using "broken Suggests are not
> allowed": this has previously been discussed, I forget where.)

I think that's much too strong, particularly during a
transition. Transitions from unstable to testing only look at
(Pre-)Depends and Build-Depends(|-Arch|-Indep), although in general we
also consider broken Recommends to be a serious bug if anyone notices
them (due to Policy §2.2.1 "must not require or recommend a package
outside of main for compilation or execution").

Suggests are specifically not considered by Policy §2.2.1 (packages in
main are allowed to suggest packages that are in contrib, in non-free,
or not in the archive at all) and the pattern of cyclic Depends in one
direction, and Recommends or Suggests in the other, is extremely common.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: 2Removal: handling circular dependencies

Andrey Rahmatullin-3
On Thu, Oct 24, 2019 at 08:49:41AM +0100, Simon McVittie wrote:
> in general we
> also consider broken Recommends to be a serious bug if anyone notices
> them (due to Policy §2.2.1 "must not require or recommend a package
> outside of main for compilation or execution").
Note that the release policy up to buster explicitly said ""Recommends:"
lines do not count as requirements.", this is not true for
https://release.debian.org/bullseye/rc_policy.txt though.

--
WBR, wRAR

signature.asc (911 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 2Removal: handling circular dependencies

Rebecca N. Palmer-2
Simon McVittie wrote:
> You've listed dbus-python as being part of this cycle, but [it probably shouldn't be]
> Perhaps some of the blocks have been added the wrong way round?

Or perhaps the script is imperfectly guessing which binaries are Python
2?  e.g. matplotlib is listed even though it's already a Python 3 only
package.

> I wonder whether parts of the 159-package cycle are actually smaller
> cycles with non-trivial intersection

Very likely: by "tangle" I meant
https://en.wikipedia.org/wiki/Strongly_connected_component

i.e. not a single cycle, but you still can't remove any subset of them
(other than all or none) without breaking at least some of the remaining
ones.

>> (Assuming we're still using "broken Suggests are not
>> allowed": this has previously been discussed, I forget where.)
>
> I think that's much too strong

If the question is still open for discussion, I'd also prefer letting
Suggests be temporarily broken over making extra uploads to remove them.
  (I haven't checked how much this would break up the 159-tangle.)

Reply | Threaded
Open this post in threaded view
|

Re: 2Removal: handling circular dependencies

Sandro Tosi-6
In reply to this post by Rebecca N. Palmer-2
Hey Rebecca, do you have an updated list of these cycles?

On Wed, Oct 23, 2019 at 6:39 PM Rebecca N. Palmer
<[hidden email]> wrote:

>
> [Summary of previous messages: I noted that packages with circular
> dependencies can't be removed one at a time without breakage.  Replies
> were to remove multiple packages at once if necessary, but ask first if
> other maintainers' packages are involved.]
>
> I have now checked what cycles we have:
>
> - 13 small sets (one of 13, others 2-4 each): small enough that "remove
> the whole set at once" is manageable.
>
> - One big tangle (159 packages).  This probably needs breaking up:
> --- Some of it involves documentation tools (e.g. sphinx).  These cycles
> can be broken by using the Python 3 version of the tool.
> --- Some of it is "A Suggests (or Recommends) A-extension, A-extension
> Depends on A" cycles (e.g. pandas<->statsmodels).  If A-extension is
> otherwise ready to remove but A is not, these can be broken by removing
> the Suggests from A.  (Assuming we're still using "broken Suggests are
> not allowed": this has previously been discussed, I forget where.)
>
> Full listing:
>
> 159 pyrex lxml html5lib sphinx mako python-scipy pandas patsy
> statsmodels seaborn sphinx-gallery matplotlib2 mpmath sympy
> texlive-extra nbconvert jupyter-notebook ipywidgets texlive-base dot2tex
> matplotlib numpydoc python-cycler ipykernel python-numpy mayavi2
> python-chaco python-envisage python-enable joblib pyzmq jupyter-client
> python-hypothesis chardet pygments python-pyface python-traitsui
> python-apptools python-traits ipython jupyter-core nbformat
> prompt-toolkit mercurial setuptools-scm python-py pytest-xdist pytest
> python-packaging python-importlib-metadata python-pluggy pyopenssl
> python-urllib3 python-future pyglet python-lz4 fdb sqlalchemy
> sphinxcontrib-websupport requests python-click incremental twisted
> automat python-service-identity python-gevent entrypoints python-flake8
> xcffib cairocffi python-keyring wheel python-pip keyrings.alt execnet
> sphinx-rtd-theme python-prometheus-client pytest-expect pytest-forked
> pytest-runner python-bleach python-mccabe python-atomicwrites
> python-attrs python-babel tap.py dbus-python python-qt4 pyqt5
> python-characteristic apipkg python-cryptography python-dateutil
> freezegun traitlets python-typing openpyxl python-tz python-et-xmlfile
> python-whoosh python-flaky wcwidth pexpect pillow python-docutils pymacs
> ropemacs python-mode enum34 sip4 configobj python-iso8601 pycairo
> pygobject-2 pygobject send2trash pygtk simplejson six unittest2
> python-mock python-psutil contextlib2 python-zipp python-traceback2
> python-debian python-apt python-linecache2 python-funcsigs
> python-concurrent.futures python-pathlib2 pickleshare testpath
> python-genty xlwt more-itertools backports.functools-lru-cache rope
> ropemode colorspacious cython pyyaml cvxopt sphinx-paramlinks
> python-pysqlite2 alabaster python-setupdocs nose mistune terminado
> python-webencodings ipython-genutils pep8 autopep8 pyxdg
> python-nose-exclude pyflakes python-greenlet xapian-bindings
>
> 2 rdflib sparql-wrapper-python
> 2 mgltools-pmv autodocktools
> 2 salutatoi sat-templates
> 2 cclib cclib-data
> 3 pastescript paste pastedeploy
> 2 sugar sugar-pippy-activity
> 13 trac-mercurial trac trac-customfieldadmin trac-graphviz
> trac-mastertickets trac-spamfilter trac-wikiprint trac-wysiwyg
> trac-xmlrpc email2trac trac-accountmanager trac-authopenid trac-bitten
> 4 hachoir-urwid hachoir-core hachoir-parser hachoir-metadata
> 4 python-pysnmp4 python-pysnmp4-mibs python-pysnmp4-apps pysmi
> 3 laditools ladish laditools # two 2Removal bugs on the same package
> 2 rpmlint rpm
> 2 crossfire crossfire-maps
> 3 python2.7 python-defaults python-stdlib-extensions
>
> (Based on recent [hidden email] messages; packages not listed
> are not in a cycle)
>


--
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

Reply | Threaded
Open this post in threaded view
|

Re: 2Removal: handling circular dependencies

Rebecca N. Palmer-2
On 16/03/2020 06:28, Sandro Tosi wrote:
> Hey Rebecca, do you have an updated list of these cycles?

Warning: this is based on open py2removal bugs, *not* actual package
relationships.

2 crossfire crossfire-maps
2 sugar sugar-pippy-activity
2 rabbyt snowballz
4 python-pysnmp4 pysmi python-pysnmp4-apps python-pysnmp4-mibs
78 mercurial mercurial-extension-utils mercurial-keyring setuptools-scm
automat twisted incremental pytest chardet beautifulsoup4 lxml html5lib
soupsieve sphinx alabaster pycairo pygobject-2 pygobject keyrings.alt
python-keyring dbus-python pygobject pygtk pycurl python-tornado
python-urllib3 requests python-concurrent.futures pygments
python-docutils enum34 python-cryptography pyopenssl
python-service-identity python-hypothesis pyhamcrest python-attrs
python-packaging python-importlib-metadata python-pluggy python-dateutil
pymacs python-mode pyrex python-numpy cython pystemmer python-debian
python-apt entrypoints pexpect pillow pytest-expect python-atomicwrites
python-babel python-cffi python-flaky python-iso8601 python-py six
more-itertools python-zipp python-genty python-mock python-pbr
python-linecache2 python-traceback2 unittest2 contextlib2
python-funcsigs python-virtualenv python-pathlib2 singledispatch
python-typing python-tz wcwidth pillow-python2 backports.functools-lru-cache
2 apr apr-util
3 python2.7 python-defaults python-stdlib-extensions