
12

Source: blis, pythonscipy
Control: found 1 blis/0.5.15
Control: found 1 pythonscipy/1.1.02
XDebbugsCC: [hidden email]
User: [hidden email]
Usertags: breaks needsupdate
Dear maintainers,
With a recent upload of blis the autopkgtest of pythonscipy fails in
testing when that autopkgtest is run with the binary packages of blis
from unstable. It passes when run with only packages from testing. In
tabular form:
pass fail
blis from testing 0.5.15
pythonscipy from testing 1.1.02
all others from testing from testing
I copied some of the output at the bottom of this report. On a quick
view, similar errors occur for python3.
Currently this regression is blocking the migration of blis to testing
[1]. Due to the nature of this issue, I filed this bug report against
both packages. Can you please investigate the situation and reassign the
bug to the right package? If needed, please change the bug's severity.
More information about this bug and the reason for filing it can be found on
https://wiki.debian.org/ContinuousIntegration/RegressionEmailInformationPaul
[1] https://qa.debian.org/excuses.php?package=blishttps://ci.debian.net/data/autopkgtest/testing/amd64/p/pythonscipy/1739165/log.gz=================================== FAILURES
===================================
________________________ TestInt32Overflow.test_matvecs
________________________
self = <scipy.sparse.tests.test_sparsetools.TestInt32Overflow object at
0x7fa1f70d05d0>
@pytest.mark.slow
def test_matvecs(self):
# Check *_matvecs routines
n = self.n
i = np.array([0, n1])
j = np.array([0, n1])
data = np.array([1, 2], dtype=np.int8)
m = coo_matrix((data, (i, j)))
b = np.ones((n, n), dtype=np.int8)
for sptype in (csr_matrix, csc_matrix, bsr_matrix):
m2 = sptype(m)
> r = m2.dot(b)
b = array([[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1],
[1, 1, ...],
[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1]], dtype=int8)
data = array([1, 2], dtype=int8)
i = array([ 0, 49999])
j = array([ 0, 49999])
m = <50000x50000 sparse matrix of type '<type 'numpy.int8'>'
with 2 stored elements in COOrdinate format>
m2 = <50000x50000 sparse matrix of type '<type 'numpy.int8'>'
with 2 stored elements in Compressed Sparse Row format>
n = 50000
self = <scipy.sparse.tests.test_sparsetools.TestInt32Overflow
object at 0x7fa1f70d05d0>
sptype = <class 'scipy.sparse.csr.csr_matrix'>
/usr/lib/python2.7/distpackages/scipy/sparse/tests/test_sparsetools.py:142:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
/usr/lib/python2.7/distpackages/scipy/sparse/base.py:361: in dot
return self * other
/usr/lib/python2.7/distpackages/scipy/sparse/base.py:470: in __mul__
return self._mul_multivector(other)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
self = <50000x50000 sparse matrix of type '<type 'numpy.int8'>'
with 2 stored elements in Compressed Sparse Row format>
other = array([[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1],
[1, 1, ...],
[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1]], dtype=int8)
def _mul_multivector(self, other):
M,N = self.shape
n_vecs = other.shape[1] # number of column vectors
result = np.zeros((M,n_vecs), dtype=upcast_char(self.dtype.char,
> other.dtype.char))
E MemoryError
M = 50000
N = 50000
n_vecs = 50000
other = array([[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1],
[1, 1, ...],
[1, 1, 1, ..., 1, 1, 1],
[1, 1, 1, ..., 1, 1, 1]], dtype=int8)
self = <50000x50000 sparse matrix of type '<type 'numpy.int8'>'
with 2 stored elements in Compressed Sparse Row format>
/usr/lib/python2.7/distpackages/scipy/sparse/compressed.py:469: MemoryError
______________________ TestInt32Overflow.test_dia_matvec
_______________________
self = <scipy.sparse.tests.test_sparsetools.TestInt32Overflow object at
0x7fa1f937fb10>
@pytest.mark.slow
def test_dia_matvec(self):
# Check: huge dia_matrix _matvec
n = self.n
> data = np.ones((n, n), dtype=np.int8)
n = 50000
self = <scipy.sparse.tests.test_sparsetools.TestInt32Overflow
object at 0x7fa1f937fb10>
/usr/lib/python2.7/distpackages/scipy/sparse/tests/test_sparsetools.py:155:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
shape = (50000, 50000), dtype = <type 'numpy.int8'>, order = 'C'
@set_module('numpy')
def ones(shape, dtype=None, order='C'):
"""
Return a new array of given shape and type, filled with ones.
Parameters

shape : int or sequence of ints
Shape of the new array, e.g., ``(2, 3)`` or ``2``.
dtype : datatype, optional
The desired datatype for the array, e.g., `numpy.int8`.
Default is
`numpy.float64`.
order : {'C', 'F'}, optional, default: C
Whether to store multidimensional data in rowmajor
(Cstyle) or columnmajor (Fortranstyle) order in
memory.
Returns

out : ndarray
Array of ones with the given shape, dtype, and order.
See Also

ones_like : Return an array of ones with shape and type of input.
empty : Return a new uninitialized array.
zeros : Return a new array setting values to zero.
full : Return a new array of given shape filled with value.
Examples

>>> np.ones(5)
array([ 1., 1., 1., 1., 1.])
>>> np.ones((5,), dtype=int)
array([1, 1, 1, 1, 1])
>>> np.ones((2, 1))
array([[ 1.],
[ 1.]])
>>> s = (2,2)
>>> np.ones(s)
array([[ 1., 1.],
[ 1., 1.]])
"""
> a = empty(shape, dtype, order)
E MemoryError
dtype = <type 'numpy.int8'>
order = 'C'
shape = (50000, 50000)
/usr/lib/python2.7/distpackages/numpy/core/numeric.py:223: MemoryError
generated xml file:
/tmp/autopkgtestlxc.b012crl_/downtmp/autopkgtest_tmp/junit.xml


Hi Paul
On 2019/01/20 21:09, Paul Gevers wrote:
> With a recent upload of blis the autopkgtest of pythonscipy fails in
> testing when that autopkgtest is run with the binary packages of blis
> from unstable. It passes when run with only packages from testing. In
> tabular form:
> pass fail
> blis from testing 0.5.15
> pythonscipy from testing 1.1.02
> all others from testing from testing
From testing's test logs [1], pythonscipy 1.1.02 passed with
blis/0.5.15 on 20190121 05:22:48 UTC. It then passed with
blis/0.5.16 and blis/0.5.17, but failed with blis/0.5.17 on
20190127 17:55:32 UTC.
From unstable's test logs [2], pythonscipy 1.1.02 seems to have many
intermittent failures since 20181202.
I'm unsure what to do with this bug, perhaps reassign to pythonscipy only?
Regards
Graham
[1] https://ci.debian.net/packages/p/pythonscipy/testing/amd64/[2] https://ci.debian.net/packages/p/pythonscipy/unstable/amd64/


On 29.01.19 12:01, Graham Inggs wrote:
> Hi Paul
>
> On 2019/01/20 21:09, Paul Gevers wrote:
>> With a recent upload of blis the autopkgtest of pythonscipy fails in
>> testing when that autopkgtest is run with the binary packages of blis
>> from unstable. It passes when run with only packages from testing. In
>> tabular form:
>> pass fail
>> blis from testing 0.5.15
>> pythonscipy from testing 1.1.02
>> all others from testing from testing
>
> From testing's test logs [1], pythonscipy 1.1.02 passed with
> blis/0.5.15 on 20190121 05:22:48 UTC. It then passed with
> blis/0.5.16 and blis/0.5.17, but failed with blis/0.5.17 on
> 20190127 17:55:32 UTC.
>
> From unstable's test logs [2], pythonscipy 1.1.02 seems to have many
> intermittent failures since 20181202.
>
> I'm unsure what to do with this bug, perhaps reassign to pythonscipy
> only?
I am confused, scipy does not use blis.
It tests libblas, openblas and atlas but blis was so far I know never
configured as a blas source in the tests.
Unless something has changed with those packages that they pull blis
this might be a false positive.
If blis is ready enough to be used prime time, let me know we can add it
to numpy's and scipy's testsuites.


reassing 919929 src:pythonscipy 1.1.02
retitle 919929 pythonscipy: autopkgtest fails intermittently
user [hidden email]
usertags 919929  breaks needsupdate
usertags 919929 flaky
severity 919929 important
thanks
Hi Graham, Julian,
On 29012019 18:59, Julian Taylor wrote:
> On 29.01.19 12:01, Graham Inggs wrote:
>> On 2019/01/20 21:09, Paul Gevers wrote:
>> From testing's test logs [1], pythonscipy 1.1.02 passed with
>> blis/0.5.15 on 20190121 05:22:48 UTC. It then passed with
>> blis/0.5.16 and blis/0.5.17, but failed with blis/0.5.17 on
>> 20190127 17:55:32 UTC.
>>
>> From unstable's test logs [2], pythonscipy 1.1.02 seems to have many
>> intermittent failures since 20181202.
That is what we call flaky behavior.
>> I'm unsure what to do with this bug, perhaps reassign to pythonscipy
>> only?
Done so.
> I am confused, scipy does not use blis.
Indeed, and in the failing log one can search for "blis" and not find it
at all, except for the call to autopkgtest. So the failure can't be
caused by blis.
> It tests libblas, openblas and atlas but blis was so far I know never
> configured as a blas source in the tests.
> Unless something has changed with those packages that they pull blis
> this might be a false positive.
Yes it is. Julian, can you please investigate the failing log files and
please make the tests robust against whatever goes wrong. If you really
can't fix it and still want the tests to run, consider adding the
"flaky" restriction to the tests that intermittently fail.
> If blis is ready enough to be used prime time, let me know we can add it
> to numpy's and scipy's testsuites.
No idea, and no opinion about that.
Paul


Package: pythonscipy
FollowupFor: Bug #919929
Something weird going on with the scipy tests. There's a clue in this
bug report already: "MemoryError".
Fails are now consistent since 25/2/2019. The funny thing is, even
when I try to access the test log for the last successful
migrationreference (10/2/2019) at
https://ci.debian.net/data/autopkgtest/testing/amd64/p/pythonscipy/1902804/log.gzaccessing that log crashes my browser tab.
Maybe there's just too much output from deprecation warnings, so all
available memory is being used up, crashing the tests. The gzipped
log file is 3.4 MB, which is not so large really. Are there memory
limits on debci tests?
Downloading with wget and reading from the filesystem, it's reporting
an excessive amount of the repeated warning from numpy:
/usr/lib/python2.7/distpackages/numpy/matrixlib/defmatrix.py:71: PendingDeprecationWarning: the matrix subclass is not the recommended way to represent matrices or deal with linear algebra (see https://docs.scipy.org/doc/numpy/user/numpyformatlabusers.html). Please adjust your code to use regular ndarray.
Could be an argument for allowing scipy 1.2.1 into unstable and testing?
(assuming scipy 1.2 has had matrix handling updated)
But probably a better resolution in the context of the release freeze
is to instruct pytest to ignore PendingDeprecationWarning. Certainly
the warning is obfuscating further inspection.


Package: pythonscipy
FollowupFor: Bug #919929
Note PendingDeprecationWarning is set to be ignored in pytest.ini.
The problem is that the tests in debian/tests do not use pytest...


Package: pythonscipy
FollowupFor: Bug #919929
No, I am wrong. Tests are pytest, the junit reference is junitxml,
indicated test reports are in junit format.
The question then, is why is the instruction in pytest.ini to ignore
deprecation warnings not being honoured?
Ah I see it. I'm looking at the source for 1.2.1. We'll need to apply
something like upstream patch 927ddf389998767 / 1955c60c0833 (if not
614847c5fc8) to stop the warnings.
Drew


Package: pythonscipy
FollowupFor: Bug #919929
Probably a simple pytest.ini diff between scipy 1.1 and 1.2 makes the
simplest patch:
 pytest.ini.old 20190305 18:36:51.532842277 +0800
+++ pytest.ini 20190228 11:57:04.155637459 +0800
@@ 7,5 +7,8 @@
error
always::scipy._lib._testutils.FPUModeChangeWarning
once:.*LAPACK bug 0038.*:RuntimeWarning
+ ignore:the matrix subclass is not the recommended way*:PendingDeprecationWarning
+ ignore:Using or importing the ABCs from 'collections'*:DeprecationWarning
+ ignore:can't resolve package from __spec__ or __package__, falling back on __name__ and __path__:ImportWarning
env =
PYTHONHASHSEED=0


Hi Drew,
On 07032019 09:03, Andreas Tille wrote:
> On Tue, Mar 05, 2019 at 07:01:54PM +0800, Drew Parsons wrote:
>> pythonscipy has recently started failing all debci tests in testing and
>> unstable, exacerbating the bug report in Bug#919929 [1].
>>
>> The failing error is a MemoryError. But understanding the problem is
>> hampered by a flood of deprecation warnings, presumably triggered by numpy
>> 1.16. scipy 1.2 added instructions to pytest.ini to ignore the warnings.
>>
>> Bug#919929 has not yet been marked RC but I guess it's about to happen.
Can you elaborate why you think that bug should be RC (as that isn't
clear to me from the report itself) and why you haven't marked it as
such if you think it should be?
>> I
>> propose in the first instance to apply a patch of the pytest.ini diff
>> between scipy 1.1 and 1.2 and see if that clears things up. I'll commit the
>> patch now. Should I proceed with upload to unstable?
>
> May be its sensible to coordinate with debianrelease (list in CC) and
> file a unblock request *before* uploading. I confirm that I usually
> upload simple fixes and ask for unblock afterwards but you intend to do
> a version change which does not qualify as simple fix (despite I agree
> with you that it is sensible).
Slightly depending on the answer above, I'll unblock an upload of
pythonscipy with only that change.
Paul


On 20190307 19:00, Paul Gevers wrote:
> Hi Drew,
>
> On 07032019 09:03, Andreas Tille wrote:
>> On Tue, Mar 05, 2019 at 07:01:54PM +0800, Drew Parsons wrote:
>>> pythonscipy has recently started failing all debci tests in testing
>>> and
>>> unstable, exacerbating the bug report in Bug#919929 [1].
>>>
>>> The failing error is a MemoryError. But understanding the problem is
>>> hampered by a flood of deprecation warnings, presumably triggered by
>>> numpy
>>> 1.16. scipy 1.2 added instructions to pytest.ini to ignore the
>>> warnings.
>>>
>>> Bug#919929 has not yet been marked RC but I guess it's about to
>>> happen.
>
> Can you elaborate why you think that bug should be RC (as that isn't
> clear to me from the report itself) and why you haven't marked it as
> such if you think it should be?
pythonscipy is currently failing all debci tests in both unstable and
testing.
I haven't marked it RC myself since I'm not 100% certain what the usual
protocol is for marking the severity of debci test failures. But as I
understood it, debci test failures is considered RC under the final
freeze which we're about to enter (but we're not quite in that deep
freeze yet).
>>> I
>>> propose in the first instance to apply a patch of the pytest.ini diff
>>> between scipy 1.1 and 1.2 and see if that clears things up. I'll
>>> commit the
>>> patch now. Should I proceed with upload to unstable?
>>
>> May be its sensible to coordinate with debianrelease (list in CC) and
>> file a unblock request *before* uploading. I confirm that I usually
>> upload simple fixes and ask for unblock afterwards but you intend to
>> do
>> a version change which does not qualify as simple fix (despite I agree
>> with you that it is sensible).
Hi Andreas, I may not have been clear. What I mean at this point is to
upload a small patch for scipy 1.1 to ignore the deprecation warnings
(scipy 1.2 already does that).
If that doesn't help pass the debci tests then we can consider uploading
scipy 1.2 instead. But pythonfluids and dipy FTBFS against scipy 1.2
(a new version of dipy is available which presumeably fixes that)
> Slightly depending on the answer above, I'll unblock an upload of
> pythonscipy with only that change.
Certainly please do unblock if the full freeze is already in place. But
my intention was to first upload pythonscipy 1.1 with a small patch,
not 1.2 just yet.
Drew


Hi Drew,
On 07032019 13:19, Drew Parsons wrote:
>> Can you elaborate why you think that bug should be RC (as that isn't
>> clear to me from the report itself) and why you haven't marked it as
>> such if you think it should be?
>
> pythonscipy is currently failing all debci tests in both unstable and
> testing.
autopkgtest only, so no FTBFS?
> I haven't marked it RC myself since I'm not 100% certain what the usual
> protocol is for marking the severity of debci test failures. But as I
> understood it, debci test failures is considered RC under the final
> freeze which we're about to enter (but we're not quite in that deep
> freeze yet).
Some of us want failing autopkgtest to be RC *after* the release of
buster. I am not aware of consensus about that yet. autopkgtest
*regression* in testing is effectively RC since the soft freeze of
1222019. The autopkgtest of pythonscipy is already failing in
testing, so it isn't a regression. Hence, it failing is not RC for buster.
> Hi Andreas, I may not have been clear. What I mean at this point is to
> upload a small patch for scipy 1.1 to ignore the deprecation warnings
> (scipy 1.2 already does that).
>
> If that doesn't help pass the debci tests then we can consider uploading
> scipy 1.2 instead. But pythonfluids and dipy FTBFS against scipy 1.2
> (a new version of dipy is available which presumeably fixes that)
Please *don't* upload scipy 1.2 to fix only this issue (nor for any
other issue without discussion first), it's for sure not worth it.
>> Slightly depending on the answer above, I'll unblock an upload of
>> pythonscipy with only that change.
>
> Certainly please do unblock if the full freeze is already in place. But
> my intention was to first upload pythonscipy 1.1 with a small patch,
> not 1.2 just yet.
If you upload now, your package will not migrate to testing before the
full freeze becomes effective so it would need an unblock. If you want
to fix this issue with the three lines I saw in the bug report, you can
go ahead. However, it is probably worth waiting for a resolution of bug
915738 and combine it with that.
Paul


On 20190307 20:46, Paul Gevers wrote:
> Hi Drew,
>
> On 07032019 13:19, Drew Parsons wrote:
>>
>> pythonscipy is currently failing all debci tests in both unstable and
>> testing.
>
> autopkgtest only, so no FTBFS?
That's right, scipy builds fine.
> Some of us want failing autopkgtest to be RC *after* the release of
> buster. I am not aware of consensus about that yet. autopkgtest
> *regression* in testing is effectively RC since the soft freeze of
> 1222019. The autopkgtest of pythonscipy is already failing in
> testing, so it isn't a regression. Hence, it failing is not RC for
> buster.
Thanks, that clarifies the appropriate severity.
>> Certainly please do unblock if the full freeze is already in place.
>> But
>> my intention was to first upload pythonscipy 1.1 with a small patch,
>> not 1.2 just yet.
>
> If you upload now, your package will not migrate to testing before the
> full freeze becomes effective so it would need an unblock. If you want
> to fix this issue with the three lines I saw in the bug report, you can
> go ahead. However, it is probably worth waiting for a resolution of bug
> 915738 and combine it with that.
There hasn't been recent movement on 915738. I'll apply Julian's patch
and see how we go.
Drew


On 20190307 20:46, Paul Gevers wrote:
>
> If you upload now, your package will not migrate to testing before the
> full freeze becomes effective so it would need an unblock. If you want
> to fix this issue with the three lines I saw in the bug report, you can
> go ahead. However, it is probably worth waiting for a resolution of bug
> 915738 and combine it with that.
>
Alas, the deprecation patch (in pythonscipy 1.1.03) doesn't actually
prevent emission of the deprecation warnings, so they're still spamming
the debci log. On the bright side, s390x is now using gfortran8
successfully (#915738).
To remove the deprecation warnings we'd need to deal with them at the
source. Upstream has patches
https://github.com/scipy/scipy/commit/614847c5fc8d5f8a618980df3c1b93540428ae46https://github.com/scipy/scipy/commit/e0cfa29e2fbe86f994924c0c7514ff5bbfffd514and for completeness
https://github.com/scipy/scipy/commit/87e48c3c54d7a85bc6628c88c1de98ac0469b6faThe deprecation problem (matrix API) appears in many places, but the fix
is straightfoward: replace np.matrix with matrix from from
scipy.sparse.sputils
Can you authorise an unblock to apply these 3 upstream patches to
pythonscipy 1.1.0 ?
That won't necessarily fix the debci failure, but it will make it a lot
easier to see what's actually failing.
Drew


Hi Drew,
On 08032019 03:08, Drew Parsons wrote:
> On 20190307 20:46, Paul Gevers wrote:
>> If you upload now, your package will not migrate to testing before the
>> full freeze becomes effective so it would need an unblock. If you want
>> to fix this issue with the three lines I saw in the bug report, you can
>> go ahead. However, it is probably worth waiting for a resolution of bug
>> 915738 and combine it with that.
>>
>
> Alas, the deprecation patch (in pythonscipy 1.1.03) doesn't actually
> prevent emission of the deprecation warnings, so they're still spamming
> the debci log.
Do you have evidence they did anything at all? If not, please revert
this if we get to a new upload.
> On the bright side, s390x is now using gfortran8
> successfully (#915738).
Well, at least some success then.
I am slightly unhappy with the second patch, as it seems to be doing
more than you describe above in a few places. This may be correct but
that is difficult to quickly judge.
Also, what is the general documented way that these wrappers can be
used? scipy is sort of taking over the maintenanceship of these
functions in this way if we're not careful.
Paul


On 20190310 03:51, Paul Gevers wrote:
> Hi Drew,
>
> On 08032019 03:08, Drew Parsons wrote:
>> On 20190307 20:46, Paul Gevers wrote:
>>> If you upload now, your package will not migrate to testing before
>>> the
>>> full freeze becomes effective so it would need an unblock. If you
>>> want
>>> to fix this issue with the three lines I saw in the bug report, you
>>> can
>>> go ahead. However, it is probably worth waiting for a resolution of
>>> bug
>>> 915738 and combine it with that.
>>>
>>
>> Alas, the deprecation patch (in pythonscipy 1.1.03) doesn't actually
>> prevent emission of the deprecation warnings, so they're still
>> spamming
>> the debci log.
>
> Do you have evidence they did anything at all? If not, please revert
> this if we get to a new upload.
It would seem it did not help. In any case, the upstream patches
supercede this patch, so it will be removed naturally.
The patches as they are don't apply cleanly to the 1.1.0 source, so I'll
need to adapt them anyway. I can retain only the ones relevant to
updating the matrix API.
> Also, what is the general documented way that these wrappers can be
> used? scipy is sort of taking over the maintenanceship of these
> functions in this way if we're not careful.
It's a good question that the other scipy maintainers might have thought
more about. As far as I can tell, the scipy tests affected here involve
sparse matrices. The trouble arises from an "inadequacy" in the core
numpy API, with numpy.matrix only being suitable for dense matrices.
scipy could be described as "numpy+algorithms", with additional
algorithms required to handle sparse matrices, provided in
scipy.sparse.sputils.matrix.
numpy.matrix is documented at
https://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.htmlThe scipy sparse matrix API is at
https://docs.scipy.org/doc/scipy/reference/sparse.html, but that's
specifically for scipy.sparse.spmatrix
There is discussion of the distinction between numpy.matrix and
numpy.ndarray (which is at the heart of the deeprecation warning) at
https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html#numpymatrixvs2dnumpyndarrayThe utility class scipy.sparse.sputils itself is apparently
undocumented, by which I infer it's intended for internal use only, not
a public API. I guess it's reasonable for a package to be testing it's
own internal functions. Strange thing is, scipy.sparse.sputils.matrix
is not actually defined in scipy/sparse/sputils.py. Must be inherited or
defined in some deep pythonfu that I haven't mastered yet.
I'll check that I can adapt those upstream patches to cleanly remove
these deprecation warnings.
Drew


On 20190310 15:46, Drew Parsons wrote:
...
>>> Can you authorise an unblock to apply these 3 upstream patches to
>>> pythonscipy 1.1.0 ?
>>> That won't necessarily fix the debci failure, but it will make it a
>>> lot
>>> easier to see what's actually failing.
>>
>> I am slightly unhappy with the second patch, as it seems to be doing
>> more than you describe above in a few places. This may be correct but
>> that is difficult to quickly judge.
I've adapted the 3 patches and pushed to salsa,
matrix_API_614847c5.patch
matrix_API_more_e0cfa29e2.patch
matrix_API_filter_check_87e48c3c5.patch
https://salsa.debian.org/pythonteam/modules/pythonscipy/tree/master/debian/patchesThe other behaviour that you saw in the second patch I think might be
replacement of "D*diag(v)" with "D@diag(v)". That's matrix
multiplication, so still relevent to the matrix API patch. @ is not
available in python2 so I changed it to numpy.matmul, which does the
same thing.
The numpy.sparse tests pass with this patch, and most of the matrix
PendingDeprecationWarnings are gone (the upstream patch missed
integrate/tests/test_ivp.py, but the remaining warnings are few enough
to not need to worry about).
>> Also, what is the general documented way that these wrappers can be
>> used? scipy is sort of taking over the maintenanceship of these
>> functions in this way if we're not careful.
>
...
> There is discussion of the distinction between numpy.matrix and
> numpy.ndarray (which is at the heart of the deeprecation warning) at
> https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html#numpymatrixvs2dnumpyndarray>
> The utility class scipy.sparse.sputils itself is apparently
> undocumented, by which I infer it's intended for internal use only,
> not a public API. I guess it's reasonable for a package to be testing
> it's own internal functions. Strange thing is,
> scipy.sparse.sputils.matrix is not actually defined in
> scipy/sparse/sputils.py. Must be inherited or defined in some deep
> pythonfu that I haven't mastered yet.
Actually scipy.sparse.sputils.matrix was defined in these patches. It
is a bit odd that upstream is wrapping numpy.matrix just to avoid
deprecation warnings, rather than following their own advice and using
numpy.ndarray instead.
With these patches, the sparse matrix tests pass. There remain three
errors unrelated to sparse matrix:
spatial/tests/test__plotutils.py::TestPlotting::test_delaunay FAILED
[ 76%]
spatial/tests/test__plotutils.py::TestPlotting::test_voronoi FAILED
[ 76%]
spatial/tests/test__plotutils.py::TestPlotting::test_convex_hull
FAILED [ 76%]
__________________________ TestPlotting.test_delaunay
__________________________
self = <scipy.spatial.tests.test__plotutils.TestPlotting object at
0x7f0f31156eb8>
def test_delaunay(self):
# Smoke test
fig = plt.figure()
obj = Delaunay(self.points)
s_before = obj.simplices.copy()
> with suppress_warnings as sup:
E AttributeError: __enter__
fig = <Figure size 640x480 with 0 Axes>
obj = <scipy.spatial.qhull.Delaunay object at 0x7f0f31163400>
s_before = array([[3, 1, 0],
[2, 3, 0]], dtype=int32)
self = <scipy.spatial.tests.test__plotutils.TestPlotting object
at 0x7f0f31156eb8>
/usr/lib/python3/distpackages/scipy/spatial/tests/test__plotutils.py:31:
AttributeError
(likewise for the other 2)
These AttributeErrors are mentioned at
https://github.com/scipy/scipy/issues/9491. Upstream doesn't seem too
concerned about them.
Apart from that, I'm happy to upload the sparse matrix patches once the
s390x update reaches testing.
Drew


On 20190311 14:39, Drew Parsons wrote:
>
> I've adapted the 3 patches and pushed to salsa,
> matrix_API_614847c5.patch
> matrix_API_more_e0cfa29e2.patch
> matrix_API_filter_check_87e48c3c5.patch
> https://salsa.debian.org/pythonteam/modules/pythonscipy/tree/master/debian/patches...
> The numpy.sparse tests pass with this patch, and most of the matrix
> PendingDeprecationWarnings are gone (the upstream patch missed
> integrate/tests/test_ivp.py, but the remaining warnings are few enough
> to not need to worry about).
Well, turns out the other warnings worried Aurelien enough to file
Bug#924396.
Is there enough will to add more scipy patches for the buster release to
reduce the remaining DeprecationWarnings? (they don't break tests,
they're just annoying)
Or should we just let it go at this point and let them get cleared in
future versions?)
...
>
> With these patches, the sparse matrix tests pass. There remain three
> errors unrelated to sparse matrix:
> spatial/tests/test__plotutils.py::TestPlotting::test_delaunay FAILED
> [ 76%]
> spatial/tests/test__plotutils.py::TestPlotting::test_voronoi FAILED
> [ 76%]
> spatial/tests/test__plotutils.py::TestPlotting::test_convex_hull
> FAILED [ 76%]
,,,
> > with suppress_warnings as sup:
> E AttributeError: __enter__
>
> Apart from that, I'm happy to upload the sparse matrix patches once
> the s390x update reaches testing.
Those errors must have been local to me. scipy 1.1.04 now passes debci
tests cleanly.
That being the case, in the interests of making a stable release that
passes it own tests, I'd like to request an unblock for
pythonscipy/1.1.04 (together with skimage/0.14.22)
Drew


Hi Drew,
On 16032019 13:48, Drew Parsons wrote:
>> The numpy.sparse tests pass with this patch, and most of the matrix
>> PendingDeprecationWarnings are gone (the upstream patch missed
>> integrate/tests/test_ivp.py, but the remaining warnings are few enough
>> to not need to worry about).
>
> Well, turns out the other warnings worried Aurelien enough to file
> Bug#924396.
>
> Is there enough will to add more scipy patches for the buster release to
> reduce the remaining DeprecationWarnings? (they don't break tests,
> they're just annoying)
> Or should we just let it go at this point and let them get cleared in
> future versions?)
I'd let it be for now.
> That being the case, in the interests of making a stable release that
> passes it own tests, I'd like to request an unblock for
> pythonscipy/1.1.04 (together with skimage/0.14.22)
skimage was already unblocked. I'll unblock pythonscipy as well.
Paul


On 20190316 22:07, Paul Gevers wrote:
> On 16032019 13:48, Drew Parsons wrote:
>>
>> Is there enough will to add more scipy patches for the buster release
>> to
>> reduce the remaining DeprecationWarnings? (they don't break tests,
>> they're just annoying)
>> Or should we just let it go at this point and let them get cleared in
>> future versions?)
>
> I'd let it be for now.
No problem, will do (more precisely, not do).
>> That being the case, in the interests of making a stable release that
>> passes it own tests, I'd like to request an unblock for
>> pythonscipy/1.1.04 (together with skimage/0.14.22)
>
> skimage was already unblocked. I'll unblock pythonscipy as well.
Thanks Paul. Looks like we'll be able to close Bug#919929 once
pythonscipy/1.1.04 is settled into testing.
Drew

12
