Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Christian Ehrhardt
Package: rdma-core
Version: 22.1-1
X-Debbugs-CC: [hidden email], [hidden email]

FYI on CC I have put some more people:
CC: Luca who maintains DPDK with me
CC: Thomas for DPDK<->Mellanox knowledge that could help

Hi,
this might be more a request to understand the correct dependencies
than a bug (unless you say it is one). But before jumping to
conclusions let me outline the issue.

We were asked [1] on package DPDK to also install rdma-core as the
packages for MLX[4/5] would not work otherwise. We checked our
packages and e.g. librte-pmd-mlx5-18.11 depends (due to shlibs magic)
on ibverbs-providers and libibverbs1.
And we thought that would be fine.

Now after being asked/told that without rdma-core this would be
useless we were wondering.
rdma-core is essentially a bunch of module config/autoloaders and
rdma-ndd/iwpmd services.
Right now we can only assume this is what librte-pmd-mlx5-18.11 needs,
as we are lacking some details.

Therefore the following questions:
@rdma-core team
- does ibverbs1/ibverbs-providers generally need rdma-core to be useful?
- if so why isn't it at least a recommends from those packages
@Thomas
- do you happen to know what exactly of rdma-core that librte-pmd-mlx[45] need?

looking forward to your reply, as I'm happy to learn more about the
dependencies among those packages.

[1]: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1820957 (I
created that bug from a mail thread)

--
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Thomas Monjalon
20/03/2019 12:51, Christian Ehrhardt:
> @rdma-core team
> - does ibverbs1/ibverbs-providers generally need rdma-core to be useful?

ibverbs is provided by rdma-core project.
Is it a separate package in your case?

> - if so why isn't it at least a recommends from those packages

> @Thomas
> - do you happen to know what exactly of rdma-core that librte-pmd-mlx[45] need?

Yes, you need libmlx4, libmlx5 and libibverbs.
It is described in the doc:
        http://doc.dpdk.org/guides/nics/mlx5.html#prerequisites

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Christian Ehrhardt
> ibverbs is provided by rdma-core project.
> Is it a separate package in your case?

No it is part of the same src package.

> Yes, you need libmlx4, libmlx5 and libibverbs.
> It is described in the doc:
>         http://doc.dpdk.org/guides/nics/mlx5.html#prerequisites

Well that is the problem, we fulfill that prereq already.
libmlx4, libmlx5 and libibverbs are installed by dependencies already
and thought all the time that would be good enough.

But we got told that we also need the package rdma-core [1] itself to
be installed to be useful.
That was a discussion with the subject "Please help us to verify DPDK
17.11.5 for Ubuntu 18.04/18.10" with Mellanox test people.
Fortunately I keep you on CC on those - so maybe you can find it
somewhere in your inbox and contact the other Mellanox people on that
thread.
After that you might help to translate their request into our terms please?

The question is:
a) what exactly is (binary package) rdma-core [1] needed for in that scenario
b) if (a) would be generally true we wondered why it had to be a dep
of the DPDK packages and not of e.g. libibverbs1 itself
That is why we opened this bug for discussions

[1]: https://packages.debian.org/sid/rdma-core

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Brian T. Smith
On Thu, Mar 21, 2019 at 3:51 AM Christian Ehrhardt
<[hidden email]> wrote:

>
> > ibverbs is provided by rdma-core project.
> > Is it a separate package in your case?
>
> No it is part of the same src package.
>
> > Yes, you need libmlx4, libmlx5 and libibverbs.
> > It is described in the doc:
> >         http://doc.dpdk.org/guides/nics/mlx5.html#prerequisites
>
> Well that is the problem, we fulfill that prereq already.
> libmlx4, libmlx5 and libibverbs are installed by dependencies already
> and thought all the time that would be good enough.
>
> But we got told that we also need the package rdma-core [1] itself to
> be installed to be useful.
> That was a discussion with the subject "Please help us to verify DPDK
> 17.11.5 for Ubuntu 18.04/18.10" with Mellanox test people.
> Fortunately I keep you on CC on those - so maybe you can find it
> somewhere in your inbox and contact the other Mellanox people on that
> thread.
> After that you might help to translate their request into our terms please?
>
> The question is:
> a) what exactly is (binary package) rdma-core [1] needed for in that scenario
> b) if (a) would be generally true we wondered why it had to be a dep
> of the DPDK packages and not of e.g. libibverbs1 itself
> That is why we opened this bug for discussions
>
> [1]: https://packages.debian.org/sid/rdma-core
>

rdma-core configures modprobe for ib/rdma modules, installs some
helper daemons and installs some udev rules. It certainly makes
administration of the server an easier task, but it isn't *required*
when configuring a Debian server for rdma and user-space ibverbs.

Perhaps libibverbs should recommend rdma-core?

--
Brian T. Smith
System Fabric Works
Senior Technical Staff
[hidden email]
GPG Key: 0xB3C2C7B73BA3CD7F

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Jason Gunthorpe-3
On Thu, Mar 21, 2019 at 01:26:04PM -0500, Brian Smith wrote:

> On Thu, Mar 21, 2019 at 3:51 AM Christian Ehrhardt
> <[hidden email]> wrote:
> >
> > > ibverbs is provided by rdma-core project.
> > > Is it a separate package in your case?
> >
> > No it is part of the same src package.
> >
> > > Yes, you need libmlx4, libmlx5 and libibverbs.
> > > It is described in the doc:
> > >         http://doc.dpdk.org/guides/nics/mlx5.html#prerequisites
> >
> > Well that is the problem, we fulfill that prereq already.
> > libmlx4, libmlx5 and libibverbs are installed by dependencies already
> > and thought all the time that would be good enough.
> >
> > But we got told that we also need the package rdma-core [1] itself to
> > be installed to be useful.
> > That was a discussion with the subject "Please help us to verify DPDK
> > 17.11.5 for Ubuntu 18.04/18.10" with Mellanox test people.
> > Fortunately I keep you on CC on those - so maybe you can find it
> > somewhere in your inbox and contact the other Mellanox people on that
> > thread.
> > After that you might help to translate their request into our terms please?
> >
> > The question is:
> > a) what exactly is (binary package) rdma-core [1] needed for in that scenario
> > b) if (a) would be generally true we wondered why it had to be a dep
> > of the DPDK packages and not of e.g. libibverbs1 itself
> > That is why we opened this bug for discussions
> >
> > [1]: https://packages.debian.org/sid/rdma-core
> >
>
> rdma-core configures modprobe for ib/rdma modules, installs some
> helper daemons and installs some udev rules. It certainly makes
> administration of the server an easier task, but it isn't *required*
> when configuring a Debian server for rdma and user-space ibverbs.

I didn't think it is required for mlx5 as the ethernet driver
autoloads the rdma modules itself..

At least, I wouldn't make rdma-core a hard dependency of dpdk.. At
worst that should be in libibverbs, as you say.

Jason

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Christian Ehrhardt
> >
> > rdma-core configures modprobe for ib/rdma modules, installs some
> > helper daemons and installs some udev rules. It certainly makes
> > administration of the server an easier task, but it isn't *required*
> > when configuring a Debian server for rdma and user-space ibverbs.
>
> I didn't think it is required for mlx5 as the ethernet driver
> autoloads the rdma modules itself..

Below is the log that I got from Mellanox (thanks Alaa and Thomas.
To me the messages related to ib_uverbs seem interesting.

@Jason - this is triggered by DPDK. In regard to the autoloading - did
you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g.
mlx5_core.ko?
I only found [1] which does not seem right, can you point at the
autoloading feature (a commit maybe) that you meant (keep in mind this
is DPDK 17.11.5)?

@Allaa/Thomas - due to the messages in the log, could you try if
instead of installing rdma-core just doing the modprobe would be
enough to resolve the problem?
  $ modprobe ib_uverbs # instead of the apt install

[1]: https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options

FYI - The test log from Mellanox showing that they need rdma-core to
run properly:

1 - Running testpmd with installing any thing
root@qa-l-vrt-111-124:~# testpmd -c 0xF -n 4 -w 0000:00:06.0 -w
0000:00:07.0 -- --txd=256 --rxd=256 --rxq=1 --txq=1 --nb-cores=3 --i
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
unreliable clock cycles !
EAL: PCI device 0000:00:06.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15b3:1016 net_mlx5
net_mlx5: cannot list devices, is ib_uverbs loaded?
EAL: Requested device 0000:00:06.0 cannot be used
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15b3:1016 net_mlx5
net_mlx5: cannot list devices, is ib_uverbs loaded?
EAL: Requested device 0000:00:07.0 cannot be used
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
size=2176, socket=0
Done

2- Installing rdma-core
root@qa-l-vrt-111-124:~# apt-get install rdma-core
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  rdma-core
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 56.8 kB of archives.
After this operation, 195 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe
amd64 rdma-core amd64 17.1-1ubuntu0.1 [56.8 kB]
Fetched 56.8 kB in 1s (79.4 kB/s)
Selecting previously unselected package rdma-core.
(Reading database ... 91266 files and directories currently installed.)
Preparing to unpack .../rdma-core_17.1-1ubuntu0.1_amd64.deb ...
Unpacking rdma-core (17.1-1ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for systemd (237-3ubuntu10.15) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up rdma-core (17.1-1ubuntu0.1) ...
rdma-hw.target is a disabled or a static unit, not starting it.
rdma-ndd.service is a disabled or a static unit, not starting it.
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for systemd (237-3ubuntu10.15) ...

3- Running testpmd again after installing rdma-core
root@qa-l-vrt-111-124:~# testpmd -c 0xF -n 4 -w 0000:00:06.0 -w
0000:00:07.0 -- --txd=256 --rxd=256 --rxq=1 --txq=1 --nb-cores=3 --i
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
unreliable clock cycles !
EAL: PCI device 0000:00:06.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15b3:1016 net_mlx5
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15b3:1016 net_mlx5
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
size=2176, socket=0
Configuring Port 0 (socket 0)
Port 0: 26:D4:B6:45:70:D7
Configuring Port 1 (socket 0)
Port 1: BA:52:9F:30:6E:BF
Checking link statuses...
Done

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Thomas Monjalon
26/03/2019 10:21, Christian Ehrhardt:

> > >
> > > rdma-core configures modprobe for ib/rdma modules, installs some
> > > helper daemons and installs some udev rules. It certainly makes
> > > administration of the server an easier task, but it isn't *required*
> > > when configuring a Debian server for rdma and user-space ibverbs.
> >
> > I didn't think it is required for mlx5 as the ethernet driver
> > autoloads the rdma modules itself..
>
> Below is the log that I got from Mellanox (thanks Alaa and Thomas.
> To me the messages related to ib_uverbs seem interesting.
>
> @Jason - this is triggered by DPDK. In regard to the autoloading - did
> you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g.
> mlx5_core.ko?
> I only found [1] which does not seem right, can you point at the
> autoloading feature (a commit maybe) that you meant (keep in mind this
> is DPDK 17.11.5)?
> [1]: https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options

There is no autoloading of kernel drivers when using DPDK.

> @Allaa/Thomas - due to the messages in the log, could you try if
> instead of installing rdma-core just doing the modprobe would be
> enough to resolve the problem?
>   $ modprobe ib_uverbs # instead of the apt install

Yes, manual modprobe partially replaces rdma-core packages.

These are the files we identified as required from the rdma-core packages:
        /etc/modprobe.d/mlx4.conf
        /etc/rdma/modules/rdma.conf
        /lib/udev/rules.d/90-rdma-hw-modules.rules
        /lib/systemd/system/rdma-hw.target

The first file (mlx4.conf) is a template helping to get the right config
by commenting out a line.
The others files are for autoloading of the kernel drivers.

I think you should include the rdma-core package when using DPDK.
It can be a dependency of the DPDK package, or a dependency of libibverbs.
Does it make sense to install libibverbs without rdma-core configuration?

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Jason Gunthorpe-3
On Tue, Mar 26, 2019 at 02:51:25PM +0100, Thomas Monjalon wrote:

> 26/03/2019 10:21, Christian Ehrhardt:
> > > >
> > > > rdma-core configures modprobe for ib/rdma modules, installs some
> > > > helper daemons and installs some udev rules. It certainly makes
> > > > administration of the server an easier task, but it isn't *required*
> > > > when configuring a Debian server for rdma and user-space ibverbs.
> > >
> > > I didn't think it is required for mlx5 as the ethernet driver
> > > autoloads the rdma modules itself..
> >
> > Below is the log that I got from Mellanox (thanks Alaa and Thomas.
> > To me the messages related to ib_uverbs seem interesting.
> >
> > @Jason - this is triggered by DPDK. In regard to the autoloading - did
> > you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g.
> > mlx5_core.ko?
> > I only found [1] which does not seem right, can you point at the
> > autoloading feature (a commit maybe) that you meant (keep in mind this
> > is DPDK 17.11.5)?
> > [1]: https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options
>
> There is no autoloading of kernel drivers when using DPDK.

It looks like it is only done if the mlx5 NIC driver detects some kind
of RDMA mode..

> > @Allaa/Thomas - due to the messages in the log, could you try if
> > instead of installing rdma-core just doing the modprobe would be
> > enough to resolve the problem?
> >   $ modprobe ib_uverbs # instead of the apt install
>
> Yes, manual modprobe partially replaces rdma-core packages.
>
> These are the files we identified as required from the rdma-core packages:
> /etc/modprobe.d/mlx4.conf
> /etc/rdma/modules/rdma.conf
> /lib/udev/rules.d/90-rdma-hw-modules.rules
> /lib/systemd/system/rdma-hw.target
>
> The first file (mlx4.conf) is a template helping to get the right config
> by commenting out a line.
> The others files are for autoloading of the kernel drivers.
>
> I think you should include the rdma-core package when using DPDK.
> It can be a dependency of the DPDK package, or a dependency of libibverbs.
> Does it make sense to install libibverbs without rdma-core configuration?

Many rdma drivers do properly autoload and don't need the udev magic.

I have a long term plan to avoid needing this, but nobody is working
on it right now.

Jason

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Thomas Monjalon
26/03/2019 19:05, Jason Gunthorpe:

> On Tue, Mar 26, 2019 at 02:51:25PM +0100, Thomas Monjalon wrote:
> > 26/03/2019 10:21, Christian Ehrhardt:
> > > > >
> > > > > rdma-core configures modprobe for ib/rdma modules, installs some
> > > > > helper daemons and installs some udev rules. It certainly makes
> > > > > administration of the server an easier task, but it isn't *required*
> > > > > when configuring a Debian server for rdma and user-space ibverbs.
> > > >
> > > > I didn't think it is required for mlx5 as the ethernet driver
> > > > autoloads the rdma modules itself..
> > >
> > > Below is the log that I got from Mellanox (thanks Alaa and Thomas.
> > > To me the messages related to ib_uverbs seem interesting.
> > >
> > > @Jason - this is triggered by DPDK. In regard to the autoloading - did
> > > you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g.
> > > mlx5_core.ko?
> > > I only found [1] which does not seem right, can you point at the
> > > autoloading feature (a commit maybe) that you meant (keep in mind this
> > > is DPDK 17.11.5)?
> > > [1]: https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options
> >
> > There is no autoloading of kernel drivers when using DPDK.
>
> It looks like it is only done if the mlx5 NIC driver detects some kind
> of RDMA mode..
>
> > > @Allaa/Thomas - due to the messages in the log, could you try if
> > > instead of installing rdma-core just doing the modprobe would be
> > > enough to resolve the problem?
> > >   $ modprobe ib_uverbs # instead of the apt install
> >
> > Yes, manual modprobe partially replaces rdma-core packages.
> >
> > These are the files we identified as required from the rdma-core packages:
> > /etc/modprobe.d/mlx4.conf
> > /etc/rdma/modules/rdma.conf
> > /lib/udev/rules.d/90-rdma-hw-modules.rules
> > /lib/systemd/system/rdma-hw.target
> >
> > The first file (mlx4.conf) is a template helping to get the right config
> > by commenting out a line.
> > The others files are for autoloading of the kernel drivers.
> >
> > I think you should include the rdma-core package when using DPDK.
> > It can be a dependency of the DPDK package, or a dependency of libibverbs.
> > Does it make sense to install libibverbs without rdma-core configuration?
>
> Many rdma drivers do properly autoload and don't need the udev magic.
>
> I have a long term plan to avoid needing this, but nobody is working
> on it right now.

So I guess the recommendation is to make DPDK depends on rdma-core package.
Luca/Christian, is it possible to do this change in your packaging please?

Reply | Threaded
Open this post in threaded view
|

Bug#925141: Should rdma-core be a dependency of ibverbs-providers or libverb1

Luca Boccassi-3
On Sun, 2019-04-07 at 11:31 +0200, Thomas Monjalon wrote:

> 26/03/2019 19:05, Jason Gunthorpe:
> > On Tue, Mar 26, 2019 at 02:51:25PM +0100, Thomas Monjalon wrote:
> > > 26/03/2019 10:21, Christian Ehrhardt:
> > > > > > rdma-core configures modprobe for ib/rdma modules, installs
> > > > > > some
> > > > > > helper daemons and installs some udev rules. It certainly
> > > > > > makes
> > > > > > administration of the server an easier task, but it isn't
> > > > > > *required*
> > > > > > when configuring a Debian server for rdma and user-space
> > > > > > ibverbs.
> > > > >
> > > > > I didn't think it is required for mlx5 as the ethernet driver
> > > > > autoloads the rdma modules itself..
> > > >
> > > > Below is the log that I got from Mellanox (thanks Alaa and
> > > > Thomas.
> > > > To me the messages related to ib_uverbs seem interesting.
> > > >
> > > > @Jason - this is triggered by DPDK. In regard to the
> > > > autoloading - did
> > > > you refer to DPDKs net_mlx5 or the kernels net_mlx5 in e.g.
> > > > mlx5_core.ko?
> > > > I only found [1] which does not seem right, can you point at
> > > > the
> > > > autoloading feature (a commit maybe) that you meant (keep in
> > > > mind this
> > > > is DPDK 17.11.5)?
> > > > [1]:
> > > > https://doc.dpdk.org/guides/nics/mlx5.html#compilation-options
> > > >
> > >
> > > There is no autoloading of kernel drivers when using DPDK.
> >
> > It looks like it is only done if the mlx5 NIC driver detects some
> > kind
> > of RDMA mode..
> >
> > > > @Allaa/Thomas - due to the messages in the log, could you try
> > > > if
> > > > instead of installing rdma-core just doing the modprobe would
> > > > be
> > > > enough to resolve the problem?
> > > >   $ modprobe ib_uverbs # instead of the apt install
> > >
> > > Yes, manual modprobe partially replaces rdma-core packages.
> > >
> > > These are the files we identified as required from the rdma-core
> > > packages:
> > > /etc/modprobe.d/mlx4.conf
> > > /etc/rdma/modules/rdma.conf
> > > /lib/udev/rules.d/90-rdma-hw-modules.rules
> > > /lib/systemd/system/rdma-hw.target
> > >
> > > The first file (mlx4.conf) is a template helping to get the right
> > > config
> > > by commenting out a line.
> > > The others files are for autoloading of the kernel drivers.
> > >
> > > I think you should include the rdma-core package when using DPDK.
> > > It can be a dependency of the DPDK package, or a dependency of
> > > libibverbs.
> > > Does it make sense to install libibverbs without rdma-core
> > > configuration?
> >
> > Many rdma drivers do properly autoload and don't need the udev
> > magic.
> >
> > I have a long term plan to avoid needing this, but nobody is
> > working
> > on it right now.
>
> So I guess the recommendation is to make DPDK depends on rdma-core
> package.
> Luca/Christian, is it possible to do this change in your packaging
> please?
Yes if the libraries can be used without the rdma-core in the general
case then we can add a dependency via the mlx pmd.

--
Kind regards,
Luca Boccassi

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#925141: reassign to DPDK to resolve as discussed

Christian Ehrhardt
In reply to this post by Christian Ehrhardt
reassign 925141 dpdk