High CPU usage after upgrading from Debian 7 to Debian 9

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

High CPU usage after upgrading from Debian 7 to Debian 9

Алексей-13
Hi!

I have two Debian boxes acting as a routesr (+iptables +quagga(BGP)). I have updated one from Debian 7 to Debian 9 recently and now experience 2x CPU utilization on the same amounts of traffic, e.g. on normal load of 1Gbps (real-world traffic) Debian 7 had 5-6% and Debian 9 has 10-12% softirq CPU time. This behaviour can also be observed during tests - 6Gbps iperf gets 3% on Debian 7 and 6% on Debian 9.

I compared ethtool -k outputs on both boxes and they are pretty similar. Syslog doesn't show anything suspicious.

Did anyone have the similar issue? Or can give me some advice on troubleshooting?

Any help will be appreciated.

Cheers,
Alex
Reply | Threaded
Open this post in threaded view
|

Re: High CPU usage after upgrading from Debian 7 to Debian 9

Matus UHLAR - fantomas
On 15.04.19 11:05, Алексей wrote:
>I have two Debian boxes acting as a routesr (+iptables +quagga(BGP)). I have updated one from Debian 7 to Debian 9 recently and now experience 2x CPU utilization on the same amounts of traffic, e.g. on normal load of 1Gbps (real-world traffic) Debian 7 had 5-6% and Debian 9 has 10-12% softirq CPU time. This behaviour can also be observed during tests - 6Gbps iperf gets 3% on Debian 7 and 6% on Debian 9.
>
>I compared ethtool -k outputs on both boxes and they are pretty similar. Syslog doesn't show anything suspicious.
>
>Did anyone have the similar issue? Or can give me some advice on troubleshooting?

we have noticed something similar due to the spectre/meltdown updates.
Turning off HT cpus did help much.
not installing the CPU microcode should help too.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I intend to live forever - so far so good.

Reply | Threaded
Open this post in threaded view
|

Re[2]: High CPU usage after upgrading from Debian 7 to Debian 9

Алексей-13
Hi!

Thanks for your answer.

I have Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz inside - no HT on it.

And I already have CPU microcode installed:

dpkg -l | grep microcode
ii  amd64-microcode                   3.20160316.3                   amd64        Processor microcode firmware for AMD CPUs
ii  intel-microcode                   3.20180807a.1~deb9u1           amd64        Processor microcode firmware for Intel CPUs

Do you think removing it would help?

Also I have Broadcom network adapters - if it is important:

Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet

Regards,
Alex

15 апреля 2019, 11:45:37, от "Matus UHLAR - fantomas" <[hidden email]>:

On 15.04.19 11:05, Алексей wrote:
>I have two Debian boxes acting as a routesr (+iptables +quagga(BGP)). I have updated one from Debian 7 to Debian 9 recently and now experience 2x CPU utilization on the same amounts of traffic, e.g. on normal load of 1Gbps (real-world traffic) Debian 7 had 5-6% and Debian 9 has 10-12% softirq CPU time. This behaviour can also be observed during tests - 6Gbps iperf gets 3% on Debian 7 and 6% on Debian 9.
>
>I compared ethtool -k outputs on both boxes and they are pretty similar. Syslog doesn't show anything suspicious.
>
>Did anyone have the similar issue? Or can give me some advice on troubleshooting?

we have noticed something similar due to the spectre/meltdown updates.
Turning off HT cpus did help much.
not installing the CPU microcode should help too.

-- 
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I intend to live forever - so far so good. 

Reply | Threaded
Open this post in threaded view
|

Re: High CPU usage after upgrading from Debian 7 to Debian 9

Matus UHLAR - fantomas
On 16.04.19 17:22, Алексей wrote:
>Thanks for your answer.
>
>I have Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz inside - no HT on it.
>
>And I already have CPU microcode installed:
>
>dpkg -l | grep microcode
>ii  amd64-microcode                   3.20160316.3                   amd64        Processor microcode firmware for AMD CPUs
>ii  intel-microcode                   3.20180807a.1~deb9u1           amd64        Processor microcode firmware for Intel CPUs

>Do you think removing it would help?

you surely don't need amd64-microcode, you can try uninstalling
intel-microcode to see if it helps.

I think there are some option to disable spectre/meltdown mitigation in the
kernel, just haven't found them yet... you can try to search for them

>Also I have Broadcom network adapters - if it is important:

should not be.

>15 апреля 2019, 11:45:37, от "Matus UHLAR - fantomas" <[hidden email]>:
>
>On 15.04.19 11:05, Алексей wrote:
>>I have two Debian boxes acting as a routesr (+iptables +quagga(BGP)). I have updated one from Debian 7 to Debian 9 recently and now experience 2x CPU utilization on the same amounts of traffic, e.g. on normal load of 1Gbps (real-world traffic) Debian 7 had 5-6% and Debian 9 has 10-12% softirq CPU time. This behaviour can also be observed during tests - 6Gbps iperf gets 3% on Debian 7 and 6% on Debian 9.
>>
>>I compared ethtool -k outputs on both boxes and they are pretty similar. Syslog doesn't show anything suspicious.
>>
>>Did anyone have the similar issue? Or can give me some advice on troubleshooting?
>
>we have noticed something similar due to the spectre/meltdown updates.
>Turning off HT cpus did help much.
>not installing the CPU microcode should help too.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
There's a long-standing bug relating to the x86 architecture that
allows you to install Windows.   -- Matthew D. Fuller