[emanicslab] systemd-journal: CPU 100%

[Guilherme Sperb Machado]

Hello EmanicsLab, David, Thierry,

I've set up a testbed environment for MyPLC, the same one that
EmanicsLab is running.

Lately, I've seen that my nodes got 100% CPU after a while... and I
observed the following:

- the node reached permanent 100% of CPU *only* when there's an LXC
slice running on the node
- the node did not present ANY abnormal log messages in the journal log
- the node never got 100% of CPU when running without any LXC slice
being instantiated

Therefore, I started to investigate a bit further and found out the following:

- actually it's the SLICE internal systemd that causes the physical
CPU to run on 100%

Then, investigating a bit more, I found many log entries (with the
"journalctl") within slices that were complaining about this:

Jun 11 12:53:57 myplc-node1-vm.mgmt.local systemd[1]: Starting Cleanup
of Temporary Directories...
Jun 11 12:53:58 myplc-node1-vm.mgmt.local systemd-journal[11]:
Suppressed 13476 messages from /system
Jun 11 12:53:58 myplc-node1-vm.mgmt.local systemd[1]:
systemd-tmpfiles-clean.service start request repeated too quickly,
refusing to start.
Jun 11 12:53:58 myplc-node1-vm.mgmt.local systemd-journal[11]:
Suppressed 12326 messages from /system
Jun 11 12:53:58 myplc-node1-vm.mgmt.local systemd[1]: Failed to start
Cleanup of Temporary Directories.

Soooo, I did the following (within the slice):

el_ipv6test1 at myplc-node1-vm.mgmt.local# systemctl stop
systemd-tmpfiles-clean.timer

Then, the CPU utilization of the MyPLC node got back to normal.

David, Thierry, can you try such thing and let me know if the timer of
tmpfiles clean process MIGHT BE the cause of the systemd-journal CPU
on 100% ? Maybe, another process INSIDE one or more slices can be the
cause of the CPU on 100%. :-)

If so, I'm more than happy to help on the matter.

Cheers,
Guilherme

-- 
----------------------------------------------------
MSc. Guilherme Sperb Machado
Researcher & PhD. candidate at UZH (Universität Zürich)
http://www.csg.uzh.ch/staff/machado/
----------------------------------------------------