Very high CPU usage in random moments, VPS goes down, can't find why

edited October 16 in Help

I have a small Hetzner VPS all with Docker containers running. My normal usage is fine. 1.7 GB Ram of 2.0 GB are in use (with buffers), uptime reports sane values all 3 around 0.60-0.90 (2 cores). Works pretty well.

In what I believe are random moments of the day, I receive a Hetrixtools notification that my stuff is down, and it is. Can't even login. In case it happens when I am logged in, I can't even give a command.

When it's over, uptime reports insane values, like 49.6 38.5 22.9 or something.

The issue: htop does not report anything out of the ordinary, docker stats the same.

All I see is typical processes which are using 2-10% of the CPU and small spikes of the usual suspects (a mysql process using 60% for a second or so).

Hetrixtools CPU monitor is even flatter. This is the last 24 hours according to it.

It does not make sense. I can't see what it is when it happens, and when I can see, nothing looks weird.
Anything else I could look at? It's even happening as I am writing this post now.

Edit: this looks ridicolous

To add to this, I am using Ubuntu 20.04, ufw enabled.

Tagged:

Comments

  • seriesnseriesn Hosting ProviderOG

    Your iowait seems pretty high. Make sure you have enough memory and not swapping.

    Thanked by (2)sgheghele vimalware
  • @seriesn thanks for trying! 0 swap in use

    I am 85% sure it's one of the containers. Will update this thread in case I find the issue.

  • Not_OlesNot_Oles Hosting ProviderContent Writer
    edited October 16

    @sgheghele said: 0 swap in use

    Seems like no swap available ( free = 0 in the screenshot ).

    Maybe having some swap might be helpful?

    https://chrisdown.name/2018/01/02/in-defence-of-swap.html

    https://news.ycombinator.com/item?id=23455051

    Good luck! Best wishes from Mexico!

    Old guy! Happy customer of OVH. Tom, not Oles! :-)
    Purveyor of fast-as-metal LXC VPSes

  • @Not_Oles said:

    @sgheghele said: 0 swap in use

    Seems like no swap available ( free = 0 in the screenshot ).

    Oh jeez, you (and @seriesn by proxy) are actually right, thanks! But this is Hetzner default Ubuntu 20.04 installation.
    Was swap removed from Ubuntu installation at some point, and I missed it? Is it Hetzner image that does not set one?
    I'll add one for sure!

    Thanked by (2)Not_Oles seriesn
  • WSCallumWSCallum Hosting Provider

    Looks like an issue running out of memory given how little is free, add swap as others suggested, it might well help.

    Thanked by (1)sgheghele

    Clouveo - SSD Cloud Servers & Hosting
    AMD EPYC Powered | DDoS Protected | Snapshots | Auto Backups

  • seriesnseriesn Hosting ProviderOG

    @sgheghele said:
    @Not_Oles said:

    @sgheghele said: 0 swap in use

    Seems like no swap available ( free = 0 in the screenshot ).

    Oh jeez, you (and @seriesn by proxy) are actually right, thanks! But this is Hetzner default Ubuntu 20.04 installation.
    Was swap removed from Ubuntu installation at some point, and I missed it? Is it Hetzner image that does not set one?
    I'll add one for sure!

    Most kvm templates these days doesn’t include swap by default. But yeah, usually OOM would be the culprit.

    I would also try to trace whatever is using up your memory and work my way back from there. Who knows, maybe you need to upgrade your plan?

    Thanked by (1)sgheghele
  • Not_OlesNot_Oles Hosting ProviderContent Writer

    @sgheghele said: Was swap removed from Ubuntu installation at some point, and I missed it? Is it Hetzner image that does not set one?

    Sorry, I don't know the current default install status of swap in Ubuntu generally or on VPSes at Hetzner. However, I recently installed Hetzner's default Ubuntu 20.04 image on a bare metal server there. top(1) on that machine shows:

    MiB Mem :  15924.4 total,  15673.9 free,    125.3 used,    125.2 buff/cache
    MiB Swap:   8183.0 total,   8183.0 free,      0.0 used.  15572.9 avail Mem
    
    Thanked by (1)sgheghele

    Old guy! Happy customer of OVH. Tom, not Oles! :-)
    Purveyor of fast-as-metal LXC VPSes

  • Not_OlesNot_Oles Hosting ProviderContent Writer

    @sgheghele said: (and @seriesn by proxy)

    Agreed!

    @seriesn is excellent for proxy!

    But @seriesn also is far better than excellent for VPN :)

    Thanked by (2)yoursunny seriesn

    Old guy! Happy customer of OVH. Tom, not Oles! :-)
    Purveyor of fast-as-metal LXC VPSes

  • edited October 17

    Ubuntu & docker - arrgh!
    KISS. Just saying. ;)

    lowendinfo.com is expiring soon.

  • @AlwaysSkint said:
    Ubuntu & docker - arrgh!
    KISS. Just saying. ;)

    I know, I know. But free livepatch!

  • edited October 17

    Alright, I finished my investigation. It was the missing SWAP / insufficient RAM. What sparkled it was however a container, as I suspected: CodiMD (a.k.a. HackMD), something is happening that makes its MySQL usage in RAM grow while idling. This caused the out of RAM issue and the missing swap was, of course, what caused the VPS to hang.

    Thanks everyone for your help!

    Second thing: I span up a fresh new Hetzner VPS with Ubuntu 20.04. I confirm that swap is not configured. I have never seen this before!

  • edited October 17

    @sgheghele said: Second thing: I span up a fresh new Hetzner VPS with Ubuntu 20.04. I confirm that swap is not configured. I have never seen this before!

    SSD/NVMe? Likely to stop folks thrashing them, when vm.swapinness is set at silly 60 default - set to 0 or 1.
    Anyway: Ubuntu for desktop; debian/CentOS for server. :p
    (I have seen other templates set with no swap but never use provider's templates when commissioning a server. Grab the network settings (where required) and use a minimal/network ISO.)
    Personally, i usually set 1GB swap for 2GB RAM -if it needs more than that then something's up: insufficient resources/poorly tuned or runaway processes (hello, clamd!).

    lowendinfo.com is expiring soon.

Sign In or Register to comment.