Unexplained memory leak? Free RAM goes down over time

hopefulhopeful OG
edited May 21 in Help

Hi LowEndHelpDesk, :p

I have a VPS with 2GB of memory. I don't use it a lot, leaving it mostly idle except the IPv6 BGP session (over 6in4) I have with he.net and NetAssist. I don't load the routing table; I just route everything to he.net.

However, I noticed that the memory usage increases slowly over time. I need to hard-reset the VPS every few weeks to restore it. This behaviour started roughly since 2020 started but I haven't made any significant changes to the configuration

Thanks to this beautiful graph from hetrixtools hopefully it explains better:

Memory usage is quite high:

[[email protected] ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          2.1Gi       1.6Gi        95Mi       1.0Mi       366Mi       289Mi
Swap:         303Mi        20Mi       283Mi

You can see that there's actually no application actively consuming memory:

[[email protected] ~]$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.3 109744  8264 ?        Ss   May13   5:58 /sbin/init
root           2  0.0  0.0      0     0 ?        S    May13   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   May13   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   May13   0:00 [rcu_par_gp]
root           6  0.0  0.0      0     0 ?        I<   May13   0:00 [kworker/0:0H-kblockd]
root           8  0.0  0.0      0     0 ?        I<   May13   0:00 [mm_percpu_wq]
root           9  0.0  0.0      0     0 ?        S    May13   1:39 [ksoftirqd/0]
root          10  0.0  0.0      0     0 ?        S    May13   0:00 [rcuc/0]
root          11  0.0  0.0      0     0 ?        I    May13  11:11 [rcu_preempt]
root          12  0.0  0.0      0     0 ?        S    May13   0:00 [rcub/0]
root          13  0.0  0.0      0     0 ?        S    May13   0:04 [migration/0]
root          14  0.0  0.0      0     0 ?        S    May13   0:00 [idle_inject/0]
root          16  0.0  0.0      0     0 ?        S    May13   0:00 [cpuhp/0]
root          17  0.0  0.0      0     0 ?        S    May13   0:00 [kdevtmpfs]
root          18  0.0  0.0      0     0 ?        I<   May13   0:00 [netns]
root          19  0.0  0.0      0     0 ?        S    May13   0:00 [rcu_tasks_kthre]
root          20  0.0  0.0      0     0 ?        S    May13   0:03 [kauditd]
root          21  0.0  0.0      0     0 ?        S    May13   0:00 [khungtaskd]
root          22  0.0  0.0      0     0 ?        S    May13   0:00 [oom_reaper]
root          23  0.0  0.0      0     0 ?        I<   May13   0:00 [writeback]
root          24  0.0  0.0      0     0 ?        S    May13   0:01 [kcompactd0]
root          25  0.0  0.0      0     0 ?        SN   May13   0:00 [ksmd]
root          26  0.0  0.0      0     0 ?        SN   May13   0:00 [khugepaged]
root         114  0.0  0.0      0     0 ?        I<   May13   0:00 [kintegrityd]
root         115  0.0  0.0      0     0 ?        I<   May13   0:00 [kblockd]
root         116  0.0  0.0      0     0 ?        I<   May13   0:00 [blkcg_punt_bio]
root         117  0.0  0.0      0     0 ?        I<   May13   0:00 [ata_sff]
root         118  0.0  0.0      0     0 ?        I<   May13   0:00 [edac-poller]
root         119  0.0  0.0      0     0 ?        I<   May13   0:00 [devfreq_wq]
root         120  0.0  0.0      0     0 ?        S    May13   0:00 [watchdogd]
root         121  0.0  0.0      0     0 ?        S    May13   4:59 [kswapd0]
root         124  0.0  0.0      0     0 ?        I<   May13   0:00 [kthrotld]
root         125  0.0  0.0      0     0 ?        I<   May13   0:00 [acpi_thermal_pm]
root         126  0.0  0.0      0     0 ?        I<   May13   0:00 [nvme-wq]
root         127  0.0  0.0      0     0 ?        I<   May13   0:00 [nvme-reset-wq]
root         128  0.0  0.0      0     0 ?        I<   May13   0:00 [nvme-delete-wq]
root         129  0.0  0.0      0     0 ?        I<   May13   0:00 [ipv6_addrconf]
root         140  0.0  0.0      0     0 ?        I<   May13   0:00 [kstrp]
root         146  0.0  0.0      0     0 ?        I<   May13   0:00 [zswap-shrink]
root         147  0.0  0.0      0     0 ?        I<   May13   0:00 [kworker/u3:0]
root         158  0.0  0.0      0     0 ?        I<   May13   0:00 [charger_manager]
root         188  0.0  0.0      0     0 ?        S    May13   0:00 [scsi_eh_0]
root         189  0.0  0.0      0     0 ?        I<   May13   0:00 [scsi_tmf_0]
root         190  0.0  0.0      0     0 ?        S    May13   0:00 [scsi_eh_1]
root         191  0.0  0.0      0     0 ?        I<   May13   0:00 [scsi_tmf_1]
root         195  0.0  0.0      0     0 ?        I<   May13   0:15 [kworker/0:1H-kblockd]
root         206  0.0  0.0      0     0 ?        S    May13   0:10 [jbd2/vda2-8]
root         207  0.0  0.0      0     0 ?        I<   May13   0:00 [ext4-rsv-conver]
root         234  0.0  2.2 172832 47940 ?        Ss   May13   3:28 /usr/lib/systemd/systemd-journald
root         242  0.0  0.0  78076   688 ?        Ss   May13   0:00 /usr/bin/lvmetad -f
root         247  0.0  0.2  30940  4732 ?        Ss   May13   0:02 /usr/lib/systemd/systemd-udevd
systemd+     250  0.0  0.2  26260  5208 ?        Ss   May13   0:05 /usr/lib/systemd/systemd-networkd
systemd+     276  0.0  0.1  91784  4276 ?        Ssl  May13   0:03 /usr/lib/systemd/systemd-timesyncd
root         289  0.0  0.0   6588  1968 ?        Ss   May13   0:07 /usr/bin/crond -n
dbus         290  0.0  0.1   6780  2772 ?        Ss   May13   3:35 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         291  0.0  0.2  17504  5660 ?        Ss   May13   1:47 /usr/lib/systemd/systemd-logind
v2ray        294  0.1  0.6 1168500 13268 ?       Ssl  May13  14:52 /usr/bin/v2ray -config /etc/v2ray/0.json
bird         295  0.0  0.0   7156  1268 ?        Ss   May13   2:00 /usr/bin/bird -s /run/bird/bird.ctl
root         296  0.0  0.0   5424  1304 tty1     Ss+  May13   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
http         315  0.0  0.5 1255500 12164 ?       Ssl  May13   0:54 /usr/bin/caddy -log stdout -agree -conf /etc/caddy/caddy.conf -root=/usr/share/caddy
root     3555765  0.0  0.0      0     0 ?        I    21:27   0:00 [kworker/0:0-rcu_gp]
root     3566553  0.0  0.0      0     0 ?        I    21:33   0:00 [kworker/u2:1-ext4-rsv-conversion]
root     3567234  0.0  0.0      0     0 ?        I    21:33   0:00 [kworker/0:1-events]
root     3569037  0.0  0.3  10704  7448 ?        Ss   21:34   0:00 sshd: me [priv]
me    3569138  0.0  0.4  18972  9776 ?        Ss   21:34   0:00 /usr/lib/systemd/systemd --user
me    3569140  0.0  0.0 113348  2116 ?        S    21:34   0:00 (sd-pam)
me    3569145  0.0  0.1  10704  4156 ?        S    21:34   0:00 sshd: [email protected]/0
me    3569146  0.0  0.1   7488  4000 pts/0    Ss   21:34   0:00 -bash
root     3577610  0.0  0.0      0     0 ?        I    21:38   0:00 [kworker/u2:0-flush-254:0]
root     3584705  0.1  0.0      0     0 ?        I    21:42   0:00 [kworker/0:2-events]
root     3588276  0.0  0.0      0     0 ?        I    21:44   0:00 [kworker/u2:2-events_unbound]
root     3591768  0.0  0.1   9636  3600 ?        S    21:46   0:00 /usr/bin/CROND -n
hetrixt+ 3591769  0.0  0.1   7144  2860 ?        Ss   21:46   0:00 /bin/sh -c bash /etc/hetrixtools/hetrixtools_agent.sh >> /etc/hetrixtools/hetrixtools_cron.log 2>&1
hetrixt+ 3591770  0.3  0.1   7144  3392 ?        S    21:46   0:00 bash /etc/hetrixtools/hetrixtools_agent.sh
hetrixt+ 3593070  0.0  0.0   7144  1816 ?        S    21:46   0:00 bash /etc/hetrixtools/hetrixtools_agent.sh
hetrixt+ 3593071  0.0  0.0   7972  1184 ?        S    21:46   0:00 vmstat 3 2
hetrixt+ 3593072  0.0  0.0   5340   580 ?        S    21:46   0:00 tail -1
me    3593073  0.0  0.1   9500  3596 pts/0    R+   21:46   0:00 ps aux
[[email protected] ~]$ cat  /proc/meminfo
MemTotal:        2163084 kB
MemFree:          111300 kB
MemAvailable:     302496 kB
Buffers:            5500 kB
Cached:            70908 kB
SwapCached:         1008 kB
Active:            56160 kB
Inactive:          54484 kB
Active(anon):      10912 kB
Inactive(anon):    22148 kB
Active(file):      45248 kB
Inactive(file):    32336 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        311292 kB
SwapFree:         289952 kB
Dirty:                 8 kB
Writeback:             0 kB
AnonPages:         33544 kB
Mapped:            46160 kB
Shmem:              1624 kB
KReclaimable:     284608 kB
Slab:            1881652 kB
SReclaimable:     284608 kB
SUnreclaim:      1597044 kB
KernelStack:        1520 kB
PageTables:         1616 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1392832 kB
Committed_AS:     476668 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       10888 kB
VmallocChunk:          0 kB
Percpu:            37680 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     1665012 kB
DirectMap2M:      563200 kB

Any ideas?

Comments

  • Looks like it's all being eaten by Slab? Which iirc is kernel-related cache. Try slabtop and maybe that can give you some hints, or you can share the output here for someone much smarter than myself to help with. lol

    🍌

  • hopefulhopeful OG
    edited May 21

    Thanks for the pointers! I never realised slab is a thing. Here's the output from slabtop and I will continue Googling tomorrow:

    $slabtop
     Active / Total Objects (% used)    : 14075841 / 14088832 (99.9%)
     Active / Total Slabs (% used)      : 332194 / 332194 (100.0%)
     Active / Total Caches (% used)     : 94 / 129 (72.9%)
     Active / Total Size (% used)       : 1835557.66K / 1838777.06K (99.8%)
     Minimum / Average / Maximum Object : 0.01K / 0.13K / 8.00K
    
      OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
    4723200 4723200 100%    0.01K   9225      512     36900K kmalloc-8
    2095170 2094988  99%    0.19K  99770       21    399080K cred_jar
    1584128 1584128 100%    0.03K  12376      128     49504K kmalloc-32
    1498624 1498324  99%    0.06K  23416       64     93664K anon_vma_chain
    749248 749248 100%    0.09K  16288       46     65152K anon_vma
    577376 577376 100%    0.12K  18043       32     72172K kernfs_node_cache
    402400 402050  99%    0.25K  25150       16    100600K filp
    296960 296960 100%    0.12K   9280       32     37120K pid
    251538 251538 100%    0.09K   5989       42     23956K kmalloc-96
    214336 214336 100%    0.06K   3349       64     13396K kmem_cache_node
    214290 214290 100%    0.44K  23810        9     95240K kmem_cache
    214284 211399  98%    0.19K  10204       21     40816K dentry
    181760 181760 100%    0.02K    710      256      2840K kmalloc-16
    177308 177308 100%    0.81K   9332       19    149312K sock_inode_cache
    148304 148256  99%    0.25K   9269       16     37076K skbuff_head_cache
    138630 138630 100%    1.06K   9242       15    147872K mm_struct
    135808 135808 100%    0.06K   2122       64      8488K kmalloc-64
    131911 131331  99%    0.60K  10147       13     81176K inode_cache
     74552  74552 100%    1.00K   9319        8     74552K PING
     71076  71050  99%    1.00K   8885        8     71080K kmalloc-1k
     46968  46968 100%    4.00K   5871        8    187872K kmalloc-4k
     32384  32384 100%    0.50K   4048        8     16192K kmalloc-512
     22197  22197 100%    0.19K   1057       21      4228K kmalloc-192
     19312  19312 100%    0.25K   1207       16      4828K kmalloc-256
     15639   8516  54%    0.10K    401       39      1604K buffer_head
     15246  15105  99%    0.57K   1089       14      8712K radix_tree_node
      9856   9856 100%    0.12K    308       32      1232K kmalloc-128
      9768   9752  99%    2.00K   1221        8     19536K kmalloc-2k
      9435   9435 100%    0.08K    185       51       740K Acpi-State
      4080   4080 100%    0.05K     48       85       192K ftrace_event_field
      2651   2174  82%    1.07K     95       29      3040K ext4_inode_cache
      2142   1836  85%    0.04K     21      102        84K ext4_extent_status
      1725   1556  90%    0.67K     75       23      1200K proc_inode_cache
      1568   1568 100%    0.07K     28       56       112K eventpoll_pwq
      1426   1426 100%    0.09K     31       46       124K trace_event_file
      1280   1280 100%    0.06K     20       64        80K vmap_area
      1120   1120 100%    0.07K     20       56        80K Acpi-Operand
      1116   1116 100%    0.11K     31       36       124K khugepaged_mm_slot
      1034   1016  98%    0.71K     94       11       752K shmem_inode_cache
       969    969 100%    0.08K     19       51        76K task_delay_info
       816    816 100%    0.04K      8      102        32K pde_opener
       714    546  76%    0.09K     17       42        68K kmalloc-rcl-96
       660    660 100%    0.31K     55       12       220K mnt_cache
       640    588  91%    0.06K     10       64        40K kmalloc-rcl-64
       510    510 100%    0.12K     15       34        60K jbd2_journal_head
       504    504 100%    0.19K     24       21        96K skbuff_ext_cache
    
    [[email protected] ~]$ sudo smem -tw
    Area                           Used      Cache   Noncache 
    firmware/hardware                 0          0          0 
    kernel image                      0          0          0 
    kernel dynamic memory       2018832     357292    1661540 
    userspace memory              59244      33928      25316 
    free memory                   85008      85008          0 
    ----------------------------------------------------------
                                2163084     476228    1686856 
    
    [[email protected] ~]$ df -i
    Filesystem     Inodes  IUsed  IFree IUse% Mounted on
    dev            268304    343 267961    1% /dev
    run            270385    438 269947    1% /run
    /dev/vda2      636480 462678 173802   73% /
    tmpfs          270385      1 270384    1% /dev/shm
    tmpfs          270385     18 270367    1% /sys/fs/cgroup
    tmpfs          270385     17 270368    1% /tmp
    tmpfs          270385     22 270363    1% /run/user/1000
    
  • mikhomikho Hosting ProviderOG

    What does cat /proc/slabinfo give?

    Get 4 or more NAT servers (mix/match between packages) and get a 20 % recurring discount. https://clients.mrvm.net

  • V2ray and caddy in-memory cache?

Sign In or Register to comment.