tryllz
January 2, 2024, 8:32am
1
Hi,
[root@nifi ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 3.3G 29G 11% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/centos-root 246G 246G 20K 100% /
/dev/sda1 488M 163M 290M 37% /boot
overlay 246G 246G 20K 100% /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa/merged
overlay 246G 246G 20K 100% /var/lib/docker/overlay2/674a640ef4b1cbef17f05b60ef143a37c44628046929e5ced1b7e61b628bffec/merged
overlay 246G 246G 20K 100% /var/lib/docker/overlay2/1862e7b7cac6cc5af5cfc67a766ee27d092e3582aab7d21c2cc99538de53715a/merged
tmpfs 6.3G 0 6.3G 0% /run/user/0
but I can’t seem to find what’s occupying this space. I have removed NiFi logs but this does not change anything.
[root@nifi ~]# du -sh /*
0 /bin
163M /boot
0 /dev
36M /etc
0 /home
0 /lib
0 /lib64
0 /media
0 /mnt
0 /opt
du: cannot access ‘/proc/1323/task/1422/fd/20’: No such file or directory
du: cannot access ‘/proc/2687/task/3165/fd/3160’: No such file or directory
du: cannot access ‘/proc/2687/task/3179/fd/3159’: No such file or directory
du: cannot access ‘/proc/2687/task/3644/fd/3184’: No such file or directory
du: cannot access ‘/proc/2687/task/3644/fdinfo/3160’: No such file or directory
du: cannot access ‘/proc/2687/task/3651/fd/3160’: No such file or directory
du: cannot access ‘/proc/2687/task/3651/fd/3175’: No such file or directory
du: cannot access ‘/proc/2687/task/3676/fdinfo/3172’: No such file or directory
du: cannot access ‘/proc/2687/task/24982/fdinfo/3175’: No such file or directory
du: cannot access ‘/proc/2687/task/18976/fdinfo/3172’: No such file or directory
du: cannot access ‘/proc/2687/task/24953/fd/3172’: No such file or directory
du: cannot access ‘/proc/2687/task/7821’: No such file or directory
du: cannot access ‘/proc/2687/task/7822’: No such file or directory
du: cannot access ‘/proc/7815/task/7815/fd/4’: No such file or directory
du: cannot access ‘/proc/7815/task/7815/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/7815/fd/4’: No such file or directory
du: cannot access ‘/proc/7815/fdinfo/4’: No such file or directory
0 /proc
56K /root
3.3G /run
0 /sbin
23G /srv
0 /sys
0 /tmp
1.6G /usr
14G /var
When I perform a reboot its shows disk space is back to normal.
[root@nifi ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 9.0M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/centos-root 246G 35G 212G 15% /
/dev/sda1 488M 163M 290M 37% /boot
overlay 246G 35G 212G 15% /var/lib/docker/overlay2/5fc6aa8604cd96df27c0084dfc1be949d14158480b01397ebcb6a369936e72a3/merged
overlay 246G 35G 212G 15% /var/lib/docker/overlay2/104ffd0f4c5609353670a49a0c094c3e49a2fbd1699eb5fb36b24a202f0898da/merged
tmpfs 6.3G 0 6.3G 0% /run/user/0
Any thoughts on what I need to do ?
11 Spice ups
Unusual for Linux, but that is a classic memory leak, my friend. Just gotta figure out which program is doing it.
furicle
January 2, 2024, 2:00pm
3
Look at your Docker storage etc… that overlay probably has downloaded images, running processes etc in it…
2 Spice ups
pigdog
January 2, 2024, 2:33pm
4
huudrych:
Hi,
but I can’t seem to find what’s occupying this space. I have removed NiFi logs but this does not change anything.
[root@nifi ~]# du -sh /*
....
Any thoughts on what I need to do ?
The wildcard “*” in your “du” command is probably not helping, as it won’t expand a name that begins with “.” (dot).
See if you can learn more with this, running it from the root directory:
du -x / | sort -nr |head
The " -x " option to du will limit the report to the root directory.
That should create a taper list starting with the largest usage stats for " / " only. Your problem should appear near the top.
1 Spice up
tryllz
January 2, 2024, 2:57pm
5
@dcmartin
Thanks, that did help, but it does not show any user files that are occupying this disk space, plus the overlay directory is only 13GB in size
[root@nifi ~]# du -xh / | sort -nr |head
1020K /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa/diff/opt/nifi/nifi-current/work/nar/extensions/nifi-metrics-reporting-nar-1.23.2.nar-unpacked/NAR-INF/bundled-dependencies
1020K /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa/diff/opt/nifi/nifi-current/work/nar/extensions/nifi-metrics-reporting-nar-1.23.2.nar-unpacked/NAR-INF
1016K /var/lib/docker/overlay2/aa23615aa873db4f3624c1be947e3a0389633fa5cba4705c6d363c823cfaa65c/diff/opt/nifi/nifi-current/work/nar/extensions/nifi-smb-nar-1.24.0.nar-unpacked/NAR-INF/bundled-dependencies
1016K /var/lib/docker/overlay2/aa23615aa873db4f3624c1be947e3a0389633fa5cba4705c6d363c823cfaa65c/diff/opt/nifi/nifi-current/work/nar/extensions/nifi-smb-nar-1.24.0.nar-unpacked/NAR-INF
1008K /var/lib/docker/overlay2/7c12e48c5e95e2291598fe0bee9749068ff54d026eea302312c28724c15749f0/diff/usr/share/locale/de/LC_MESSAGES
1008K /var/lib/docker/overlay2/7c12e48c5e95e2291598fe0bee9749068ff54d026eea302312c28724c15749f0/diff/usr/share/locale/de
1004K /usr/lib/firmware/mediatek/mt8183
1004K /usr/libexec/openssh
997M /srv/nifi/conf/archive
996K /var/lib/docker/overlay2/fa714aeb98ae45570b8d49ed1b23aa52fb968752000baf233a4a928aa37409c5/diff/var/lib
Current disk space since last reboot
[root@nifi ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 9.1M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/centos-root 246G 168G 79G 69% /
/dev/sda1 488M 163M 290M 37% /boot
overlay 246G 168G 79G 69% /var/lib/docker/overlay2/5fc6aa8604cd96df27c0084dfc1be949d14158480b01397ebcb6a369936e72a3/merged
overlay 246G 168G 79G 69% /var/lib/docker/overlay2/104ffd0f4c5609353670a49a0c094c3e49a2fbd1699eb5fb36b24a202f0898da/merged
overlay 246G 168G 79G 69% /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa/merged
tmpfs 6.3G 0 6.3G 0% /run/user/0
Depending on what you’re running through NiFi, the content, flowfile, and provenance repos can fill up fast. If you have auditing turned on, you can quickly fill up a disk when using NiFi. One one of our NiFi serves, we were generating 1TB of audit events a week before we added exclusions.
You could run a one-line script to watch what directories start filling up:
while true; do du -ah / 2>/dev/null | sort -rh | head ; echo ; sleep 60; done
1 Spice up
tryllz
January 2, 2024, 3:39pm
7
@ericross1623
Thanks, tried that, it does not show anything new thats consuming the disk space…
An output over the last 10 minutes…
[root@nifi ~]# while true; do du -ah / 2>/dev/null | sort -rh | head ; echo ; sleep 3; done
40G /
25G /srv
24G /srv/nifi
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
24G /srv/nifi
24G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
10G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
10G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
24G /srv/nifi
24G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
40G /
25G /srv/nifi
25G /srv
14G /var/lib/docker
14G /var/lib
14G /var
13G /var/lib/docker/overlay2
12G /srv/nifi/content_repository
11G /srv/nifi/provenance_repository
4.9G /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa
[root@nifi ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 9.1M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/centos-root 246G 182G 65G 74% /
/dev/sda1 488M 163M 290M 37% /boot
overlay 246G 182G 65G 74% /var/lib/docker/overlay2/5fc6aa8604cd96df27c0084dfc1be949d14158480b01397ebcb6a369936e72a3/merged
overlay 246G 182G 65G 74% /var/lib/docker/overlay2/104ffd0f4c5609353670a49a0c094c3e49a2fbd1699eb5fb36b24a202f0898da/merged
overlay 246G 182G 65G 74% /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa/merged
tmpfs 6.3G 0 6.3G 0% /run/user/0
[root@nifi ~]#
I’m not really familiar on the overlay filesystem.
You could try the same script above, but use “docker system df” instead to see what that shows:
while true; do docker system df ; echo ; sleep 60; done
1 Spice up
pigdog
January 2, 2024, 5:22pm
9
Like Eric, I am also unfamiliar withe the overlay filesystem .
Your earlier “df” showed 3 different filesystems whose pathnames starts with “/var/lib/docker/overlay2”:
/var/lib/docker/overlay2/5fc6aa8604cd96df27c0084dfc1be949d14158480b01397ebcb6a369936e72a3/merged overlay
/var/lib/docker/overlay2/104ffd0f4c5609353670a49a0c094c3e49a2fbd1699eb5fb36b24a202f0898da/merged overlay
/var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa/merged
And all 3 of the above filesystems were 100% full (your initial post).
Your root filesystem was 100% full in your initial post, and all 4 of these filesystems are identically sized at 246G.
I’m guessing that “overlay” means these are components of an overlay filesystem , and I know nothing about how these work.
But I’m wondering if they should NOT be subdirectories of the root filesystem. Your “du -xh / | sort -nr | head” reported on the root filesystem only, and there are 7 directories with “overlay2” in their name. I would think these should be separate mountpoint, ie, not part of the root fs, but I’m really not sure.
Try “df -T” to show filesystem types, and maybe “df -i” to show inodes.
1 Spice up
tryllz
January 3, 2024, 8:07am
10
[root@nifi ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 32G 0 32G 0% /dev
tmpfs tmpfs 32G 0 32G 0% /dev/shm
tmpfs tmpfs 32G 9.1M 32G 1% /run
tmpfs tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 246G 50G 197G 20% /
/dev/sda1 ext4 488M 163M 290M 37% /boot
overlay overlay 246G 50G 197G 20% /var/lib/docker/overlay2/b980a394c1a7b42c7c7cd34dc45437bbe02ec9d5f7c0f205d7927c0595cae1bb/merged
overlay overlay 246G 50G 197G 20% /var/lib/docker/overlay2/b3b456a3ed41af48e813b9489b2b0de466d1b9d3bc091ebaafb8df86093f504f/merged
overlay overlay 246G 50G 197G 20% /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa/merged
tmpfs tmpfs 6.3G 0 6.3G 0% /run/user/0
[root@nifi ~]# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 7.9M 361 7.9M 1% /dev
tmpfs 7.9M 1 7.9M 1% /dev/shm
tmpfs 7.9M 610 7.9M 1% /run
tmpfs 7.9M 16 7.9M 1% /sys/fs/cgroup
/dev/mapper/centos-root 123M 95K 123M 1% /
/dev/sda1 32K 342 32K 2% /boot
overlay 123M 95K 123M 1% /var/lib/docker/overlay2/b980a394c1a7b42c7c7cd34dc45437bbe02ec9d5f7c0f205d7927c0595cae1bb/merged
overlay 123M 95K 123M 1% /var/lib/docker/overlay2/b3b456a3ed41af48e813b9489b2b0de466d1b9d3bc091ebaafb8df86093f504f/merged
overlay 123M 95K 123M 1% /var/lib/docker/overlay2/33ff9979f5bb4ba27c513d9cf7df52cde3db9fb73b128271e389486a0beb98fa/merged
tmpfs 7.9M 1 7.9M 1% /run/user/0
[root@nifi ~]#
tryllz
January 3, 2024, 8:33am
11
No change in this either over the course of 20 minutes…
[root@nifi ~]# while true; do docker system df ; echo ; sleep 60; done
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 7 4 6.526GB 2.471GB (37%)
Containers 4 3 2.791GB 1.201GB (43%)
Local Volumes 8 8 3.068MB 0B (0%)
Build Cache 0 0 0B 0B
pigdog
January 3, 2024, 8:23pm
12
“df -i” was a longshot. Your results confirmed that you have plenty of inodes, so that’s not the problem.
“df -Th” confirmed that the filesystem on the rootfs is type xfs, and that the 3 filesystems with the lengthy numbers in their mountpoints are type “overlay.”
I don’t know how overlay filesystems work. It looks like your “overlays” depend on the xfs filesystem mounted at / (root)? I noticed in an earlier post that the overlay filesystems and the root filesystem had filled up to 100%.
Have a look at this post to see if it helps:
https://serverfault.com/questions/971935/why-does-each-overlay-mount-in-docker-say-that-it-is-the-full-size-of-the-root-f
tryllz
January 4, 2024, 9:24am
13
Thanks everyone…
I found the issue to be the log file that I deleted, which would was open as a process after reboot.
I let NiFi recreate the log file which I deleted and am now running a Cron job to truncate the file to 0 every 30 minutes.
I found that configuring the log size and rotation will only work for new containers, and not for existing one (unless someone knows how to do this for existing containers).
The following was added to /etc/crontab file
*/30 * * * * root /bin/bash /var/lib/docker/truncateloghourly.sh
and the following in the script file
truncate -s 0 /var/lib/docker/containers/e84c78a16195e0d72765414d8306702f006cae20c74b26bfbaf7047e9eebe9af/e84c78a16195e0d72765414d8306702f006cae20c74b26bfbaf7047e9eebe9af-json.log
1 Spice up
Great you figured it out. We have NiFi running in VM’s and I’ve had to have a script delete older logs because it can get out of control quickly.
furicle
January 7, 2024, 3:35pm
15
This thread may be interesting to others following along
General Discussions
General
Hello, I just want to share a tad of experience I gathered today with this issue, which might hopefully help some people here to understand what might be going on. We used a docker server for continuously deploying new software commits to a...
1 Spice up
Advertisement