One of the questions that a colleague of mine recently asked was: “How can I find out how much swap space a PostgreSQL process is really using?”. This triggered a piece of curiosity which I will talk about in this post.
At first, it might seem like we can use the Linux top command which has memory related columns. Let’s run it for an arbitrary process and see the output:
[root@localhost ~]# top -p 3748
top - 00:42:16 up 3 days, 7:43, 5 users, load average: 0.00, 0.01, 0.05
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16266588 total, 15392968 free, 233036 used, 640584 buff/cache
KiB Swap: 1679356 total, 1615172 free, 64184 used. 15566964 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND SWAP
3748 postgres 20 0 8851352 4520 3344 S 0.0 0.0 0:00.01 postmaster 972
The top command reports that the process (pid 3748, line 9, last column) is utilizing 972 KB of swap space. There are three questions here:
1) Is this really swap and the OS does not fake it?
2) If it is, then how accurate is this number?
3) Why does the OS allocate swap to this process while there is free RAM?
Here, I will try to answer the first two questions (third one is for another post).
According to the Linux top manual the reported swap column can include any of the quadrants from #1 through #3 each of which includes memory pages allocated via certain functions. For example, quadrant #1 includes process stack and this does not actually tell anything about being swapped. It only tells us that memory pages allocated for process stack are eligible for being swapped in/out.
So, how can we find out then how much of the process address space is mapped and really is on the swap file?
To answer this question, let’s get back to some fundamentals.
A Linux process is provided with a virtual address range. A page from an address from this range can be allocated either from RAM or secondary storage (HDD, SSD, etc.). The process page table maintains mapping from a virtual address to a physical address. Moreover, it also holds control bits each of which tracks some information about the page, e.g., dirty bit in the page table indicates whether the page’s content is in sync with the disk, unevictable bit indicates that the page cannot be evicted from the LRU list, and so on. This information can be queried from /proc/kpageflags file which in the latest kernel holds 26 flags.
Two of the flags are related to swapping out of which swapcache is the one we need. This bit indicates whether the page table entry (PTE) has an associated swap entry.
Let’s now see how we can calculate the total swap space occupied by a process.
The environment for experiments is PostgreSQL 14 running on CentOS 7.7. Let’s first connect to the database and retrieve it’s OS PID to have a process to work with:
-bash-4.2$ psql postgres
psql (14.2)
Type "help" for help.
postgres=# select pg_backend_pid();
pg_backend_pid
----------------
16460
(1 row)
The next step is to get swapcache bit value for every page in every PTE for this process. This information is available in /proc/$PID/pagemaps from where we can find out which physical frame a virtual page is mapped to (the process page table) and flag values (this file must be parsed to be human readable; I used the parser available here). For the sake of demonstration, I will show just a single range of virtual addresses which can be obtained from /proc/$PID/maps.
The file holds all virtual address ranges for a process including libraries associated with each range, permissions and etc:
[root@localhost ~]# tail /proc/16460/maps
7f5a72e70000-7f5a72e77000 rw-s 00000000 00:13 246403 /dev/shm/PostgreSQL.1005331588
7f5a72e77000-7f5a72e91000 rw-p 00000000 00:00 0
7f5a72e96000-7f5a72e97000 rw-s 00000000 00:04 327680 /SYSV01008d37 (deleted)
7f5a72e97000-7f5a72e98000 rw-p 00000000 00:00 0
7f5a72e98000-7f5a72e99000 r--p 00021000 fd:00 35678 /usr/lib64/ld-2.17.so;623f47b9 (deleted)
7f5a72e99000-7f5a72e9a000 rw-p 00022000 fd:00 35678 /usr/lib64/ld-2.17.so;623f47b9 (deleted)
7f5a72e9a000-7f5a72e9b000 rw-p 00000000 00:00 0
7fff241ac000-7fff241cd000 rw-p 00000000 00:00 0 [stack]
7fff241de000-7fff241e0000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]Let’s take the range associated with the stack space (line 9) 7fff241ac000-7fff241cd000 and pass it to pagemap executable:
[root@localhost pagemap-master]# ./pagemap 16460 0x7f585720f000 0x7f585721b000
0x7f585720f000 : pfn 3e2942 soft-dirty 1 file/shared 1 swapped 0 present 1
0x7f5857210000 : pfn 3e2943 soft-dirty 1 file/shared 1 swapped 0 present 1
0x7f5857211000 : pfn 41c95c soft-dirty 1 file/shared 1 swapped 0 present 1
0x7f5857212000 : pfn 0 soft-dirty 1 file/shared 0 swapped 0 present 0
0x7f5857213000 : pfn 0 soft-dirty 1 file/shared 0 swapped 0 present 0
0x7f5857214000 : pfn 41c95f soft-dirty 1 file/shared 1 swapped 0 present 1
0x7f5857215000 : pfn 3ff0e8 soft-dirty 1 file/shared 1 swapped 0 present 1
0x7f5857216000 : pfn 0 soft-dirty 1 file/shared 0 swapped 0 present 0
0x7f5857217000 : pfn 0 soft-dirty 1 file/shared 0 swapped 0 present 0
0x7f5857218000 : pfn 3ff0eb soft-dirty 1 file/shared 1 swapped 0 present 1
0x7f5857219000 : pfn 0 soft-dirty 1 file/shared 0 swapped 0 present 0
0x7f585721a000 : pfn 0 soft-dirty 1 file/shared 0 swapped 0 present 0As an example, consider line 1 of the output. The first value is the virtual address 0x7f585720f000 followed by the pfn: 3e2942 (page frame number – the actual physical address of the frame in hexadecimal), followed by attribute names and their values. For this address we see that swapped bit is 0. This is what we were looking for. Thus, for any page from the process page table, we can get the bit value which tells us whether the page is associated with a swap entry.
However, just the fact that the page is mapped to a swap entry does not yet mean that it has been been swapped out. We also have to make sure that the associated page in the swap space is been used. This can be obtained by looking at the locked bit in /proc/kpageflags which is set when a page has exclusively been locked by the process (e.g., when doing I/O). As seen from the last output, the pagemap parser does not report this bit but this can be added. Moreover, the parser has also another executable (pagemap2) which prints all pages of a process.
To sum up, the actual swap space used by a process can be calculated as the number of pages with swapped and locked bits both set times the page size. I will not pretend that I have written a script that calculates it for a running process but making one seems to be a good idea that can help troubleshooting in certain memory-related scenarios.
Note: Although the motivation originated from a PostgreSQL database related problem, the approach should be valid for an arbitrary user process.