Problem / Symptoms

The macOS WindowServer process leaks memory, about +16KB (one page) every few seconds.

After a while, this leads to the WindowServer process consuming GBs of RAM.

Usually one will notice small lags when it starts swapping, only to find a grown-fat WindowServer process in Activity Manager.

Even when logging out, the memory is not freed again!

Workarounds

killall -HUP WindowServer

That causes the current user being logged out.

After logging in again, the WindowServer process is significantly smaller.

A system reboot also helps, obviously.

Gathering more information

% sudo footprint $(pgrep WindowServer) 
======================================================================
WindowServer [382]: 64-bit    Footprint: 471 MB (16384 bytes per page)
======================================================================

  Dirty      Clean  Reclaimable    Regions    Category
    ---        ---          ---        ---    ---
 238 MB        0 B        82 MB        394    IOAccelerator (graphics)
  45 MB        0 B          0 B         24    MALLOC_SMALL
->39 MB      80 KB          0 B       1393    untagged (VM_ALLOCATE) <- !!!
  36 MB        0 B       274 MB        421    IOSurface
  30 MB        0 B      6144 KB        144    Owned physical footprint (unmapped)
  21 MB        0 B       528 KB         34    MALLOC_TINY
  17 MB        0 B      1440 KB          1    MALLOC_NANO
  16 MB        0 B          0 B         11    IOAccelerator
  16 MB        0 B          0 B         10    MALLOC_MEDIUM
5424 KB      48 KB          0 B        651    CoreAnimation
1763 KB        0 B          0 B        464    __DATA
1044 KB        0 B          0 B          1    page table
 912 KB        0 B          0 B         37    IOKit
 784 KB        0 B          0 B         42    ColorSync
 712 KB        0 B          0 B       1045    unused dyld shared cache area
 672 KB        0 B        64 KB         44    stack
 558 KB        0 B          0 B        136    __DATA_DIRTY
 512 KB        0 B          0 B         36    MALLOC metadata
 496 KB      16 KB          0 B        494    __DATA_CONST
 423 KB        0 B          0 B        298    __AUTH
 144 KB        0 B          0 B        468    __AUTH_CONST
  80 KB        0 B          0 B          2    __TPRO_CONST
  48 KB      15 MB          0 B         22    mapped file
  48 KB        0 B          0 B          1    Activity Tracing
  32 KB        0 B          0 B          6    CG backing stores
  32 KB        0 B          0 B          2    CoreGraphics
  16 KB        0 B          0 B          1    os_alloc_once
    0 B      12 MB          0 B        515    __TEXT
    0 B     496 KB          0 B         28    __LINKEDIT
    0 B        0 B          0 B          4    ImageIO
    0 B        0 B          0 B          1    __FONT_DATA
    0 B        0 B          0 B          1    dyld private memory
    0 B        0 B          0 B          2    __SLSERVER
    0 B        0 B          0 B          1    __CTF
    0 B        0 B          0 B        142    Mach message
    ---        ---          ---        ---    ---
 471 MB      28 MB       364 MB       6878    TOTAL

Auxiliary data:
    phys_footprint: 471 MB
    phys_footprint_peak: 920 MB

“untagged (VM_ALLOCATE)” is going up 16KB every few seconds (best seen after a reboot).

The 39MB seen here are a still rather low value, not long after a reboot.

sudo vmmap $(pgrep WindowServer)
REGION TYPE                    START - END         [ VSIZE  RSDNT  DIRTY   SWAP] PRT/MAX SHRMOD PURGE    REGION DETAIL
...
VM_ALLOCATE                 3051d0000-3051d4000    [   16K    16K    16K     0K] rw-/rwx SM=COW  
...

This shows a lot of 16K allocations, their address range and also the overall RAM usage of all VM_ALLOCATE allocations.

Then I used a Python script and lldb (after temporarily disabling SIP) to dump all the 16K VSIZE RSDNT DIRTY memory regions to disk files.

Then another Python script to classify the disk files by their content hash to identify duplicate content.

What I found

There were a lot of identical-content 16KB memory pages, all containing a single 0x0A byte followed by 16383 0x00 bytes.