Commit Graph

24 Commits

  • feat(memory): treat memory the kernel adopted outside its range as kernel memory
    A kernel may manage DRAM below its own physical range (a second bank it
    adopts as VM memory). Cache maintenance, the write-combined attribute,
    the sampled readers and the owned-page bitmap now ask the pmap whether a
    page is managed instead of comparing against gPhysBase and gPhysSize, so
    a buffer object with such pages is maintained like any other. The first
    failed kernel allocation is recorded (where, status, size, address bits,
    contiguous), and each CPU records the pool run it is zero-filling.
    
    Run on hardware: the GPU starts and runs with 458 MiB of adopted memory.
  • fix(memory): keep the pool's zero fill out of the driver's lock
    A BoAlloc prepares its pages before it takes the driver's lock, and the
    pool's allocation runs there with its zero fill, which maps the run in
    the kernel map. When the preparation gave nothing, allocateSystem asked
    the pool a second time under the lock, so a VM mapping, a zero fill and
    an unmap ran while every other client's submit, map and wait queued
    behind. The lock path now always offers its (possibly empty) prepared
    pages, and allocateSystem leaves the pool alone when it was offered any:
    the VM's pages take the request there, as they did before the pool.
    
    Not run on hardware.
  • feat(memory): take write-combined system BOs from a pool of DRAM outside the kernel's
    The board registers free DRAM the kernel does not manage with native_adreno_pool_add. A write-combined system BO first asks the pool (NativeAdrenoPool.cpp: a bitmap a range, best single run, else the longest runs first, zero fill through the range's own write-combined kernel mapping before a grant is handed out); whatever the pool does not serve takes the descriptor path unchanged. A pool BO has a grant and no descriptor: its spans are the grant's runs, nothing of it is cleaned, audited or read through the cacheable aperture, its client view maps its physical runs write-combined and holds the grant until IOKit frees the view, and an import of pool pages is refused. Pages a client had a view of are retired unless gpu-pool=2; gpu-pool=0 turns the pool off. Counters: native_adreno_pool_state, native_adreno_pool_fallbacks, native_adreno_pool_zero.
    
    Host tests: test_pool (allocator against a page model) and test_pool_memory (BO paths); the existing 23 pass as before, test_client and test_gmuconfig fail as at the base. Every changed kernel source passes the kernel's compile recipe as a syntax check. Not built into an image and not run on hardware.
  • perf(maintenance): leave the driver's synchronized buffers out of the per-submission cache passes
    Every process table maps the GPU context's eleven kernel buffers (8548 KiB,
    two preemption records of 4192 KiB among them), and each submission cleaned
    and invalidated all of them: 11 passes and 8548 KiB a submission in macOS
    runs 57 to 61, about 351 of a submission's 579 us in run61.
    
    Their users keep the GPU's view current themselves (a non-cacheable view, or
    a clean after a store through the kernel's mapping), so the buffers are
    declared synchronized and a map, an unmap and a submission leave them alone.
    The probe's store through the kernel's mapping is cleaned where it is made.
    The first 8 submissions and every 1024th audit the head of each buffer
    against DRAM; a dirty line restores the maintenance for good.
    
    A submission, a completion and the write-combine re-assertion visit only the
    mappings they have work for instead of every mapping of the client. The
    submission log reads the IB words only with diag=1 and no longer through one
    kernel mapping per word.
    
    gpu-maint-all=1 restores the passes over everything for a comparison.
    Counters: native_adreno_maint_kind, native_adreno_maint_walk,
    native_adreno_sync_audit, native_adreno_sync_skips, native_adreno_submit_us.
    
    Host tests cover it (test_maintenance, test_gpuctx, test_submit). Not run on
    hardware.
  • feat(diagnostics): count imported memory and BO views by process
    native_adreno_import_memory holds the imports alive, their KiB, the peak and
    the imports made; native_adreno_import_owners holds the importing processes
    (pid, live KiB, live imports, peak KiB). An import pins its pages from
    importTask until release, so these are the bytes the driver keeps wired
    besides its own BOs.
    
    native_adreno_view_owners holds the BO views each process maps. A view is an
    IOKit mapping, which the VM bills to the mapping task by its whole size
    (iokit_mapped, part of the footprint the memory limits judge).
    
    The holder table of native_adreno_bo_owners is shared by the three tables
    (NativeAdrenoNoteHolder). Host tests cover the holder rows and the import
    accounting. Not run on hardware.
  • feat(diagnostics): count the system BOs by holding process and by size
    native_adreno_bo_owners keeps the 24 processes that hold the most system BO memory (pid, live and peak KiB, live BOs) and native_adreno_bo_sizes the live BOs in six size classes, for a board log. Built only; not yet run on hardware.
  • refactor: drop the sheng- prefix from boot arguments
    Every boot argument loses its sheng- prefix (sheng-touch=4 is now
    touch=4). No XNU or Apple kext boot argument of iPadOS 27 or macOS 27.3
    uses any of the new names.
  • perf(bo): free system BO pages after the device lock
    BoFree unmaps the BO under the lock and hands its still-wired descriptor out; the pages are unwired and freed once the lock is let go. i e38a0b1eb: a BoFree held the lock up to 38 ms.
  • perf(wait): poll fences without the device lock
    A fence wait now reads the ring's fence word without the lock while the CP has not reached the fence, and takes the lock only once it has passed or the timeout is up. The call ring records how long each call held the lock, and a hook reports the phases of BoAllocs of 256 KiB and more.
  • fix(adreno): ask for contiguous pages only up to 128 KiB, back off after a failure
    A contiguous request the kernel cannot serve scans every free page first: in ibb10331f24 124 such searches (0.1 to 64 MiB) took 21.7 s in all, synchronously in the render server's and the widget renderer's allocations. Every success was 64 to 128 KiB. Larger BOs no longer ask, and after a failure no request of that size or larger asks for 5 s. i55eb173a6: 4 failures (0.84 s), none in the widget page, Settings and Safari launch windows; BoAlloc max there 137, 228, 48 ms (was 260, 297, 239).
  • perf(memory): make and wire BoAlloc's pages outside the lock
    The descriptor of a user BO's system memory is made, wired and, for
    default-attribute pages, written back before the driver's lock is taken;
    allocateSystem adopts it under the lock and pages it does not take are
    given back after the call. i14aef633e held the lock 4.0 s in 5454
    allocations while every other process's submit and wait poll queued.
  • perf(memory): user BOs from system memory, no fixed BO pool
    User BOs are pageable IOBufferMemoryDescriptors with no task and no
    kernel mapping, wired by prepare(), as Metal resources are system
    memory. The VM zero-fills them and sets the write-combined attribute in
    one batch while wiring, and restores it when the object is reaped.
    Client views map the descriptor itself, so no device-pager view resets
    the page attribute and submissions no longer re-assert it.
    
    The 1536 MiB boot-time pool is gone. In untagged-ed1611b1 it filled
    with the runtime's blocks, 404 BOs fell back to scattered kernel memory,
    96 allocations of 96 MiB failed and the pressure flag stayed set.
    Memory pressure is now reported when the system refuses a BO.
    Imports are proven exclusive through a page bitmap of the driver's own
    BOs instead of a pairwise span check.
  • perf(adreno): no contiguous-memory search under the publication lock
    Once the boot reservation filled, every BoAlloc that found no room asked
    XNU for a physically contiguous chunk, 2 MiB halved down to 64 KiB.
    XNU finds one by scanning physical memory, and BoAlloc runs under the
    publication lock, so every client's submissions and waits stood behind
    the scans: 60 chunks made and 13 searches refused, BoAlloc took 120 s
    of a 420 s run, render server commits took seconds. A BO the
    reservation cannot hold now takes VM pages as msm's shmem-backed GEM
    objects do (the scattered path), and the pool reports memory pressure
    so clients give back what they cache.
  • test(adreno): host doubles and deferred BO checks for the current driver
    The host fixture gains kIOReturnInternalError and the hostsim the
    maintenance, page-attribute and pmap doubles the memory code calls. The
    deferred BO checks expect the growing retire queue (no wait for the GPU)
    and BOs the cache may keep. NativeAdrenoMemory no longer includes
    kern/clock.h.
  • feat(adreno): grow object, queue and job tables with demand
    No fixed object count: the handle index, the VM's mappings and ranges,
    a client's entries and its BO views grow with memory only. Queues grow
    as msm's submitqueues do; each client owns a 2^32 range of the shared
    queue space. Jobs are bounded by ring space: a full ring answers
    kIOReturnBusy, as msm waits for room. A BO is limited by the client's
    GPU address window, which Query reports as max_bo_bytes; the 256 MiB
    budget and the ABI's queue and size constants are gone.
  • feat(adreno): grow the BO pool in contiguous chunks
    The pool is a set of physically contiguous chunks instead of one fixed
    block. The board's reservation is the first chunk and stays; when no
    chunk has room a new one is taken from the system (2 MiB, the 4 KiB
    granule's block size, halved down to 64 KiB, or a large BO's own size),
    write-combined once as it is made, and given back when it empties.
    Memory pressure is reported only when the system cannot give a chunk
    (that BO falls back to scattered pages) and ends once the pool serves an
    allocation again; the seven-eighths rule and the 2 s window are gone.
  • feat(adreno): report BO pool pressure in Query flags
    NativeAdrenoParamsPoolPressure tells clients to give cached BOs back.
  • perf(adreno): count cache maintenance per phase and skip uncached imports
    Phase counters for the pool, allocate, map, submit, unmap, complete and table passes. Imports the owner maps uncached need no maintenance. Process page tables come from cacheable blocks with the walk coherence proved at board start, so a table page is ordered instead of cleaned. A freed BO whose first pages a user task still maps is counted. Submit records, handle index and user client changes of the iPadOS 27 bring-up are included.
  • perf(vm): map owned and exclusive imported memory without alias scans
    The VM keeps its ranges sorted with a binary-search lookup, holds up to
    16384 objects, and tracks the spans mapped per cache class.
    
    Memory the VM owns (pooled or scattered BOs) is mapped through
    ArmSmmu4KLiveMapContextBatchExclusive, which skips the physical-alias
    pass against every mapped range. An import takes the same path when
    all mapped imports share its cache class and none of its spans overlap
    the pool or an owned mapping; otherwise it keeps the full check.
    
    Per-kind timing counters (owned map, fast and full import, unmap by
    page and by ASID) are exported for the boot log.
    
    On sheng (i570) a map cost 0.20 ms, down from 0.66 ms (i568) and 1.42 ms
    before. The import fast path is not yet measured on hardware.
  • fix(memory): hold imported pages through a memory entry
    Core Animation unmaps an image's Shmem once its no-copy buffer is
    released, which can come before the runtime frees the import. The
    render thread of backboardd then stayed in vm_deallocate while the
    import still had the pages wired by its UPL, and the desktop froze.
    
    An import of user memory now asks for kIOMemoryPersistent, so the
    descriptor keeps a memory entry and with it a reference on the VM
    object; the owner's unmap leaves the object to the import until it
    completes. A range no memory entry can be made of is imported as
    before. Two counters tell the cases apart.
  • feat(publish): pool only user BOs and serve one connection per process
    The SMMU tables and ring/fence/SQE buffers no longer come from the
    write-combined pool, so CPU table walks stay cached. The published
    service keeps one user-client core per connection: the render server
    gets the 1 GiB window at 0x20000000 and two other processes 256 MiB
    each, with their own queue ranges and teardown.
  • fix(memory): write-combine the BO pool once
    Pooled BOs no longer change page attributes one pmap call at a time on
    allocation and release; the pool is made write-combined when reserved
    and a pooled BO is zeroed through the pool mapping. Map and unmap
    observe only the global policy, the context and the owned routes; the
    full slot scan stays in the audit every 64 operations.
  • Import NativeAdreno
    Co-authored-by: rpone <rponeawa@qq.com>