18 Commits

  • feat(memory): take write-combined system BOs from a pool of DRAM outside the kernel's
    The board registers free DRAM the kernel does not manage with native_adreno_pool_add. A write-combined system BO first asks the pool (NativeAdrenoPool.cpp: a bitmap a range, best single run, else the longest runs first, zero fill through the range's own write-combined kernel mapping before a grant is handed out); whatever the pool does not serve takes the descriptor path unchanged. A pool BO has a grant and no descriptor: its spans are the grant's runs, nothing of it is cleaned, audited or read through the cacheable aperture, its client view maps its physical runs write-combined and holds the grant until IOKit frees the view, and an import of pool pages is refused. Pages a client had a view of are retired unless gpu-pool=2; gpu-pool=0 turns the pool off. Counters: native_adreno_pool_state, native_adreno_pool_fallbacks, native_adreno_pool_zero.
    
    Host tests: test_pool (allocator against a page model) and test_pool_memory (BO paths); the existing 23 pass as before, test_client and test_gmuconfig fail as at the base. Every changed kernel source passes the kernel's compile recipe as a syntax check. Not built into an image and not run on hardware.
  • feat(diag): record who set which GPU priority
    The newest 32 setPriority calls keep the pid, both values, the task's GPU
    role, the time and how many times the connection had told the driver
    before, so a priority that the frontend forwards can be told apart from
    the runtime's own call at device open.
    
    Submissions carry the task's GPU role to the timeline.
  • feat(gpu): CP preemption between two rings (A1)
    Add hardware CP context switch via CP_CONTEXT_SWITCH_CNTL on a second
    ring. Use a conservative BO-disjoint rule: only allow a queued job to
    preempt the running ring when their buffer sets share no handle. Two
    4192 KiB preemption records and two a7xx_cp_smmu_info pages are mapped
    at fixed IOVAs. Ring priority comes from the kernel GPU role of the
    submitting task (BACKGROUND/UTILITY → ring 1, UI/focal/realtime → ring 0).
    Diagnostics gated on sheng-diag=1; feature gated on sheng-gpu-preempt=1.
  • perf(bo): free system BO pages after the device lock
    BoFree unmaps the BO under the lock and hands its still-wired descriptor out; the pages are unwired and freed once the lock is let go. i e38a0b1eb: a BoFree held the lock up to 38 ms.
  • perf(memory): make and wire BoAlloc's pages outside the lock
    The descriptor of a user BO's system memory is made, wired and, for
    default-attribute pages, written back before the driver's lock is taken;
    allocateSystem adopts it under the lock and pages it does not take are
    given back after the call. i14aef633e held the lock 4.0 s in 5454
    allocations while every other process's submit and wait poll queued.
  • feat(telemetry): each client's own BO bytes live and at peak
    The kernel's total of system BOs could not say which process held them.
    Each user client now counts its own BOs (imports not counted) for the
    board log's [gpu objects] row.
  • perf(memory): user BOs from system memory, no fixed BO pool
    User BOs are pageable IOBufferMemoryDescriptors with no task and no
    kernel mapping, wired by prepare(), as Metal resources are system
    memory. The VM zero-fills them and sets the write-combined attribute in
    one batch while wiring, and restores it when the object is reaped.
    Client views map the descriptor itself, so no device-pager view resets
    the page attribute and submissions no longer re-assert it.
    
    The 1536 MiB boot-time pool is gone. In untagged-ed1611b1 it filled
    with the runtime's blocks, 404 BOs fell back to scattered kernel memory,
    96 allocations of 96 MiB failed and the pressure flag stayed set.
    Memory pressure is now reported when the system refuses a BO.
    Imports are proven exclusive through a page bitmap of the driver's own
    BOs instead of a pairwise span check.
  • feat(adreno): grow object, queue and job tables with demand
    No fixed object count: the handle index, the VM's mappings and ranges,
    a client's entries and its BO views grow with memory only. Queues grow
    as msm's submitqueues do; each client owns a 2^32 range of the shared
    queue space. Jobs are bounded by ring space: a full ring answers
    kIOReturnBusy, as msm waits for room. A BO is limited by the client's
    GPU address window, which Query reports as max_bo_bytes; the 256 MiB
    budget and the ABI's queue and size constants are gone.
  • perf(adreno): count cache maintenance per phase and skip uncached imports
    Phase counters for the pool, allocate, map, submit, unmap, complete and table passes. Imports the owner maps uncached need no maintenance. Process page tables come from cacheable blocks with the walk coherence proved at board start, so a table page is ordered instead of cleaned. A freed BO whose first pages a user task still maps is counted. Submit records, handle index and user client changes of the iPadOS 27 bring-up are included.
  • perf(client): find objects through a handle index and hold 16384 of them
    A client's objects are found through a handle-to-entry index instead of
    a walk over every entry, free entries start from a hint, and the live
    count is kept rather than recounted. A BO's view token names its own
    slot, so CPU map and unmap no longer walk every view.
    
    The object cap goes from 4096 to 16384 per client, as many as the shared
    VM table holds: at 4096 backboardd ran out of objects and crashed twice
    (i567). On sheng (i568) no respring happened with a peak of 5725.
  • feat(submit): take a whole frame of up to 128 IBs in one submission
    ABI version 2 raises the IB list to 128 entries, so the ring prologue
    (cache flush, idle wait, thread switch) runs once per frame instead of
    once per 16 IBs. The submit lists live in the client and submit objects
    instead of the kernel stack. The stall handler drops the translations and
    gives the write pointer again after 10 waits; the IB counts per submission
    are recorded.
  • feat(userclient): record live imports and refused frees
    A render server thread waited inside vm_deallocate for minutes on
    int-233 and int-234. To tell whether an import of user pages is still
    held over that range, the client now keeps the count of live imports,
    the frees of an import it refused with the last status, and the eight
    newest imports (address, owner pid, bytes, handle, state).
  • fix(vm): raise the GPU object limits to 4096
    The shared VM mapping table held 1024 objects. int-237 filled it twice
    (BoMap refused with kIOReturnNoResources at objects=1024): Turnip turned
    the refusal into VK_ERROR_OUT_OF_DEVICE_MEMORY inside a command stream
    reservation it does not check, and backboardd crashed writing through a
    NULL stream pointer; the other run froze with the same refusals.
    
    Raise the VM table, the per-client entry table and the per-client view
    cache to 4096, and record per client slot the live objects, their peak,
    refused BoAllocs and the window's highest mapped end
    (native_adreno_client_stats).
    
    Claude-Session: https://claude.ai/code/session_01Ci5sg8z8jkQHR4QoXMBqfY
  • fix(userclient): serve metal clients beside the render server
    - Four client windows: the render server keeps 1 GiB, and the
      wallpaper, icon and widget processes share the rest of the aperture
      (170 MiB each).
    - A client that owns no queue range yet is no longer refused by
      setScope, held up in close or waited on in releaseAll because the
      render server has jobs in flight in the default range 0 (int-200).
      New regression block in the user-client host test.
    - BoCpuUnmap keeps the shared CPU view while another mapping lease of
      the same BO is held; dropping it unmapped the other holder's pages
      and crashed the render server in vkCmdCopyBuffer (int-177).
    - Diagnostics: the reason the latest mount failed, mappings that fall
      outside a client window, a text row per non-render-server process
      and a name for the mailbox poller thread.
  • feat(publish): pool only user BOs and serve one connection per process
    The SMMU tables and ring/fence/SQE buffers no longer come from the
    write-combined pool, so CPU table walks stay cached. The published
    service keeps one user-client core per connection: the render server
    gets the 1 GiB window at 0x20000000 and two other processes 256 MiB
    each, with their own queue ranges and teardown.
  • Import NativeAdreno
    Co-authored-by: rpone <rponeawa@qq.com>