Commit Graph

21 Commits

  • fix(recovery): a GPU access fault no longer fails other address spaces
    An address space's check of the SMMU returned the fault status while the
    GPU's own faulting access was still latched, and the address space that
    happened to map, unmap or attach next was taken out of use: seven processes
    in one run, the render server among them. The status says nothing about
    that address space's tables, so the operation now goes on and the fault is
    counted. The hang check ends the fault, and also clears the SMMU's global
    fault status where it is set.
    
    A stalled job's command buffers are searched for the IB2 address the thread
    stopped in, to tell an address the job's commands named from one they did
    not.
  • fix(recovery): keep the GPU usable after one process's address space fails
    An address space taken out of use quarantined every buffer it mapped,
    including the context's shared buffers that every address space maps. No
    address space could be made afterwards, so a restarted render server never
    got a device. Shared buffers are now declared as such and stay in use; the
    failed address space's reference still keeps them from being released.
    
    A job of a process whose address space is out of use ends without retiring
    its references instead of stopping the submission engine for everyone.
    
    The address space failure records its status and source line, and every
    exit of the recovery routine has a trace code.
  • feat(vm): record the mapping a failed map met
    native_adreno_map_met keeps, for the last failed BoMap, the address and
    bytes asked, whether the handle was mapped already, and the mapping at
    or past that address: its address, bytes, handle, jobs and kind.
    
    Run on hardware: a refused map at the window's first address showed
    another mapping of the same size there.
  • feat(memory): treat memory the kernel adopted outside its range as kernel memory
    A kernel may manage DRAM below its own physical range (a second bank it
    adopts as VM memory). Cache maintenance, the write-combined attribute,
    the sampled readers and the owned-page bitmap now ask the pmap whether a
    page is managed instead of comparing against gPhysBase and gPhysSize, so
    a buffer object with such pages is maintained like any other. The first
    failed kernel allocation is recorded (where, status, size, address bits,
    contiguous), and each CPU records the pool run it is zero-filling.
    
    Run on hardware: the GPU starts and runs with 458 MiB of adopted memory.
  • feat(memory): take write-combined system BOs from a pool of DRAM outside the kernel's
    The board registers free DRAM the kernel does not manage with native_adreno_pool_add. A write-combined system BO first asks the pool (NativeAdrenoPool.cpp: a bitmap a range, best single run, else the longest runs first, zero fill through the range's own write-combined kernel mapping before a grant is handed out); whatever the pool does not serve takes the descriptor path unchanged. A pool BO has a grant and no descriptor: its spans are the grant's runs, nothing of it is cleaned, audited or read through the cacheable aperture, its client view maps its physical runs write-combined and holds the grant until IOKit frees the view, and an import of pool pages is refused. Pages a client had a view of are retired unless gpu-pool=2; gpu-pool=0 turns the pool off. Counters: native_adreno_pool_state, native_adreno_pool_fallbacks, native_adreno_pool_zero.
    
    Host tests: test_pool (allocator against a page model) and test_pool_memory (BO paths); the existing 23 pass as before, test_client and test_gmuconfig fail as at the base. Every changed kernel source passes the kernel's compile recipe as a syntax check. Not built into an image and not run on hardware.
  • perf(maintenance): leave the driver's synchronized buffers out of the per-submission cache passes
    Every process table maps the GPU context's eleven kernel buffers (8548 KiB,
    two preemption records of 4192 KiB among them), and each submission cleaned
    and invalidated all of them: 11 passes and 8548 KiB a submission in macOS
    runs 57 to 61, about 351 of a submission's 579 us in run61.
    
    Their users keep the GPU's view current themselves (a non-cacheable view, or
    a clean after a store through the kernel's mapping), so the buffers are
    declared synchronized and a map, an unmap and a submission leave them alone.
    The probe's store through the kernel's mapping is cleaned where it is made.
    The first 8 submissions and every 1024th audit the head of each buffer
    against DRAM; a dirty line restores the maintenance for good.
    
    A submission, a completion and the write-combine re-assertion visit only the
    mappings they have work for instead of every mapping of the client. The
    submission log reads the IB words only with diag=1 and no longer through one
    kernel mapping per word.
    
    gpu-maint-all=1 restores the passes over everything for a comparison.
    Counters: native_adreno_maint_kind, native_adreno_maint_walk,
    native_adreno_sync_audit, native_adreno_sync_skips, native_adreno_submit_us.
    
    Host tests cover it (test_maintenance, test_gpuctx, test_submit). Not run on
    hardware.
  • perf(vm): index mappings, not every span
    The VM kept every mapped span (one per 4 KiB page of system memory) in one sorted list that each map inserted into and each growth copied under the device lock: BoMap held it up to 70 ms. Mappings are now ordered by GPU address per region; the table transaction gets the new spans for a map and the mapping's own spans for its unmap, and the import alias pass looks only at other-class imports when no page is the driver's own.
  • perf(vm): check import aliases once per map
    preflight already checks a non-exclusive import against every range of
    another class. The table transaction then sorted every other-class range
    again for the same answer: import maps that were not exclusive took
    3.56 ms each, 79% of all map time (i35abaf83). The transaction now takes
    preflight's answer through the exclusive batch.
  • perf(memory): user BOs from system memory, no fixed BO pool
    User BOs are pageable IOBufferMemoryDescriptors with no task and no
    kernel mapping, wired by prepare(), as Metal resources are system
    memory. The VM zero-fills them and sets the write-combined attribute in
    one batch while wiring, and restores it when the object is reaped.
    Client views map the descriptor itself, so no device-pager view resets
    the page attribute and submissions no longer re-assert it.
    
    The 1536 MiB boot-time pool is gone. In untagged-ed1611b1 it filled
    with the runtime's blocks, 404 BOs fell back to scattered kernel memory,
    96 allocations of 96 MiB failed and the pressure flag stayed set.
    Memory pressure is now reported when the system refuses a BO.
    Imports are proven exclusive through a page bitmap of the driver's own
    BOs instead of a pairwise span check.
  • feat(adreno): grow object, queue and job tables with demand
    No fixed object count: the handle index, the VM's mappings and ranges,
    a client's entries and its BO views grow with memory only. Queues grow
    as msm's submitqueues do; each client owns a 2^32 range of the shared
    queue space. Jobs are bounded by ring space: a full ring answers
    kIOReturnBusy, as msm waits for room. A BO is limited by the client's
    GPU address window, which Query reports as max_bo_bytes; the 256 MiB
    budget and the ABI's queue and size constants are gone.
  • feat(adreno): grow the BO pool in contiguous chunks
    The pool is a set of physically contiguous chunks instead of one fixed
    block. The board's reservation is the first chunk and stays; when no
    chunk has room a new one is taken from the system (2 MiB, the 4 KiB
    granule's block size, halved down to 64 KiB, or a large BO's own size),
    write-combined once as it is made, and given back when it empties.
    Memory pressure is reported only when the system cannot give a chunk
    (that BO falls back to scattered pages) and ends once the pool serves an
    allocation again; the seven-eighths rule and the 2 s window are gone.
  • feat(adreno): on-demand GPU page tables and a 2^ias address space
    Each GPU address space takes its table pages one at a time from the
    system (NativeAdrenoTablePages) instead of a 1027-page pool, so the
    aperture is the bank's whole input size: ias from ID2.UBS as Linux
    arm_smmu_id_size_to_bits reads it. Every process maps from
    ADRENO_VM_START (4 GiB) to 2^ias, as msm's adreno_private_vm_size.
    The GMU keeps its 0..2 GiB space. The alias scratch belongs to each
    VM and grows with its range table. A map that cannot get a table page
    is undone and returns kIOReturnNoMemory without poisoning the VM.
    Host tests cover a sparse 48-bit space and the refusal.
  • fix(adreno): let a VM's range table grow to 1048576 spans
    A scattered 64 MiB Metal heap is 16384 spans; 131072 filled up and maps
    failed with kIOReturnNoResources.
  • perf(adreno): count cache maintenance per phase and skip uncached imports
    Phase counters for the pool, allocate, map, submit, unmap, complete and table passes. Imports the owner maps uncached need no maintenance. Process page tables come from cacheable blocks with the walk coherence proved at board start, so a table page is ordered instead of cleaned. A freed BO whose first pages a user task still maps is counted. Submit records, handle index and user client changes of the iPadOS 27 bring-up are included.
  • perf(vm): map owned and exclusive imported memory without alias scans
    The VM keeps its ranges sorted with a binary-search lookup, holds up to
    16384 objects, and tracks the spans mapped per cache class.
    
    Memory the VM owns (pooled or scattered BOs) is mapped through
    ArmSmmu4KLiveMapContextBatchExclusive, which skips the physical-alias
    pass against every mapped range. An import takes the same path when
    all mapped imports share its cache class and none of its spans overlap
    the pool or an owned mapping; otherwise it keeps the full check.
    
    Per-kind timing counters (owned map, fast and full import, unmap by
    page and by ASID) are exported for the boot log.
    
    On sheng (i570) a map cost 0.20 ms, down from 0.66 ms (i568) and 1.42 ms
    before. The import fast path is not yet measured on hardware.
  • fix(vm): map GPU-written imports non-coherent and skip their per-submission cache work
    Imports the GPU writes (render targets) are mapped non-cacheable instead of
    write-back inner-shareable: the GPU stopped inside its end-of-pass cache clean
    when system memory passes wrote them coherently. The owner's CPU does not touch
    their pixels while the GPU owns them, so they were cleaned and invalidated at
    map time and need no maintenance on every submission (about 60 ms per frame).
    Read-only imports stay coherent.
  • fix(vm): read fewer SMMU registers per map, unmap and submission
    Owned routes are observed in full every 64th table operation only, and
    the TLBIALL before a submission skips the image observation once a
    table update has recorded the bank offset.
  • fix(memory): write-combine the BO pool once
    Pooled BOs no longer change page attributes one pmap call at a time on
    allocation and release; the pool is made write-combined when reserved
    and a pooled BO is zeroed through the pool mapping. Map and unmap
    observe only the global policy, the context and the owned routes; the
    full slot scan stays in the audit every 64 operations.
  • Import NativeAdreno
    Co-authored-by: rpone <rponeawa@qq.com>