Commit Graph

33 Commits

  • fix(recovery): keep the GPU usable after one process's address space fails
    An address space taken out of use quarantined every buffer it mapped,
    including the context's shared buffers that every address space maps. No
    address space could be made afterwards, so a restarted render server never
    got a device. Shared buffers are now declared as such and stay in use; the
    failed address space's reference still keeps them from being released.
    
    A job of a process whose address space is out of use ends without retiring
    its references instead of stopping the submission engine for everyone.
    
    The address space failure records its status and source line, and every
    exit of the recovery routine has a trace code.
  • fix(submit): blame no job for a ring switch the CP never finished
    A first-ring job that waits while the CP stays on the second ring after
    a switch back timed out is stranded, as a second-ring job already was:
    the restart replays every job and no client loses its device. The
    switch timeout record also carries CP_CONTEXT_SWITCH_CNTL and the two
    rings of the switch.
  • fix(submit): count the BV thread's progress as GPU progress
    With concurrent binning BR waits for the visibility pass that BV runs,
    so BR's ring and IB words stand still while the GPU works. The hang
    check and the stall handler read the BV read pointer, IB bases and
    dwords left as well, and treat a change in them as progress.
  • feat(memory): treat memory the kernel adopted outside its range as kernel memory
    A kernel may manage DRAM below its own physical range (a second bank it
    adopts as VM memory). Cache maintenance, the write-combined attribute,
    the sampled readers and the owned-page bitmap now ask the pmap whether a
    page is managed instead of comparing against gPhysBase and gPhysSize, so
    a buffer object with such pages is maintained like any other. The first
    failed kernel allocation is recorded (where, status, size, address bits,
    contiguous), and each CPU records the pool run it is zero-filling.
    
    Run on hardware: the GPU starts and runs with 458 MiB of adopted memory.
  • feat(pool): return a viewed grant's pages once the kernel reports no mapping is left
    A client's view of a pool BO is a physical-range descriptor; a task can
    copy its mapping in the VM, and the copy refers to the device pager's VM
    object, not to the descriptor, so the view's free did not mean the pages
    were unmapped and every viewed grant was retired for good (run68: 77
    grants, 25524 KiB).
    
    Where the kernel has iokit_physical_descriptor_close_notify (XNU patch
    gpupool-device-pager-close), a view registers the grant with it before
    anything can map the view, and its reference to the grant goes only when
    the kernel reports the pager's close: IOKit has released the descriptor
    and the VM has reported the object's last unmap, which it does only once
    no map entry, memory entry or copy object refers to it. The pages then
    return to the pool, marked for the cache pass before their next zero
    fill. A view the kernel cannot track (the symbol is weak; kernels
    without the patch) retires its grant as before, and gpu-pool=2 still
    recycles those too. native_adreno_pool_views counts both kinds, the
    reports and the grants given back.
    
    Not run on hardware.
  • perf(submit): run the line audits only with gpu-audit=1 and time every job
    The submission's audit phase compared CPU cache lines with DRAM for the
    write-combined IBs of the first 16 submissions and for the synchronized
    driver buffers of the first 8 and every 1024th. They only detect a CPU
    cache that holds a line the GPU should see; they found none in any run
    (dirty=0), pool BOs have no cacheable alias for them to compare, and the
    write-combine re-assertion, the correction, still runs every time. In
    run68 the 8 buffer audits were 11.9 of the phase's 12.1 ms (1.5 ms each
    under the driver's lock), and every 1024th submission paid one more.
    gpu-audit=1 runs them as before.
    
    Every retired job is now timed, with or without diag=1: by the CP's
    stamps when gpu-timeline is on, else by the driver's clock (from the later
    of its submission and the previous retirement on its ring). The 8 longest
    since boot and the newest 16 over 8 ms are kept with pid, queue, ring,
    priority, IB count and bytes (native_adreno_long_jobs, _slow_jobs,
    _job_time).
    
    A stall's first stage (write pointer again, translations dropped) now
    records the CP's IB progress at the fence's first timed-out wait, at the
    stage and 500 us later, the job, and the time from the stage to the job's
    retirement (native_adreno_stall_kick). The CP snapshot is taken before the
    stage acts, so its registers show the stall rather than the stage's
    result.
    
    Not run on hardware.
  • feat(memory): take write-combined system BOs from a pool of DRAM outside the kernel's
    The board registers free DRAM the kernel does not manage with native_adreno_pool_add. A write-combined system BO first asks the pool (NativeAdrenoPool.cpp: a bitmap a range, best single run, else the longest runs first, zero fill through the range's own write-combined kernel mapping before a grant is handed out); whatever the pool does not serve takes the descriptor path unchanged. A pool BO has a grant and no descriptor: its spans are the grant's runs, nothing of it is cleaned, audited or read through the cacheable aperture, its client view maps its physical runs write-combined and holds the grant until IOKit frees the view, and an import of pool pages is refused. Pages a client had a view of are retired unless gpu-pool=2; gpu-pool=0 turns the pool off. Counters: native_adreno_pool_state, native_adreno_pool_fallbacks, native_adreno_pool_zero.
    
    Host tests: test_pool (allocator against a page model) and test_pool_memory (BO paths); the existing 23 pass as before, test_client and test_gmuconfig fail as at the base. Every changed kernel source passes the kernel's compile recipe as a syntax check. Not built into an image and not run on hardware.
  • perf(maintenance): leave the driver's synchronized buffers out of the per-submission cache passes
    Every process table maps the GPU context's eleven kernel buffers (8548 KiB,
    two preemption records of 4192 KiB among them), and each submission cleaned
    and invalidated all of them: 11 passes and 8548 KiB a submission in macOS
    runs 57 to 61, about 351 of a submission's 579 us in run61.
    
    Their users keep the GPU's view current themselves (a non-cacheable view, or
    a clean after a store through the kernel's mapping), so the buffers are
    declared synchronized and a map, an unmap and a submission leave them alone.
    The probe's store through the kernel's mapping is cleaned where it is made.
    The first 8 submissions and every 1024th audit the head of each buffer
    against DRAM; a dirty line restores the maintenance for good.
    
    A submission, a completion and the write-combine re-assertion visit only the
    mappings they have work for instead of every mapping of the client. The
    submission log reads the IB words only with diag=1 and no longer through one
    kernel mapping per word.
    
    gpu-maint-all=1 restores the passes over everything for a comparison.
    Counters: native_adreno_maint_kind, native_adreno_maint_walk,
    native_adreno_sync_audit, native_adreno_sync_skips, native_adreno_submit_us.
    
    Host tests cover it (test_maintenance, test_gpuctx, test_submit). Not run on
    hardware.
  • fix(wait): wake fence waiters from the completion pass
    A waiter slept on the board's pause (the next GPU interrupt of any kind, at
    most 1 ms) and then polled: 34767 of 42325 pauses of one boot ended by the
    timeout. Three defects came with it:
    
    - The poll before the sleep used the display's completed prefix, which stays
      below a finished fence of the high ring while an older fence of the low
      ring is pending. Such a waiter slept until its deadline (100 ms slices)
      with its own fence written.
    - Nothing looked at the switch between the rings when a fence completed, so
      a low-ring job queued behind the high ring got the CP only at the next
      submission, zero-timeout poll or expired deadline.
    - A finished switch was dropped when its handler found the lock busy, and
      the incoming ring's write pointer waited for the next locked call.
    
    The driver exports native_adreno_gpu_interrupt(status) for the board's GPU
    interrupt handler to call with the status word it read and cleared. A fence
    or a finished switch asks for the completion pass on a thread call:
    it retires finished jobs (which runs the ring switch decision and completes a
    finished switch), then wakes exactly the waiters whose fence the CP has
    written on its own ring (NativeAdrenoSubmit::fencePending). The hang check
    runs one pass per period for an interrupt that never came. A waiter arms
    itself before it looks at the fence and sleeps on its own record until the
    pass disarms it or the caller's deadline passes, so no completion is lost
    and no timed poll is left. Until the board delivers interrupts, or with
    gpu-fence-wake=0, the waiter keeps the board's pause, with the per-ring
    check. Counters: native_adreno_fence_wake. Host tests cover fencePending on
    one and two rings. Not run on hardware.
  • fix(submit): look at the ring switch after every pause of a wait
    A wait with a timeout runs with the driver's lock held for all of it (a
    connection's teardown and the table hand-back wait 2 s, a submission's
    wait_fence, the board's calls). It decided the switch between the two rings
    once, at its start, and then only read the fence word between pauses, while
    the lock kept every other caller of the switch out. A low-ring job behind a
    busy high ring therefore stayed without the CP after the high ring went idle,
    for the whole timeout; a switch the CP finished meanwhile was not completed
    either. The hang check then found a low-ring job older than 2 s with the
    high ring idle, took the ring for unreachable, gave it up and restarted the
    GPU (one boot log shows a 2 s wait followed by "hangs 1 restarts 1", "dead 3
    rescues 1" and a 2.17 s wait of another caller).
    
    wait() and observeFence() go through waitOnRing(), which with two rings
    runs driveSwitch again after every pause. A host test plays the CP from the
    pause: without the change the wait times out, with it the switch is
    triggered, completed and the low-ring job finishes. Not run on hardware.
  • feat(diagnostics): count imported memory and BO views by process
    native_adreno_import_memory holds the imports alive, their KiB, the peak and
    the imports made; native_adreno_import_owners holds the importing processes
    (pid, live KiB, live imports, peak KiB). An import pins its pages from
    importTask until release, so these are the bytes the driver keeps wired
    besides its own BOs.
    
    native_adreno_view_owners holds the BO views each process maps. A view is an
    IOKit mapping, which the VM bills to the mapping task by its whole size
    (iokit_mapped, part of the footprint the memory limits judge).
    
    The holder table of native_adreno_bo_owners is shared by the three tables
    (NativeAdrenoNoteHolder). Host tests cover the holder rows and the import
    accounting. Not run on hardware.
  • feat(adreno): job timeline, stall ring and CP counter stamps
    Framed submissions stamp the CP always-on counter before and after their buffers (sheng-gpu-timeline, default on); the retired jobs, the stall handler's actions and a ring status word are kept for the boot export. Drops the large-import record. Host tests cover the stamps and the shim symbols.
  • perf(memory): make and wire BoAlloc's pages outside the lock
    The descriptor of a user BO's system memory is made, wired and, for
    default-attribute pages, written back before the driver's lock is taken;
    allocateSystem adopts it under the lock and pages it does not take are
    given back after the call. i14aef633e held the lock 4.0 s in 5454
    allocations while every other process's submit and wait poll queued.
  • test(hostsim): answer only the GPU services, pass the rest to IOKit
    A host instrument that also uses IOSurface or another system service
    (CARenderer over the frontend) needs those services from the system;
    the shim answered every IOKit call with the GPU's.
  • perf(memory): user BOs from system memory, no fixed BO pool
    User BOs are pageable IOBufferMemoryDescriptors with no task and no
    kernel mapping, wired by prepare(), as Metal resources are system
    memory. The VM zero-fills them and sets the write-combined attribute in
    one batch while wiring, and restores it when the object is reaped.
    Client views map the descriptor itself, so no device-pager view resets
    the page attribute and submissions no longer re-assert it.
    
    The 1536 MiB boot-time pool is gone. In untagged-ed1611b1 it filled
    with the runtime's blocks, 404 BOs fell back to scattered kernel memory,
    96 allocations of 96 MiB failed and the pressure flag stayed set.
    Memory pressure is now reported when the system refuses a BO.
    Imports are proven exclusive through a page bitmap of the driver's own
    BOs instead of a pairwise span check.
  • test(adreno): host doubles and deferred BO checks for the current driver
    The host fixture gains kIOReturnInternalError and the hostsim the
    maintenance, page-attribute and pmap doubles the memory code calls. The
    deferred BO checks expect the growing retire queue (no wait for the GPU)
    and BOs the cache may keep. NativeAdrenoMemory no longer includes
    kern/clock.h.
  • feat(adreno): grow object, queue and job tables with demand
    No fixed object count: the handle index, the VM's mappings and ranges,
    a client's entries and its BO views grow with memory only. Queues grow
    as msm's submitqueues do; each client owns a 2^32 range of the shared
    queue space. Jobs are bounded by ring space: a full ring answers
    kIOReturnBusy, as msm waits for room. A BO is limited by the client's
    GPU address window, which Query reports as max_bo_bytes; the 256 MiB
    budget and the ABI's queue and size constants are gone.
  • feat(adreno): grow the BO pool in contiguous chunks
    The pool is a set of physically contiguous chunks instead of one fixed
    block. The board's reservation is the first chunk and stays; when no
    chunk has room a new one is taken from the system (2 MiB, the 4 KiB
    granule's block size, halved down to 64 KiB, or a large BO's own size),
    write-combined once as it is made, and given back when it empties.
    Memory pressure is reported only when the system cannot give a chunk
    (that BO falls back to scattered pages) and ends once the pool serves an
    allocation again; the seven-eighths rule and the 2 s window are gone.
  • feat(adreno): msm hangcheck and GPU restart with replay
    A thread checks the ring every 500 ms as msm's hangcheck_handler does:
    no fence progress with work pending and no CP progress (a6xx_progress,
    at most three retries) is a hang. Recovery follows recover_worker: the
    hung job's VM takes the fault and, unless it is the kernel's, becomes
    unusable; the fence moves past it; the board's restart hook resets the
    GPU and runs hw_init and CP init on the same ring; every later job is
    replayed with its own fence. Clients read the global fault count plus
    their own VM's, so only the guilty client sees its device lost. The
    ring gains restart, unfenced submission, a6xx_idle, progress state and
    the GBIF bus halt; CP init can run unfenced.
  • feat(adreno): on-demand GPU page tables and a 2^ias address space
    Each GPU address space takes its table pages one at a time from the
    system (NativeAdrenoTablePages) instead of a 1027-page pool, so the
    aperture is the bank's whole input size: ias from ID2.UBS as Linux
    arm_smmu_id_size_to_bits reads it. Every process maps from
    ADRENO_VM_START (4 GiB) to 2^ias, as msm's adreno_private_vm_size.
    The GMU keeps its 0..2 GiB space. The alias scratch belongs to each
    VM and grows with its range table. A map that cannot get a table page
    is undone and returns kIOReturnNoMemory without poisoning the VM.
    Host tests cover a sparse 48-bit space and the refusal.
  • perf(adreno): count cache maintenance per phase and skip uncached imports
    Phase counters for the pool, allocate, map, submit, unmap, complete and table passes. Imports the owner maps uncached need no maintenance. Process page tables come from cacheable blocks with the walk coherence proved at board start, so a table page is ordered instead of cleaned. A freed BO whose first pages a user task still maps is counted. Submit records, handle index and user client changes of the iPadOS 27 bring-up are included.
  • fix(ring): write WPTR as a fenced register and keep the ring's register window alive
    The ring kept a pointer to a register window on the stack of the service's
    start routine. The window is a service member now. WPTR writes check the
    GMU AHB fence status and are given again while it reports the write
    dropped.
  • feat(submit): take a whole frame of up to 128 IBs in one submission
    ABI version 2 raises the IB list to 128 entries, so the ring prologue
    (cache flush, idle wait, thread switch) runs once per frame instead of
    once per 16 IBs. The submit lists live in the client and submit objects
    instead of the kernel stack. The stall handler drops the translations and
    gives the write pointer again after 10 waits; the IB counts per submission
    are recorded.
  • fix(userclient): serve metal clients beside the render server
    - Four client windows: the render server keeps 1 GiB, and the
      wallpaper, icon and widget processes share the rest of the aperture
      (170 MiB each).
    - A client that owns no queue range yet is no longer refused by
      setScope, held up in close or waited on in releaseAll because the
      render server has jobs in flight in the default range 0 (int-200).
      New regression block in the user-client host test.
    - BoCpuUnmap keeps the shared CPU view while another mapping lease of
      the same BO is held; dropping it unmapped the other holder's pages
      and crashed the render server in vkCmdCopyBuffer (int-177).
    - Diagnostics: the reason the latest mount failed, mappings that fall
      outside a client window, a text row per non-render-server process
      and a name for the mailbox poller thread.
  • feat(publish): pool only user BOs and serve one connection per process
    The SMMU tables and ring/fence/SQE buffers no longer come from the
    write-combined pool, so CPU table walks stay cached. The published
    service keeps one user-client core per connection: the render server
    gets the 1 GiB window at 0x20000000 and two other processes 256 MiB
    each, with their own queue ranges and teardown.
  • Import NativeAdreno
    Co-authored-by: rpone <rponeawa@qq.com>