Commit Graph

39 Commits

  • Preserve pending jobs during context-switch recovery
    Count each failed switch once. Replay the oldest unfinished job during scheduling rescue, and route captured SMMU faults through guilty-job recovery. Trace event 70 retains the fault, active table, restart prefix and recovery outcome. The cause of the observed null-address fault remains unresolved.
    
    Validation: focused submit fixture passes with AddressSanitizer and UndefinedBehaviorSanitizer, including repeated timeout observation, oldest-job replay and fault classification.
  • fix(recovery): a GPU access fault no longer fails other address spaces
    An address space's check of the SMMU returned the fault status while the
    GPU's own faulting access was still latched, and the address space that
    happened to map, unmap or attach next was taken out of use: seven processes
    in one run, the render server among them. The status says nothing about
    that address space's tables, so the operation now goes on and the fault is
    counted. The hang check ends the fault, and also clears the SMMU's global
    fault status where it is set.
    
    A stalled job's command buffers are searched for the IB2 address the thread
    stopped in, to tell an address the job's commands named from one they did
    not.
  • fix(recovery): keep the GPU usable after one process's address space fails
    An address space taken out of use quarantined every buffer it mapped,
    including the context's shared buffers that every address space maps. No
    address space could be made afterwards, so a restarted render server never
    got a device. Shared buffers are now declared as such and stay in use; the
    failed address space's reference still keeps them from being released.
    
    A job of a process whose address space is out of use ends without retiring
    its references instead of stopping the submission engine for everyone.
    
    The address space failure records its status and source line, and every
    exit of the recovery routine has a trace code.
  • fix(submit): blame no job for a ring switch the CP never finished
    A first-ring job that waits while the CP stays on the second ring after
    a switch back timed out is stranded, as a second-ring job already was:
    the restart replays every job and no client loses its device. The
    switch timeout record also carries CP_CONTEXT_SWITCH_CNTL and the two
    rings of the switch.
  • fix(submit): count the BV thread's progress as GPU progress
    With concurrent binning BR waits for the visibility pass that BV runs,
    so BR's ring and IB words stand still while the GPU works. The hang
    check and the stall handler read the BV read pointer, IB bases and
    dwords left as well, and treat a change in them as progress.
  • fix(submit): leave a working GPU alone in the stall handler
    The first stage of the stall handler dropped the translations and gave
    the write pointer again once a fence had waited 50 ms, although its own
    sample showed the CP moving: every such case of four runs was a long
    job. A sample that finds the CP moving now restarts the clock, and the
    stages act only after their time without progress.
    
    Run on hardware: four samples of moving jobs, none acted on; no hang in
    two runs.
  • feat(submit): keep the CP state of a ring switch that never finished
    The first two switches the CP does not finish within the watchdog keep
    the CP's stall words and the ring's progress words for the board log.
    
    Not run on hardware.
  • perf(submit): run the line audits only with gpu-audit=1 and time every job
    The submission's audit phase compared CPU cache lines with DRAM for the
    write-combined IBs of the first 16 submissions and for the synchronized
    driver buffers of the first 8 and every 1024th. They only detect a CPU
    cache that holds a line the GPU should see; they found none in any run
    (dirty=0), pool BOs have no cacheable alias for them to compare, and the
    write-combine re-assertion, the correction, still runs every time. In
    run68 the 8 buffer audits were 11.9 of the phase's 12.1 ms (1.5 ms each
    under the driver's lock), and every 1024th submission paid one more.
    gpu-audit=1 runs them as before.
    
    Every retired job is now timed, with or without diag=1: by the CP's
    stamps when gpu-timeline is on, else by the driver's clock (from the later
    of its submission and the previous retirement on its ring). The 8 longest
    since boot and the newest 16 over 8 ms are kept with pid, queue, ring,
    priority, IB count and bytes (native_adreno_long_jobs, _slow_jobs,
    _job_time).
    
    A stall's first stage (write pointer again, translations dropped) now
    records the CP's IB progress at the fence's first timed-out wait, at the
    stage and 500 us later, the job, and the time from the stage to the job's
    retirement (native_adreno_stall_kick). The CP snapshot is taken before the
    stage acts, so its registers show the stall rather than the stage's
    result.
    
    Not run on hardware.
  • perf(maintenance): leave the driver's synchronized buffers out of the per-submission cache passes
    Every process table maps the GPU context's eleven kernel buffers (8548 KiB,
    two preemption records of 4192 KiB among them), and each submission cleaned
    and invalidated all of them: 11 passes and 8548 KiB a submission in macOS
    runs 57 to 61, about 351 of a submission's 579 us in run61.
    
    Their users keep the GPU's view current themselves (a non-cacheable view, or
    a clean after a store through the kernel's mapping), so the buffers are
    declared synchronized and a map, an unmap and a submission leave them alone.
    The probe's store through the kernel's mapping is cleaned where it is made.
    The first 8 submissions and every 1024th audit the head of each buffer
    against DRAM; a dirty line restores the maintenance for good.
    
    A submission, a completion and the write-combine re-assertion visit only the
    mappings they have work for instead of every mapping of the client. The
    submission log reads the IB words only with diag=1 and no longer through one
    kernel mapping per word.
    
    gpu-maint-all=1 restores the passes over everything for a comparison.
    Counters: native_adreno_maint_kind, native_adreno_maint_walk,
    native_adreno_sync_audit, native_adreno_sync_skips, native_adreno_submit_us.
    
    Host tests cover it (test_maintenance, test_gpuctx, test_submit). Not run on
    hardware.
  • fix(wait): wake fence waiters from the completion pass
    A waiter slept on the board's pause (the next GPU interrupt of any kind, at
    most 1 ms) and then polled: 34767 of 42325 pauses of one boot ended by the
    timeout. Three defects came with it:
    
    - The poll before the sleep used the display's completed prefix, which stays
      below a finished fence of the high ring while an older fence of the low
      ring is pending. Such a waiter slept until its deadline (100 ms slices)
      with its own fence written.
    - Nothing looked at the switch between the rings when a fence completed, so
      a low-ring job queued behind the high ring got the CP only at the next
      submission, zero-timeout poll or expired deadline.
    - A finished switch was dropped when its handler found the lock busy, and
      the incoming ring's write pointer waited for the next locked call.
    
    The driver exports native_adreno_gpu_interrupt(status) for the board's GPU
    interrupt handler to call with the status word it read and cleared. A fence
    or a finished switch asks for the completion pass on a thread call:
    it retires finished jobs (which runs the ring switch decision and completes a
    finished switch), then wakes exactly the waiters whose fence the CP has
    written on its own ring (NativeAdrenoSubmit::fencePending). The hang check
    runs one pass per period for an interrupt that never came. A waiter arms
    itself before it looks at the fence and sleeps on its own record until the
    pass disarms it or the caller's deadline passes, so no completion is lost
    and no timed poll is left. Until the board delivers interrupts, or with
    gpu-fence-wake=0, the waiter keeps the board's pause, with the per-ring
    check. Counters: native_adreno_fence_wake. Host tests cover fencePending on
    one and two rings. Not run on hardware.
  • feat(diagnostics): record which ring a stall snapshot is about
    The stall snapshot reads the registers of the ring the CP runs and names the
    fence after the waited ring's last written one, which with two rings can
    belong to the other ring or be retired already. native_adreno_stall_ring
    keeps, with each snapshot, the waited fence and its ring, the ring the CP
    runs, a switch in flight, the unfinished jobs of each ring, the named
    fence's job and how long the waited job has been submitted, so a job that
    waits for the ring switch can be told from a CP that stopped. Not run on
    hardware.
  • fix(submit): look at the ring switch after every pause of a wait
    A wait with a timeout runs with the driver's lock held for all of it (a
    connection's teardown and the table hand-back wait 2 s, a submission's
    wait_fence, the board's calls). It decided the switch between the two rings
    once, at its start, and then only read the fence word between pauses, while
    the lock kept every other caller of the switch out. A low-ring job behind a
    busy high ring therefore stayed without the CP after the high ring went idle,
    for the whole timeout; a switch the CP finished meanwhile was not completed
    either. The hang check then found a low-ring job older than 2 s with the
    high ring idle, took the ring for unreachable, gave it up and restarted the
    GPU (one boot log shows a 2 s wait followed by "hangs 1 restarts 1", "dead 3
    rescues 1" and a 2.17 s wait of another caller).
    
    wait() and observeFence() go through waitOnRing(), which with two rings
    runs driveSwitch again after every pause. A host test plays the CP from the
    pause: without the change the wait times out, with it the switch is
    triggered, completed and the low-ring job finishes. Not run on hardware.
  • refactor: drop the sheng- prefix from boot arguments
    Every boot argument loses its sheng- prefix (sheng-touch=4 is now
    touch=4). No XNU or Apple kext boot argument of iPadOS 27 or macOS 27.3
    uses any of the new names.
  • fix(diagnostics): keep only seven-issue GPU trace events
    The kernel ring keeps submission, retirement, display arrival, fence and
    latch, build identity and the render server's render, encode and completion
    thread identities. Pixel capture, dispatch hooks and the switch, recovery,
    fence-query and import events are removed; the ring holds 65536 records.
  • feat(preempt): give up a second ring the CP cannot reach
    A switch that does not complete, a trigger whose register writes were
    refused, or a low-ring job left waiting with the high ring idle ends the
    second ring for good. recover() then starts from the oldest job that never
    ran and replays it on the first ring, whichever ring it sat on, so a
    broken switch costs the background work and not the session.
    
    Ring 0 is the high ring: ring 1 is only switched to when ring 0 has no
    pending job, or the low job has waited PreemptAgingUs. Only priorities 2
    and 3 take ring 1.
    
    A ring the CP does not run keeps its write pointer for the switch to
    give, and every framed submission on a ring with a preemption record ends
    with the yield.
    
    The preemption record keeps what the CP saved into it; only the write
    pointer is rewritten before a switch.
    
    The switch log records the registers and both records around each switch,
    and a switch that times out carries the CP's IB state.
  • feat(gpu): CP preemption between two rings (A1)
    Add hardware CP context switch via CP_CONTEXT_SWITCH_CNTL on a second
    ring. Use a conservative BO-disjoint rule: only allow a queued job to
    preempt the running ring when their buffer sets share no handle. Two
    4192 KiB preemption records and two a7xx_cp_smmu_info pages are mapped
    at fixed IOVAs. Ring priority comes from the kernel GPU role of the
    submitting task (BACKGROUND/UTILITY → ring 1, UI/focal/realtime → ring 0).
    Diagnostics gated on sheng-diag=1; feature gated on sheng-gpu-preempt=1.
  • feat(timeline): owner pid and retire time per GPU job
    Each timeline entry records the submitting process and when the driver saw the job retired, so a frame's wait for the ring and for completion can be split.
  • feat(adreno): allocate the job timeline at attach and grow it to 16384 jobs
    Same reason as the swap ring: the static array no longer fits the kernel data segment. The stamps are recorded only when the ring was allocated.
  • feat(adreno): job timeline, stall ring and CP counter stamps
    Framed submissions stamp the CP always-on counter before and after their buffers (sheng-gpu-timeline, default on); the retired jobs, the stall handler's actions and a ring status word are kept for the boot export. Drops the large-import record. Host tests cover the stamps and the shim symbols.
  • fix(adreno): end a stalled SMMU fault before a GPU restart
    A hang with a translation fault left the context bank stalled (FSR.SS)
    and the restart's bus halt failed with kIOReturnNotResponding, so every
    client lost its device. As msm does before recovery, the stalled
    transaction is terminated through CB_RESUME and FSR cleared. The stall
    snapshot now reads the hung job's IBs through its own address space.
  • feat(adreno): grow object, queue and job tables with demand
    No fixed object count: the handle index, the VM's mappings and ranges,
    a client's entries and its BO views grow with memory only. Queues grow
    as msm's submitqueues do; each client owns a 2^32 range of the shared
    queue space. Jobs are bounded by ring space: a full ring answers
    kIOReturnBusy, as msm waits for room. A BO is limited by the client's
    GPU address window, which Query reports as max_bo_bytes; the 256 MiB
    budget and the ABI's queue and size constants are gone.
  • feat(adreno): msm hangcheck and GPU restart with replay
    A thread checks the ring every 500 ms as msm's hangcheck_handler does:
    no fence progress with work pending and no CP progress (a6xx_progress,
    at most three retries) is a hang. Recovery follows recover_worker: the
    hung job's VM takes the fault and, unless it is the kernel's, becomes
    unusable; the fence moves past it; the board's restart hook resets the
    GPU and runs hw_init and CP init on the same ring; every later job is
    replayed with its own fence. Clients read the global fault count plus
    their own VM's, so only the guilty client sees its device lost. The
    ring gains restart, unfenced submission, a6xx_idle, progress state and
    the GBIF bus halt; CP init can run unfenced.
  • perf(adreno): count cache maintenance per phase and skip uncached imports
    Phase counters for the pool, allocate, map, submit, unmap, complete and table passes. Imports the owner maps uncached need no maintenance. Process page tables come from cacheable blocks with the walk coherence proved at board start, so a table page is ordered instead of cleaned. A freed BO whose first pages a user task still maps is counted. Submit records, handle index and user client changes of the iPadOS 27 bring-up are included.
  • perf(submit): stop emptying the GPU TLB on every submission
    A map fills only invalid leaves, an unmap drops its own pages before it
    returns and the tables above the leaves never change, so the TLBIALL
    before each submission is dropped (10944 in i568, each frame then began
    with page walks). A stalled fence still invalidates.
    
    The coherence sample now runs on every 64th submission, and the IB
    audit only on the first 16: i566 found no dirty line in 2765132 lines
    over 1358 audits of about 2 ms each under the publish lock.
    
    Not yet measured on hardware.
  • fix(submit): time the stall handler instead of counting waits
    A job lasts about 5 ms against waits of 1 ms or less, so counting waits
    treated ordinary jobs as stalls: i553 ran the handler on 6712 of 6911
    jobs. The TLB invalidate and write pointer kick now run after 50 ms
    without progress, the bus drain after 200 ms, and the CP snapshot at
    50 ms and 1 s. How long each fence stayed pending is kept as a
    histogram.
  • feat(submit): take a whole frame of up to 128 IBs in one submission
    ABI version 2 raises the IB list to 128 entries, so the ring prologue
    (cache flush, idle wait, thread switch) runs once per frame instead of
    once per 16 IBs. The submit lists live in the client and submit objects
    instead of the kernel stack. The stall handler drops the translations and
    gives the write pointer again after 10 waits; the IB counts per submission
    are recorded.
  • feat(submit): drain the GPU bus and re-issue the write pointer when a fence stalls
    After 100 timed-out waits the GPU's translations are dropped and CP_RB_WPTR is written again; after 400 the GBIF client and AXI requests are halted and released as before a GPU reset. The counters are exported for the board log. GPU-written imports are mapped non-coherent and skipped by the per-submission cache work.
  • feat(publish): pool only user BOs and serve one connection per process
    The SMMU tables and ring/fence/SQE buffers no longer come from the
    write-combined pool, so CPU table walks stay cached. The published
    service keeps one user-client core per connection: the render server
    gets the 1 GiB window at 0x20000000 and two other processes 256 MiB
    each, with their own queue ranges and teardown.
  • Import NativeAdreno
    Co-authored-by: rpone <rponeawa@qq.com>