An address space's check of the SMMU returned the fault status while the
GPU's own faulting access was still latched, and the address space that
happened to map, unmap or attach next was taken out of use: seven processes
in one run, the render server among them. The status says nothing about
that address space's tables, so the operation now goes on and the fault is
counted. The hang check ends the fault, and also clears the SMMU's global
fault status where it is set.
A stalled job's command buffers are searched for the IB2 address the thread
stopped in, to tell an address the job's commands named from one they did
not.
An address space taken out of use quarantined every buffer it mapped,
including the context's shared buffers that every address space maps. No
address space could be made afterwards, so a restarted render server never
got a device. Shared buffers are now declared as such and stay in use; the
failed address space's reference still keeps them from being released.
A job of a process whose address space is out of use ends without retiring
its references instead of stopping the submission engine for everyone.
The address space failure records its status and source line, and every
exit of the recovery routine has a trace code.
Every process table maps the GPU context's eleven kernel buffers (8548 KiB,
two preemption records of 4192 KiB among them), and each submission cleaned
and invalidated all of them: 11 passes and 8548 KiB a submission in macOS
runs 57 to 61, about 351 of a submission's 579 us in run61.
Their users keep the GPU's view current themselves (a non-cacheable view, or
a clean after a store through the kernel's mapping), so the buffers are
declared synchronized and a map, an unmap and a submission leave them alone.
The probe's store through the kernel's mapping is cleaned where it is made.
The first 8 submissions and every 1024th audit the head of each buffer
against DRAM; a dirty line restores the maintenance for good.
A submission, a completion and the write-combine re-assertion visit only the
mappings they have work for instead of every mapping of the client. The
submission log reads the IB words only with diag=1 and no longer through one
kernel mapping per word.
gpu-maint-all=1 restores the passes over everything for a comparison.
Counters: native_adreno_maint_kind, native_adreno_maint_walk,
native_adreno_sync_audit, native_adreno_sync_skips, native_adreno_submit_us.
Host tests cover it (test_maintenance, test_gpuctx, test_submit). Not run on
hardware.
Add hardware CP context switch via CP_CONTEXT_SWITCH_CNTL on a second
ring. Use a conservative BO-disjoint rule: only allow a queued job to
preempt the running ring when their buffer sets share no handle. Two
4192 KiB preemption records and two a7xx_cp_smmu_info pages are mapped
at fixed IOVAs. Ring priority comes from the kernel GPU role of the
submitting task (BACKGROUND/UTILITY → ring 1, UI/focal/realtime → ring 0).
Diagnostics gated on sheng-diag=1; feature gated on sheng-gpu-preempt=1.
User BOs are pageable IOBufferMemoryDescriptors with no task and no
kernel mapping, wired by prepare(), as Metal resources are system
memory. The VM zero-fills them and sets the write-combined attribute in
one batch while wiring, and restores it when the object is reaped.
Client views map the descriptor itself, so no device-pager view resets
the page attribute and submissions no longer re-assert it.
The 1536 MiB boot-time pool is gone. In untagged-ed1611b1 it filled
with the runtime's blocks, 404 BOs fell back to scattered kernel memory,
96 allocations of 96 MiB failed and the pressure flag stayed set.
Memory pressure is now reported when the system refuses a BO.
Imports are proven exclusive through a page bitmap of the driver's own
BOs instead of a pairwise span check.
After a hang the guilty VM became unusable and reported its device
lost, as msm does. When that client was the render server, backboardd
ended itself and SpringBoard's restart stalled on callservicesd, so the
desktop never came back (untagged-1f4ba0a24). As Metal does after a GPU
restart, only the hung command buffer is lost: the process keeps its
device, its queued work runs again, and only a restart that failed is
everyone's loss. The fault check before a restart ignores FSR's format
bits (0x400), which read as a fault on every hang.
A hang with a translation fault left the context bank stalled (FSR.SS)
and the restart's bus halt failed with kIOReturnNotResponding, so every
client lost its device. As msm does before recovery, the stalled
transaction is terminated through CB_RESUME and FSR cleared. The stall
snapshot now reads the hung job's IBs through its own address space.
The pool is a set of physically contiguous chunks instead of one fixed
block. The board's reservation is the first chunk and stays; when no
chunk has room a new one is taken from the system (2 MiB, the 4 KiB
granule's block size, halved down to 64 KiB, or a large BO's own size),
write-combined once as it is made, and given back when it empties.
Memory pressure is reported only when the system cannot give a chunk
(that BO falls back to scattered pages) and ends once the pool serves an
allocation again; the seven-eighths rule and the 2 s window are gone.
Each GPU address space takes its table pages one at a time from the
system (NativeAdrenoTablePages) instead of a 1027-page pool, so the
aperture is the bank's whole input size: ias from ID2.UBS as Linux
arm_smmu_id_size_to_bits reads it. Every process maps from
ADRENO_VM_START (4 GiB) to 2^ias, as msm's adreno_private_vm_size.
The GMU keeps its 0..2 GiB space. The alias scratch belongs to each
VM and grows with its range table. A map that cannot get a table page
is undone and returns kIOReturnNoMemory without poisoning the VM.
Host tests cover a sparse 48-bit space and the refusal.
With the arg set and the coherence proof passed, the GPU's own BOs are not write-combined: the ABI reports NativeAdrenoParamsOwnedCoherent so the client maps its CPU view write-back, and the GPU maps the pages write-back inner-shareable. The write-combined audit and per-submission cache maintenance are skipped for them. Default off.
Phase counters for the pool, allocate, map, submit, unmap, complete and table passes. Imports the owner maps uncached need no maintenance. Process page tables come from cacheable blocks with the walk coherence proved at board start, so a table page is ordered instead of cleaned. A freed BO whose first pages a user task still maps is counted. Submit records, handle index and user client changes of the iPadOS 27 bring-up are included.
Pooled BOs no longer change page attributes one pmap call at a time on
allocation and release; the pool is made write-combined when reserved
and a pooled BO is zeroed through the pool mapping. Map and unmap
observe only the global policy, the context and the owned routes; the
full slot scan stays in the audit every 64 operations.