An address space's check of the SMMU returned the fault status while the
GPU's own faulting access was still latched, and the address space that
happened to map, unmap or attach next was taken out of use: seven processes
in one run, the render server among them. The status says nothing about
that address space's tables, so the operation now goes on and the fault is
counted. The hang check ends the fault, and also clears the SMMU's global
fault status where it is set.
A stalled job's command buffers are searched for the IB2 address the thread
stopped in, to tell an address the job's commands named from one they did
not.
An address space taken out of use quarantined every buffer it mapped,
including the context's shared buffers that every address space maps. No
address space could be made afterwards, so a restarted render server never
got a device. Shared buffers are now declared as such and stay in use; the
failed address space's reference still keeps them from being released.
A job of a process whose address space is out of use ends without retiring
its references instead of stopping the submission engine for everyone.
The address space failure records its status and source line, and every
exit of the recovery routine has a trace code.
A first-ring job that waits while the CP stays on the second ring after
a switch back timed out is stranded, as a second-ring job already was:
the restart replays every job and no client loses its device. The
switch timeout record also carries CP_CONTEXT_SWITCH_CNTL and the two
rings of the switch.
With concurrent binning BR waits for the visibility pass that BV runs,
so BR's ring and IB words stand still while the GPU works. The hang
check and the stall handler read the BV read pointer, IB bases and
dwords left as well, and treat a change in them as progress.
The first stage of the stall handler dropped the translations and gave
the write pointer again once a fence had waited 50 ms, although its own
sample showed the CP moving: every such case of four runs was a long
job. A sample that finds the CP moving now restarts the clock, and the
stages act only after their time without progress.
Run on hardware: four samples of moving jobs, none acted on; no hang in
two runs.
native_adreno_map_met keeps, for the last failed BoMap, the address and
bytes asked, whether the handle was mapped already, and the mapping at
or past that address: its address, bytes, handle, jobs and kind.
Run on hardware: a refused map at the window's first address showed
another mapping of the same size there.
The first two switches the CP does not finish within the watchdog keep
the CP's stall words and the ring's progress words for the board log.
Not run on hardware.
A kernel may manage DRAM below its own physical range (a second bank it
adopts as VM memory). Cache maintenance, the write-combined attribute,
the sampled readers and the owned-page bitmap now ask the pmap whether a
page is managed instead of comparing against gPhysBase and gPhysSize, so
a buffer object with such pages is maintained like any other. The first
failed kernel allocation is recorded (where, status, size, address bits,
contiguous), and each CPU records the pool run it is zero-filling.
Run on hardware: the GPU starts and runs with 458 MiB of adopted memory.
A client's view of a pool BO is a physical-range descriptor; a task can
copy its mapping in the VM, and the copy refers to the device pager's VM
object, not to the descriptor, so the view's free did not mean the pages
were unmapped and every viewed grant was retired for good (run68: 77
grants, 25524 KiB).
Where the kernel has iokit_physical_descriptor_close_notify (XNU patch
gpupool-device-pager-close), a view registers the grant with it before
anything can map the view, and its reference to the grant goes only when
the kernel reports the pager's close: IOKit has released the descriptor
and the VM has reported the object's last unmap, which it does only once
no map entry, memory entry or copy object refers to it. The pages then
return to the pool, marked for the cache pass before their next zero
fill. A view the kernel cannot track (the symbol is weak; kernels
without the patch) retires its grant as before, and gpu-pool=2 still
recycles those too. native_adreno_pool_views counts both kinds, the
reports and the grants given back.
Not run on hardware.
The submission's audit phase compared CPU cache lines with DRAM for the
write-combined IBs of the first 16 submissions and for the synchronized
driver buffers of the first 8 and every 1024th. They only detect a CPU
cache that holds a line the GPU should see; they found none in any run
(dirty=0), pool BOs have no cacheable alias for them to compare, and the
write-combine re-assertion, the correction, still runs every time. In
run68 the 8 buffer audits were 11.9 of the phase's 12.1 ms (1.5 ms each
under the driver's lock), and every 1024th submission paid one more.
gpu-audit=1 runs them as before.
Every retired job is now timed, with or without diag=1: by the CP's
stamps when gpu-timeline is on, else by the driver's clock (from the later
of its submission and the previous retirement on its ring). The 8 longest
since boot and the newest 16 over 8 ms are kept with pid, queue, ring,
priority, IB count and bytes (native_adreno_long_jobs, _slow_jobs,
_job_time).
A stall's first stage (write pointer again, translations dropped) now
records the CP's IB progress at the fence's first timed-out wait, at the
stage and 500 us later, the job, and the time from the stage to the job's
retirement (native_adreno_stall_kick). The CP snapshot is taken before the
stage acts, so its registers show the stall rather than the stage's
result.
Not run on hardware.
A BoAlloc prepares its pages before it takes the driver's lock, and the
pool's allocation runs there with its zero fill, which maps the run in
the kernel map. When the preparation gave nothing, allocateSystem asked
the pool a second time under the lock, so a VM mapping, a zero fill and
an unmap ran while every other client's submit, map and wait queued
behind. The lock path now always offers its (possibly empty) prepared
pages, and allocateSystem leaves the pool alone when it was offered any:
the VM's pages take the request there, as they did before the pool.
Not run on hardware.
Registering the pool mapped every range whole, write-combined, into the
kernel map before the GPU's own windows were mapped, and on sheng the
GPU then did not power up (GMU start stalled, no SMMU window). The
kernel keeps no mapping of a range now: the value the allocator carries
for a range is its physical address, and each scrub maps just the run
it fills for the call.
Host tests pass; not yet run on hardware.
The board registers free DRAM the kernel does not manage with native_adreno_pool_add. A write-combined system BO first asks the pool (NativeAdrenoPool.cpp: a bitmap a range, best single run, else the longest runs first, zero fill through the range's own write-combined kernel mapping before a grant is handed out); whatever the pool does not serve takes the descriptor path unchanged. A pool BO has a grant and no descriptor: its spans are the grant's runs, nothing of it is cleaned, audited or read through the cacheable aperture, its client view maps its physical runs write-combined and holds the grant until IOKit frees the view, and an import of pool pages is refused. Pages a client had a view of are retired unless gpu-pool=2; gpu-pool=0 turns the pool off. Counters: native_adreno_pool_state, native_adreno_pool_fallbacks, native_adreno_pool_zero.
Host tests: test_pool (allocator against a page model) and test_pool_memory (BO paths); the existing 23 pass as before, test_client and test_gmuconfig fail as at the base. Every changed kernel source passes the kernel's compile recipe as a syntax check. Not built into an image and not run on hardware.
RunningBoard sets the Deny role on most macOS processes that render
nothing visible (345 of 484 role changes in one session). Only the
background and utility roles took a queue's background value, so a
denied task's submissions ran at foreground priority beside the render
server's.
Not run on hardware.
Every process table maps the GPU context's eleven kernel buffers (8548 KiB,
two preemption records of 4192 KiB among them), and each submission cleaned
and invalidated all of them: 11 passes and 8548 KiB a submission in macOS
runs 57 to 61, about 351 of a submission's 579 us in run61.
Their users keep the GPU's view current themselves (a non-cacheable view, or
a clean after a store through the kernel's mapping), so the buffers are
declared synchronized and a map, an unmap and a submission leave them alone.
The probe's store through the kernel's mapping is cleaned where it is made.
The first 8 submissions and every 1024th audit the head of each buffer
against DRAM; a dirty line restores the maintenance for good.
A submission, a completion and the write-combine re-assertion visit only the
mappings they have work for instead of every mapping of the client. The
submission log reads the IB words only with diag=1 and no longer through one
kernel mapping per word.
gpu-maint-all=1 restores the passes over everything for a comparison.
Counters: native_adreno_maint_kind, native_adreno_maint_walk,
native_adreno_sync_audit, native_adreno_sync_skips, native_adreno_submit_us.
Host tests cover it (test_maintenance, test_gpuctx, test_submit). Not run on
hardware.
The mailbox poller woke every millisecond from the render server's first
connection on, although a request can only appear once the owner's task
maps the page. It now sleeps until clientMemoryForType hands the page to
the owner and goes back to sleep when the owner changes or closes. At
rest the thread took 86 to 130 ms of CPU per 10 s on sheng.
Not run on hardware.
A waiter slept on the board's pause (the next GPU interrupt of any kind, at
most 1 ms) and then polled: 34767 of 42325 pauses of one boot ended by the
timeout. Three defects came with it:
- The poll before the sleep used the display's completed prefix, which stays
below a finished fence of the high ring while an older fence of the low
ring is pending. Such a waiter slept until its deadline (100 ms slices)
with its own fence written.
- Nothing looked at the switch between the rings when a fence completed, so
a low-ring job queued behind the high ring got the CP only at the next
submission, zero-timeout poll or expired deadline.
- A finished switch was dropped when its handler found the lock busy, and
the incoming ring's write pointer waited for the next locked call.
The driver exports native_adreno_gpu_interrupt(status) for the board's GPU
interrupt handler to call with the status word it read and cleared. A fence
or a finished switch asks for the completion pass on a thread call:
it retires finished jobs (which runs the ring switch decision and completes a
finished switch), then wakes exactly the waiters whose fence the CP has
written on its own ring (NativeAdrenoSubmit::fencePending). The hang check
runs one pass per period for an interrupt that never came. A waiter arms
itself before it looks at the fence and sleeps on its own record until the
pass disarms it or the caller's deadline passes, so no completion is lost
and no timed poll is left. Until the board delivers interrupts, or with
gpu-fence-wake=0, the waiter keeps the board's pause, with the per-ring
check. Counters: native_adreno_fence_wake. Host tests cover fencePending on
one and two rings. Not run on hardware.
The stall snapshot reads the registers of the ring the CP runs and names the
fence after the waited ring's last written one, which with two rings can
belong to the other ring or be retired already. native_adreno_stall_ring
keeps, with each snapshot, the waited fence and its ring, the ring the CP
runs, a switch in flight, the unfinished jobs of each ring, the named
fence's job and how long the waited job has been submitted, so a job that
waits for the ring switch can be told from a CP that stopped. Not run on
hardware.
A wait with a timeout runs with the driver's lock held for all of it (a
connection's teardown and the table hand-back wait 2 s, a submission's
wait_fence, the board's calls). It decided the switch between the two rings
once, at its start, and then only read the fence word between pauses, while
the lock kept every other caller of the switch out. A low-ring job behind a
busy high ring therefore stayed without the CP after the high ring went idle,
for the whole timeout; a switch the CP finished meanwhile was not completed
either. The hang check then found a low-ring job older than 2 s with the
high ring idle, took the ring for unreachable, gave it up and restarted the
GPU (one boot log shows a 2 s wait followed by "hangs 1 restarts 1", "dead 3
rescues 1" and a 2.17 s wait of another caller).
wait() and observeFence() go through waitOnRing(), which with two rings
runs driveSwitch again after every pause. A host test plays the CP from the
pause: without the change the wait times out, with it the switch is
triggered, completed and the low-ring job finishes. Not run on hardware.
native_adreno_import_memory holds the imports alive, their KiB, the peak and
the imports made; native_adreno_import_owners holds the importing processes
(pid, live KiB, live imports, peak KiB). An import pins its pages from
importTask until release, so these are the bytes the driver keeps wired
besides its own BOs.
native_adreno_view_owners holds the BO views each process maps. A view is an
IOKit mapping, which the VM bills to the mapping task by its whole size
(iokit_mapped, part of the footprint the memory limits judge).
The holder table of native_adreno_bo_owners is shared by the three tables
(NativeAdrenoNoteHolder). Host tests cover the holder rows and the import
accounting. Not run on hardware.
native_adreno_bo_owners keeps the 24 processes that hold the most system BO memory (pid, live and peak KiB, live BOs) and native_adreno_bo_sizes the live BOs in six size classes, for a board log. Built only; not yet run on hardware.
Every boot argument loses its sheng- prefix (sheng-touch=4 is now
touch=4). No XNU or Apple kext boot argument of iPadOS 27 or macOS 27.3
uses any of the new names.
Accept Turnip passes (36), transfers between tile memory and memory
(37) and pass attachments (44) through the trace method and keep them in
the ring. Turnip sends them only from the render server with
SHENG_GPU_TRACE.
Event 65 carries a refusal text of a Metal client. iconservicesagent
crashed eight times on a nil Core Image program without a record of what
the frontend refused.
The kernel ring keeps submission, retirement, display arrival, fence and
latch, build identity and the render server's render, encode and completion
thread identities. Pixel capture, dispatch hooks and the switch, recovery,
fence-query and import events are removed; the ring holds 65536 records.
The AdrenoMetal repository was NativeMetal; the accelerator's include of
../NativeMetal/OriginalAcceleratorABI.h resolved only beside a leftover clone
of the old name and fails on a clean tree.
Every branch listed below was compared with main: its changes are already
on main (rebased or reimplemented) or are superseded by it, so main's tree
is kept unchanged and the branch tips become part of main's history.
- origin/feat/gpu-aperture-4g
- origin/feat/ipados27-gpu-parity
- origin/feat/stall-snapshot
- origin/perf/gpu-smoothness
- origin/wip/submit-128-ibs
The newest 32 setPriority calls keep the pid, both values, the task's GPU
role, the time and how many times the connection had told the driver
before, so a priority that the frontend forwards can be told apart from
the runtime's own call at device open.
Submissions carry the task's GPU role to the timeline.
A GPU client can leave pass and pipeline records in the kernel's log
through method 18, for what only the process knows and has no file of its
own to put anywhere (a sandboxed extension). Rows go to two rings of 512,
allocated on the first note because the kernel's data segment has no room
for them.
The newest eight GPU restarts keep their fence, status, replayed count,
stranded and guilty flags, the reason the second ring was given up and
each ring's newest fence.
checkCpSwitch records how far it got, native_adreno_preempt_stats exports
the switch state and both records, and native_adreno_now_us gives the
submission clock to the loggers outside the submit path.
A switch that does not complete, a trigger whose register writes were
refused, or a low-ring job left waiting with the high ring idle ends the
second ring for good. recover() then starts from the oldest job that never
ran and replays it on the first ring, whichever ring it sat on, so a
broken switch costs the background work and not the session.
Ring 0 is the high ring: ring 1 is only switched to when ring 0 has no
pending job, or the low job has waited PreemptAgingUs. Only priorities 2
and 3 take ring 1.
A ring the CP does not run keeps its write pointer for the switch to
give, and every framed submission on a ring with a preemption record ends
with the yield.
The preemption record keeps what the CP saved into it; only the write
pointer is rewritten before a switch.
The switch log records the registers and both records around each switch,
and a switch that times out carries the CP's IB state.
Add hardware CP context switch via CP_CONTEXT_SWITCH_CNTL on a second
ring. Use a conservative BO-disjoint rule: only allow a queued job to
preempt the running ring when their buffer sets share no handle. Two
4192 KiB preemption records and two a7xx_cp_smmu_info pages are mapped
at fixed IOVAs. Ring priority comes from the kernel GPU role of the
submitting task (BACKGROUND/UTILITY → ring 1, UI/focal/realtime → ring 0).
Diagnostics gated on sheng-diag=1; feature gated on sheng-gpu-preempt=1.
Each timeline entry records the submitting process and when the driver saw the job retired, so a frame's wait for the ring and for completion can be split.