391 Commits

  • build: use automatic library type (#266)
    ## What does this PR do?
    
    Bumps xtool-core et al, switching products to automatic instead of
    forcing dynamic libraries. LGPL dependencies (LIMD, XADI) are still
    packaged as dynamic.
    
    ## How was it tested?
    
    - [x] Ran locally on macOS + Linux and confirmed `xtool dev` works
    - [x] Inspected .app bundle to make sure we have the expected dynamic
    frameworks
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [x] **0** - No AI was used to write code
    - [ ] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
    
    
    <!-- This is an auto-generated comment: release notes by coderabbit.ai
    -->
    
    ## Summary by CodeRabbit
    
    * **Chores**
    * Updated underlying package dependencies to newer minimum versions for
    improved compatibility and maintenance.
    
    <!-- end of auto-generated comment: release notes by coderabbit.ai -->
  • Bump xadi to 0.4.1 (#265)
    ## What does this PR do?
    
    Fixes a bug where installation would crash due to `plist_from_bin`
    hitting Provision's version of the function which uses D runtime
    shenanigans.
    
    ## How was it tested?
    
    Locally on macOS
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [x] **0** - No AI was used to write code
    - [ ] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
    
    
    <!-- This is an auto-generated comment: release notes by coderabbit.ai
    -->
    
    ## Summary by CodeRabbit
    
    * **Chores**
      * Updated the `xadi` package dependency to version 0.4.1.
    
    <!-- end of auto-generated comment: release notes by coderabbit.ai -->
  • sdk: integrate OpenAppleMacros (#101)
    ## What does this PR do?
    
    Adds support for proprietary Apple macros (like SwiftUI's `@Entry` and
    `@State`) via
    [OpenAppleMacros](https://github.com/xtool-org/OpenAppleMacros).
    
    We have to re-implement these because the macros need to run on the
    _host_, and the Apple SDKs include them as macOS-only dylibs.
    
    ## How was it tested?
    
    - [x] Extensive integration tests in OpenAppleMacros that compare
    Apple's expansions against ours
    - [x] Confirmed that `@Entry`, `#Preview`, and `@State` work in Swift
    6.3 and 6.4 (State is a PW in the former, macro in the latter)
    - [x] Confirmed that OSS toolchain macros continue to work (tested
    `@Observable`, `#bundle`)
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [x] **0** - No AI was used to write code
    - [ ] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • tests: add integration tests (#252)
    ## What does this PR do?
    
    Adds integration tests
    
    ## How was it tested?
    
    CI
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [x] **0** - No AI was used to write code
    - [ ] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
    
    
    <!-- This is an auto-generated comment: release notes by coderabbit.ai
    -->
    
    ## Summary by CodeRabbit
    
    * **Tests**
    * Added integration smoke tests covering version checks, app creation,
    iOS build metadata, and slim SDK update behavior.
    * Added validation for locating Xcode and verifying the Swift toolchain.
    
    * **Chores**
    * Added automated integration-test execution for pushes and pull
    requests.
    * Added Docker and Compose support for running integration tests in a
    consistent environment.
    
    <!-- end of auto-generated comment: release notes by coderabbit.ai -->
  • sdk: self-updating darwin tools (#110)
    ## What does this PR do?
    
    Allows xtool to auto-update the Darwin SDK format when there's a new
    Darwin toolset, or when we change the SDK format. Downstream this also
    allows us to auto-update when there's changes to OpenAppleMacros (#101).
    
    One callout: the disk footprint of `darwin.artifactbundle` is now ~7 GB
    instead of ~3 GB.
    
    Reason: `xtool sdk install /path/to/Xcode.[app|xip]` now performs a
    "normal" install by default, which preserves the entirety of
    `Xcode.app/Contents/Developer` — even the bits we don't currently use,
    like `AppleTVOS.platform`. This makes it so if we start using more
    components in the future, the user doesn't need to download Xcode again
    and rebuild the SDK themselves.
    
    You can still build an old-style SDK (containing just the components we
    need) using `xtool sdk install --slim` or the dedicated `xtool sdk
    build`.
    
    ## How was it tested?
    
    Tested all of the install modes. Confirmed that auto-updating works.
    Added integration tests in #252.
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [ ] **0** - No AI was used to write code
    - [x] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
    
    
    <!-- This is an auto-generated comment: release notes by coderabbit.ai
    -->
    ## Summary by CodeRabbit
    
    - **New Features**
    - Added SDK flavor reporting for slim, normal, and legacy installations.
    - Added `sdk update` support to rebuild and install SDKs from an
    installed Xcode bundle.
      - Added a `--slim` option for SDK installation.
      - SDK status now displays installation path, flavor, and version.
    - SDK setup and build operations now ensure a suitable SDK is available
    automatically.
    
    - **Bug Fixes**
    - Improved temporary SDK bundle handling, validation, replacement, and
    cross-filesystem copying.
      - Copy failures now provide detailed diagnostic information.
    <!-- end of auto-generated comment: release notes by coderabbit.ai -->
  • anisette: use XADI Swift Package (#258)
    ## What does this PR do?
    
    The xadi repo (https://github.com/xtool-org/xadi) now vends an
    artifactbundle for xadi, so we don't need to separately build/install it
    during the xtool build. This also makes it super easy to support macOS
    later (#259).
    
    ## How was it tested?
    
    - [x] Confirmed I can sign in (including provisioning)
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [ ] **0** - No AI was used to write code
    - [x] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • sdk: move DarwinSDK into PackLib (#256)
    ## What does this PR do?
    
    Mechanical move of DarwinSDK from XToolSupport -> PackLib. Needed for
    https://github.com/xtool-org/xtool/pull/229.
    
    ## How was it tested?
    
    Compiled + CI
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [x] **0** - No AI was used to write code
    - [ ] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • gsa: fix auth again (#257)
    ## What does this PR do?
    
    Fixes HTTP 503 on GsService… again (followup from #244)
    
    Seems like GSA has stopped accepting `com.apple.dt.Xcode` in the client
    info string. Use `akd` instead.
    
    ## How was it tested?
    
    - [x] Logging in before failed
    - [x] Logging in after succeeds
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [x] **0** - No AI was used to write code
    - [ ] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • sdk: fix permission issue when copying clang resources (#255)
    ## What does this PR do?
    
    Fixes #254 by using `cp -R` instead of `FileManager.copyItem`. The
    latter tries to preserve the original owner of files instead of using
    the caller, which fails if you're non-root and the original files are
    root-owned.
    
    ## How was it tested?
    
    I was able to repro the original issue by installing an SDK as a
    non-root user in an xtool Docker container.
    
    Applied this patch, tried `xtool sdk install` again, and it worked.
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [ ] **0** - No AI was used to write code
    - [x] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • docs: update contributor guide (#250)
    ## What does this PR do?
    
    Adds more info for contributors, plus AI policy.
    
    ## How was it tested?
    
    Read the docs on GitHub
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [x] **0** - No AI was used to write code
    - [ ] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • http: add some debugging knobs (#249)
    ## What does this PR do?
    
    Improves support for HTTP MITM
    
    ## How was it tested?
    
    Did some MITMing on macOS + Linux via Docker, using Proxyman and
    mitmproxy. Didn't test `XTL_HTTP_PROXY` but Sol pinkie promises me that
    it's correct.
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [ ] **0** - No AI was used to write code
    - [x] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • gsa: disregard keepalive (#247)
    GSA seems to, sometimes, reject HTTP requests made on a kept-alive
    connection with a 503. This was either the real root cause, or a second
    root cause, for #243. Fix this by creating a new HTTP connection for
    each GSA request.
    
    See also: https://github.com/rileytestut/AltSign/pull/52 and
    https://github.com/rileytestut/AltSign/pull/47
    
    Speculatively fixes #232
  • http: throw on unsuccessful responses (#246)
    Ref #243. `makeRequest` never inspected `response.status`, so a non-2xx
    body was
    returned as if it had succeeded. Every caller decodes that body as a
    property list
    or JSON, so Apple answering with an HTML error page surfaced as:
    
    ```
    Error: DecodingError.dataCorrupted: Data was corrupted. Debug description: The given
    data was not a valid property list.. Underlying error: ... Encountered unknown tag
    html on line 1
    ```
    
    which says nothing about what actually happened. In the case that
    prompted #243, GSA
    was returning `503 Service Temporarily Unavailable` for the `apptokens`
    operation. It
    now reads:
    
    ```
    Error: POST https://gsa.apple.com/grandslam/GsService2 failed: HTTP 503 Service
    Unavailable (text/html): <html> <head><title>503 Service Temporarily
    Unavailable</title></head> ...
    ```
    
    ### Notes
    
    - The response body is only included in the message for `text/html` and
    `text/plain`,
    i.e. gateway error pages. Bodies from the auth endpoints can carry
    account details
    (the 2FA requests use `Accept: application/x-buddyml`), and errors get
    printed
    verbatim in places such as `DevCommand.swift:164`, so those are left
    out. It also
      avoids emitting replacement characters for binary bodies.
    - The non-2xx path does not report progress, so `XADIProvider`'s
    download no longer
      prints `100%` immediately before failing.
    - Body collection moved to `collect(_:onProgress:)` so there is a single
    exit point.
      Buffer pre-sizing and progress reporting are unchanged.
    - `SDKBuilder` already does this check by hand on `send()`, so it is
    unaffected.
    
    ### Behaviour changes worth a look
    
    - `XADIProvider` writes the Apple Music APK download straight to disk; a
    failed
      download now throws instead of leaving a corrupt `applemusic.apk`.
    - The 2FA requests (`GrandSlamTrustedDeviceRequest`,
    `GrandSlamSMSAuthRequest`,
    `GrandSlamValidateRequest`) use no-op decoders, so a rejected request is
    currently
    discarded and xtool goes on to prompt for a code that was never sent —
    the symptom
      in #235. Those now fail loudly.
    
    ### Two things I left alone
    
    - `GrandSlamTwoFactorAuthenticateOperation.swift:47` catches
    `GrandSlamOperationError`
    with code `-21669` to report an incorrect verification code, but that
    error is only
    thrown by `GrandSlamOperationDecoder`, which the validate requests do
    not use, so the
    catch is unreachable. With this change a wrong code surfaces as a raw
    HTTP error
    instead. Mapping the status back to `incorrectVerificationCode` seemed
    like your call,
      since I do not know what GSA returns for a bad code.
    - If any endpoint reports errors in a decodable body under a non-2xx
    status, this guard
    would pre-empt it. I could not find one — `developerservices2` returns
    HTTP 200 with
      `resultCode`/`userString` in the body, which is untouched here.
    
    Built and linted on Linux (`swift build --product xtool`, `make lint`: 0
    violations).
    
    ---------
    
    Co-authored-by: Robetronica <noreply@Robetronica.com>
    Co-authored-by: Kabir Oberai <oberai.kabir@gmail.com>
  • gsa: update user agent to fix auth (#244)
    Fixes #243. GrandSlam seems to be rejecting clients that self-report as
    really old.
  • docs: enable copy-to-clipboard (#241)
    Ref https://github.com/swiftlang/swift-docc/pull/1273
    
    Could be a good idea to later add `nocopy` to some of the code blocks
    that aren't commands
  • Fix tmpdir reentrancy (#234)
    Previously, we would clear xtool's temporary directory on startup. This
    (practically) meant you couldn't have multiple instances running in
    parallel, because the second instance could delete the first instance's
    temp resources while the first instance still needed them. Fix this by
    having xtool instances declare locks on their temporary directories.
  • Fix clang headers (#228)
    A slightly different approach to #215.
    
    ---------
    
    Co-authored-by: sfusco <slf@stevenfusco.com>
  • Add build --sign option (#224)
    Running `xtool dev build --sign [--ipa]` will produce a codesigned
    app/ipa, without installing
  • Fix capability creation via public API (#220)
    Ref #216: the public API seems to now complain when passing in
    `capabilities`, but the field is still required when using the private
    API.
    
    Co-authored-by: sfusco <slf@stevenfusco.com>
  • fix(linux): use home cache dir to avoid tmpfs→ext4 EINVAL on SDK install (#211)
    ## Problem
    
    `xtool setup` fails on Linux Mint (and other distros) with:
    
    ```
    Error: Error Domain=NSCocoaErrorDomain Code=512 "(null)" UserInfo={
      NSUserStringVariant=["Copy"],
      NSSourceFilePathErrorKey=/tmp/sh.xtool/tmp-DarwinSDKBuild-.../darwin.artifactbundle/...,
      NSDestinationFilePath=/home/<user>/.swiftpm/swift-sdks/darwin.artifactbundle/...,
      NSUnderlyingError=Error Domain=NSPOSIXErrorDomain Code=22 "Invalid argument"
    }
    ```
    
    ## Root Cause
    
    On Linux Mint (and Ubuntu-based distros), `/tmp` is mounted as **tmpfs**
    while the home directory is on **ext4**. Foundation's
    `FileManager.copyItem` uses `sendfile(2)` internally, which returns
    `EINVAL (22)` when copying between different filesystem types.
    
    `TemporaryDirectoryRoot` defaults to
    `FileManager.default.temporaryDirectory` → `/tmp` on Linux. When `swift
    sdk install` copies the built SDK from `/tmp/sh.xtool/...` to
    `~/.swiftpm/swift-sdks/`, it crosses filesystem boundaries and fails.
    
    ## Fix
    
    On Linux, default the temp directory root to `$XDG_CACHE_HOME/xtool`
    (falling back to `~/.cache/xtool`) instead of `/tmp`. This keeps the
    build directory and install destination on the same filesystem, so the
    copy always succeeds.
    
    - `$XTL_TMPDIR` and `$TMPDIR` overrides continue to work as before
    - macOS behaviour is unchanged
    - Follows XDG Base Directory Specification
    
    ## Changes
    
    One file, `Sources/XUtils/TemporaryDirectory.swift` — 11 lines added
    inside a `#if os(Linux)` block.
    
    Fixes #181
  • Validate SDK before overwriting (#192)
    Running an invalid `sdk` subcommand like `xtool sdk list` (there's no
    `list` subcommand) is interpreted as "install the sdk at path `./list`".
    We previously unconditionally removed the existing SDK during
    installation. By switching the order so that we validate the path first,
    we avoid performing a potentially destructive action unless we have
    clear intent.
    
    See #186.
    
    Before:
    
    ```bash
    $ xtool sdk list
    Removing existing SDK... # destructive!
    Error: Could not read file or directory at path 'list'.
    ```
    
    After:
    
    ```bash
    $ xtool sdk list
    Error: Could not read file or directory at path 'list'.
      See 'xtool help sdk' for usage.
    ```
  • Bump zsign to 1.7.0 (#180)
    Includes the (speculative) fix for #14
  • Bump deployment to macOS 13, iOS 16 (#173)
    Necessary in order to pull in swift-subprocess
  • Clean up some entitlements code (#165)
    - We already do the `isFree` declarations in
    `DeveloperServiceCapability.swift`. Don't need to duplicate it on the
    entitlement types.
    - Add support for creating capabilities with raw names
  • Bump Swift version, dependencies (#166)
    Notably:
    
    - Build xtool with Swift 6.2
    - Resolve https://github.com/advisories/GHSA-xvr7-p2c6-j83w. That
    advisory shouldn't really affect us since we don't use NIO as a server
    but good hygiene anyway.
  • Enable cross import overlays (#153)
    Fixes #152
    
    We might want to figure out if there are any other flags we don't pass
    but should
  • Handle static xcframeworks not being unpacked (#156)
    SwiftPM may not unpack static xcframeworks. Handle this gracefully, if
    SwiftPM doesn't need the framework we don't either.
    
    Fixes #155.
  • Improve tmpdir management (#147)
    - We now have a single tmpdir "root" that can be recreated at launch to
    clean up old stragglers
    - The location of this tmpdir root can be controlled by `XTL_TMPDIR` or
    `TMPDIR`. In general the path is `$TMPDIR/sh.xtool`.
    
    With this change it should be possible to `export XTL_TMPDIR=/var/tmp`
    if `/tmp` doesn't have enough space, which fixes #23.
  • Update ASC spec to v4.0.0 (#142)
    No breaking changes afaik
    
    One notable change is the addition of
    `components.schemas.PagingInformation.paging.nextCursor` (which is
    possibly supposed to be the query parameter portion of `links.next`?
    could simplify the pagination code) but this doesn't seem to be returned
    by the prod ASC API right now.
  • Update PublicStaging path (#132)
    Previously this was shadowing the user's cwd path onto the device, e.g.
    if you were installing from `/home/foo/bar` it would create
    `/home/foo/bar/PublicStaging` on the device. For uniformity with other
    tools like `ideviceinstaller`, clean this up and just create
    `/PublicStaging`.
  • Add support for App Extensions (#97)
    This pull request adds support for Apple's [App
    Extensions](https://developer.apple.com/app-extensions/).
    
    Closes: #45
    
    ---------
    
    Co-authored-by: ErrorErrorError <16653389+ErrorErrorError@users.noreply.github.com>
    Co-authored-by: Kabir Oberai <oberai.kabir@gmail.com>
  • fix: resolve ASC invalid bearer token (#129)
    This pull request resolves potential clock drift between ASC and bearer
    token.
    
    Ref: https://github.com/xtool-org/xtool/pull/97#discussion_r2203584344