Commit Graph

187 Commits

  • sdk: fix permission issue when copying clang resources (#255)
    ## What does this PR do?
    
    Fixes #254 by using `cp -R` instead of `FileManager.copyItem`. The
    latter tries to preserve the original owner of files instead of using
    the caller, which fails if you're non-root and the original files are
    root-owned.
    
    ## How was it tested?
    
    I was able to repro the original issue by installing an SDK as a
    non-root user in an xtool Docker container.
    
    Applied this patch, tried `xtool sdk install` again, and it worked.
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [ ] **0** - No AI was used to write code
    - [x] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • http: add some debugging knobs (#249)
    ## What does this PR do?
    
    Improves support for HTTP MITM
    
    ## How was it tested?
    
    Did some MITMing on macOS + Linux via Docker, using Proxyman and
    mitmproxy. Didn't test `XTL_HTTP_PROXY` but Sol pinkie promises me that
    it's correct.
    
    ## AI tool usage
    
    How much of this PR was AI-assisted? (check one)
    
    - [ ] **0** - No AI was used to write code
    - [x] **1** - I was assisted by AI. I reviewed the finished result.
    - [ ] **2** - I set the AI going and left it to it; nobody has read the
    result - no review, or AI review only
    
    <!-- If an AI agent is filling this in: declare the level honestly, and
    open as a draft if it is #2. Do not lower the declared level to get the
    PR reviewed. -->
  • gsa: disregard keepalive (#247)
    GSA seems to, sometimes, reject HTTP requests made on a kept-alive
    connection with a 503. This was either the real root cause, or a second
    root cause, for #243. Fix this by creating a new HTTP connection for
    each GSA request.
    
    See also: https://github.com/rileytestut/AltSign/pull/52 and
    https://github.com/rileytestut/AltSign/pull/47
    
    Speculatively fixes #232
  • http: throw on unsuccessful responses (#246)
    Ref #243. `makeRequest` never inspected `response.status`, so a non-2xx
    body was
    returned as if it had succeeded. Every caller decodes that body as a
    property list
    or JSON, so Apple answering with an HTML error page surfaced as:
    
    ```
    Error: DecodingError.dataCorrupted: Data was corrupted. Debug description: The given
    data was not a valid property list.. Underlying error: ... Encountered unknown tag
    html on line 1
    ```
    
    which says nothing about what actually happened. In the case that
    prompted #243, GSA
    was returning `503 Service Temporarily Unavailable` for the `apptokens`
    operation. It
    now reads:
    
    ```
    Error: POST https://gsa.apple.com/grandslam/GsService2 failed: HTTP 503 Service
    Unavailable (text/html): <html> <head><title>503 Service Temporarily
    Unavailable</title></head> ...
    ```
    
    ### Notes
    
    - The response body is only included in the message for `text/html` and
    `text/plain`,
    i.e. gateway error pages. Bodies from the auth endpoints can carry
    account details
    (the 2FA requests use `Accept: application/x-buddyml`), and errors get
    printed
    verbatim in places such as `DevCommand.swift:164`, so those are left
    out. It also
      avoids emitting replacement characters for binary bodies.
    - The non-2xx path does not report progress, so `XADIProvider`'s
    download no longer
      prints `100%` immediately before failing.
    - Body collection moved to `collect(_:onProgress:)` so there is a single
    exit point.
      Buffer pre-sizing and progress reporting are unchanged.
    - `SDKBuilder` already does this check by hand on `send()`, so it is
    unaffected.
    
    ### Behaviour changes worth a look
    
    - `XADIProvider` writes the Apple Music APK download straight to disk; a
    failed
      download now throws instead of leaving a corrupt `applemusic.apk`.
    - The 2FA requests (`GrandSlamTrustedDeviceRequest`,
    `GrandSlamSMSAuthRequest`,
    `GrandSlamValidateRequest`) use no-op decoders, so a rejected request is
    currently
    discarded and xtool goes on to prompt for a code that was never sent —
    the symptom
      in #235. Those now fail loudly.
    
    ### Two things I left alone
    
    - `GrandSlamTwoFactorAuthenticateOperation.swift:47` catches
    `GrandSlamOperationError`
    with code `-21669` to report an incorrect verification code, but that
    error is only
    thrown by `GrandSlamOperationDecoder`, which the validate requests do
    not use, so the
    catch is unreachable. With this change a wrong code surfaces as a raw
    HTTP error
    instead. Mapping the status back to `incorrectVerificationCode` seemed
    like your call,
      since I do not know what GSA returns for a bad code.
    - If any endpoint reports errors in a decodable body under a non-2xx
    status, this guard
    would pre-empt it. I could not find one — `developerservices2` returns
    HTTP 200 with
      `resultCode`/`userString` in the body, which is untouched here.
    
    Built and linted on Linux (`swift build --product xtool`, `make lint`: 0
    violations).
    
    ---------
    
    Co-authored-by: Robetronica <noreply@Robetronica.com>
    Co-authored-by: Kabir Oberai <oberai.kabir@gmail.com>
  • gsa: update user agent to fix auth (#244)
    Fixes #243. GrandSlam seems to be rejecting clients that self-report as
    really old.
  • Fix tmpdir reentrancy (#234)
    Previously, we would clear xtool's temporary directory on startup. This
    (practically) meant you couldn't have multiple instances running in
    parallel, because the second instance could delete the first instance's
    temp resources while the first instance still needed them. Fix this by
    having xtool instances declare locks on their temporary directories.
  • Fix clang headers (#228)
    A slightly different approach to #215.
    
    ---------
    
    Co-authored-by: sfusco <slf@stevenfusco.com>
  • Add build --sign option (#224)
    Running `xtool dev build --sign [--ipa]` will produce a codesigned
    app/ipa, without installing
  • Fix capability creation via public API (#220)
    Ref #216: the public API seems to now complain when passing in
    `capabilities`, but the field is still required when using the private
    API.
    
    Co-authored-by: sfusco <slf@stevenfusco.com>
  • fix(linux): use home cache dir to avoid tmpfs→ext4 EINVAL on SDK install (#211)
    ## Problem
    
    `xtool setup` fails on Linux Mint (and other distros) with:
    
    ```
    Error: Error Domain=NSCocoaErrorDomain Code=512 "(null)" UserInfo={
      NSUserStringVariant=["Copy"],
      NSSourceFilePathErrorKey=/tmp/sh.xtool/tmp-DarwinSDKBuild-.../darwin.artifactbundle/...,
      NSDestinationFilePath=/home/<user>/.swiftpm/swift-sdks/darwin.artifactbundle/...,
      NSUnderlyingError=Error Domain=NSPOSIXErrorDomain Code=22 "Invalid argument"
    }
    ```
    
    ## Root Cause
    
    On Linux Mint (and Ubuntu-based distros), `/tmp` is mounted as **tmpfs**
    while the home directory is on **ext4**. Foundation's
    `FileManager.copyItem` uses `sendfile(2)` internally, which returns
    `EINVAL (22)` when copying between different filesystem types.
    
    `TemporaryDirectoryRoot` defaults to
    `FileManager.default.temporaryDirectory` → `/tmp` on Linux. When `swift
    sdk install` copies the built SDK from `/tmp/sh.xtool/...` to
    `~/.swiftpm/swift-sdks/`, it crosses filesystem boundaries and fails.
    
    ## Fix
    
    On Linux, default the temp directory root to `$XDG_CACHE_HOME/xtool`
    (falling back to `~/.cache/xtool`) instead of `/tmp`. This keeps the
    build directory and install destination on the same filesystem, so the
    copy always succeeds.
    
    - `$XTL_TMPDIR` and `$TMPDIR` overrides continue to work as before
    - macOS behaviour is unchanged
    - Follows XDG Base Directory Specification
    
    ## Changes
    
    One file, `Sources/XUtils/TemporaryDirectory.swift` — 11 lines added
    inside a `#if os(Linux)` block.
    
    Fixes #181
  • Validate SDK before overwriting (#192)
    Running an invalid `sdk` subcommand like `xtool sdk list` (there's no
    `list` subcommand) is interpreted as "install the sdk at path `./list`".
    We previously unconditionally removed the existing SDK during
    installation. By switching the order so that we validate the path first,
    we avoid performing a potentially destructive action unless we have
    clear intent.
    
    See #186.
    
    Before:
    
    ```bash
    $ xtool sdk list
    Removing existing SDK... # destructive!
    Error: Could not read file or directory at path 'list'.
    ```
    
    After:
    
    ```bash
    $ xtool sdk list
    Error: Could not read file or directory at path 'list'.
      See 'xtool help sdk' for usage.
    ```
  • Bump deployment to macOS 13, iOS 16 (#173)
    Necessary in order to pull in swift-subprocess
  • Clean up some entitlements code (#165)
    - We already do the `isFree` declarations in
    `DeveloperServiceCapability.swift`. Don't need to duplicate it on the
    entitlement types.
    - Add support for creating capabilities with raw names
  • Enable cross import overlays (#153)
    Fixes #152
    
    We might want to figure out if there are any other flags we don't pass
    but should
  • Handle static xcframeworks not being unpacked (#156)
    SwiftPM may not unpack static xcframeworks. Handle this gracefully, if
    SwiftPM doesn't need the framework we don't either.
    
    Fixes #155.
  • Improve tmpdir management (#147)
    - We now have a single tmpdir "root" that can be recreated at launch to
    clean up old stragglers
    - The location of this tmpdir root can be controlled by `XTL_TMPDIR` or
    `TMPDIR`. In general the path is `$TMPDIR/sh.xtool`.
    
    With this change it should be possible to `export XTL_TMPDIR=/var/tmp`
    if `/tmp` doesn't have enough space, which fixes #23.
  • Update ASC spec to v4.0.0 (#142)
    No breaking changes afaik
    
    One notable change is the addition of
    `components.schemas.PagingInformation.paging.nextCursor` (which is
    possibly supposed to be the query parameter portion of `links.next`?
    could simplify the pagination code) but this doesn't seem to be returned
    by the prod ASC API right now.
  • Update PublicStaging path (#132)
    Previously this was shadowing the user's cwd path onto the device, e.g.
    if you were installing from `/home/foo/bar` it would create
    `/home/foo/bar/PublicStaging` on the device. For uniformity with other
    tools like `ideviceinstaller`, clean this up and just create
    `/PublicStaging`.
  • Add support for App Extensions (#97)
    This pull request adds support for Apple's [App
    Extensions](https://developer.apple.com/app-extensions/).
    
    Closes: #45
    
    ---------
    
    Co-authored-by: ErrorErrorError <16653389+ErrorErrorError@users.noreply.github.com>
    Co-authored-by: Kabir Oberai <oberai.kabir@gmail.com>
  • fix: resolve ASC invalid bearer token (#129)
    This pull request resolves potential clock drift between ASC and bearer
    token.
    
    Ref: https://github.com/xtool-org/xtool/pull/97#discussion_r2203584344
  • Harden mobileprovision.c to avoid UB on OOM (#122)
    # fix: hardening `mobileprovision.c` to avoid future crashes.
    
    ## Context:
    
    I found two common weakness inside `Sources/CXKit/mobileprovision.c`,
    which seems to be an important layer of `xtool`'s signing logic.
    
    ### Common weaknesses found:
    
    - CWE-476
    - CWE-119
    
    ---------
    
    Co-authored-by: Kabir Oberai <oberai.kabir@gmail.com>
  • Support unxipping Xcode betas (#115)
    Xcode betas are stored in the xip as Xcode-beta.app. Support this by
    avoiding hardcoding the `Xcode.app` path.
  • Drop the "Logging in" stage (#99)
    This was vestigial, and besides the fact that we don't actually do any
    work in this stage, we also never update the progress here so it gets
    stuck at 0%.
  • Make ALL response enums extensible (#92)
    Another bugfix for #64. Fifth… time's the charm?
  • FetchDeviceOperation: account for pagination (#91)
    - Always try to register the device and just return gracefully if it
    fails (prevents races, pagination issues)
    - Allow `SigningContext` (and therefore `AutoSigner`) to be created
    without a target device: just skip device registration.
  • DeveloperServices: support paginated requests (#89)
    Realized that we need to paginate due to #87
  • Fix duplicate App ID registration (#90)
    We pull the list of App IDs in `upsertApp` to see if we already created
    the App ID. This broke when the user had too many App IDs, since DS
    returned a paginated response and we only looked at the first page.
    
    We could paginate the whole list (as seen in #89) but that's slow.
    Instead, just filter the query to the expected App ID. This is a prefix
    search so we might still receive multiple responses, but it's much less
    likely that the user has like 10(? I'm not sure what the default/max
    pagination limit is) apps registered with the XTL-ABC.com.foo.bar prefix
    (they could have PlugIns but usually just a small number of them.)
    
    This is a tentative fix for #87.
  • Fix changeCurrentDirectory warning (#81)
    This is `discardableResult` on macOS (perhaps because it goes through
    objc bridging?) but not on Linux so we get a warning on the latter
  • Improve OpenAPI error handling (#78)
    - Make some enums non-exhaustive (closes #77)
      - Also a tentative fix for #62 and #64
    - Improve logging on deserialization failure
    
    We now use raw JS to patch the OpenAPI spec instead of openapi-overlay.
    It's a lot more flexible this way.
  • OpenAPI spec tweaks (#76)
    - Add `BundleIdPlatform.macos` (potential fix for #64)
    - Fix deprecation warnings
    - Improve overlay organization
  • Support for logging OpenAPI responses (#75)
    Allow setting `XTL_DEV_LOG` to a regex (e.g. `export
    XTL_DEV_LOG='profiles_.*'`) to log the responses of all matching
    operations
    
    Also add `BundleIdPlatform.services` as seen
    [here](https://github.com/AvdLee/appstoreconnect-swift-sdk/blob/60bbc198c8d9f9992807bfebef86f45d6173a1f1/Makefile#L9)
  • Support building for iOS simulator (#74)
    Closes #68
    
    ## On any platform
    
    You can now do
    
    ```bash
    xtool dev build --triple arm64-apple-ios-simulator [--ipa]
    ```
    
    To build a .app/.ipa that works on the simulator.
    
    ## On macOS
    
    You can do
    
    ```bash
    xtool dev --simulator
    ```
    
    To one-shot build-and-install on the booted simulator.
  • Update Netlify footer copy (#63)
    Netlify wants specific copy in order to be eligible for their [Open
    Source plan](https://www.netlify.com/legal/open-source-policy)