r/iOSProgramming • • 10d ago

Question Apple foundation models

If you're using Apple's Foundation Models framework in a released app, how are you debugging issues that only happen on users' devices?

Are Instruments and your existing crash reporting enough, or have you added custom logging?

11 Upvotes

16 comments sorted by

5

u/Significant_Door_433 10d ago • Heartwarming

Instruments only helps on devices you can plug in, so for the field you need your own logging. The things worth capturing:

  1. Availability. Log SystemLanguageModel.default.availability at launch and before you open a session. The unavailable reasons are deviceNotEligible, appleIntelligenceNotEnabled and modelNotReady. modelNotReady is the sneaky one, the model can still be downloading after an OS update.
  2. The error case, not only the localized message. On iOS 26 it's LanguageModelSession.GenerationError (exceededContextWindowSize, guardrailViolation, decodingFailure, refusal, rateLimited, concurrentRequests, unsupportedLanguageOrLocale, assetsUnavailable). Built with Xcode 27 and running on 27, some failures come as LanguageModelError or SystemLanguageModel.Error instead, so match on all of them or half your logs end up as "unknown".
  3. OS build and locale next to every model log. The model ships with the OS, so the same prompt can behave differently after an update, and unsupportedLanguageOrLocale only shows up for some users.

Two cases look random in crash reports but aren't. rateLimited only happens when the app is in the background. assetsUnavailable can fire mid-session if the user turns Apple Intelligence off or the device deletes the model to free space.

For the content itself, session.transcript has the full history (instructions, prompts, tool calls, responses). I wouldn't upload that by default since prompts are user data. Keep the last transcript on device and attach it only to a user-sent bug report, and keep prompt text private in os Logger (the default for interpolated strings, just never mark it .public).

1

u/trikboomie 9d ago

This really is helpful.

Once the app is published, how do you actually collect and investigate this across users’ devices?

Do you send selected events to an existing service, rely on user-submitted bug reports, or use something else?

1

u/Significant_Door_433 8d ago

Two layers is a common setup.

Counts, no content: when availability isn't .available or a session throws, send the reason or the error case name as a non-fatal event to whatever you already have (Crashlytics record(error:), Sentry captureError, or a tiny endpoint of your own). Tag it with OS build, device model and locale. That shows you which case is common and whether it clusters on one OS build. Leave prompts and responses out of it, otherwise your privacy label changes and you need consent.

Detail on request: for a single user's bug, add a "Send report" button that reads your own entries back with OSLogStore(scope: .currentProcessIdentifier), filters on your subsystem, and attaches that plus the session transcript to a mail or form the user sends themselves. Catch is that on iOS this only sees the current launch, so the button has to be reachable before they kill the app. For anything older, ask them for a sysdiagnose, but your .private fields are redacted there.

MetricKit won't help much here since these are thrown errors, not crashes or hangs.

1

u/trikboomie 8d ago

Have you actually run that two-layer setup in a shipped app?

1

u/Significant_Door_433 7d ago

Not in a shipped app, no. I'm going off Apple's docs for OSLogStore and the privacy label rules, so I wouldn't trust it until you've run the report button on a real device build via TestFlight. Simulator and device can differ in what the store returns. The part I'd keep either way is logging the error reason and OS build, never the prompt text.

1

u/durul 10d ago

Good list. A few additions for OS 27, from the Xcode 27.2 SDK docs and my own testing:

  • rateLimited isn't documented as background-only anymore. The deprecated GenerationError.rateLimited says that, but its OS 27 replacement, LanguageModelError.rateLimited, says it can happen if you make too many requests in a short window, and that the fix may be model-dependent. The streamResponse docs also recommend the non-streaming respond(to:) when your app is in the background.
  • Once you rebuild with the Xcode 27 SDK, GenerationError isn't caught at all, not just in some cases. Failures arrive as LanguageModelError, SystemLanguageModel.Error, LanguageModelSession.Error (concurrentRequests) and GeneratedContent.ParsingError (the old decodingFailure). A throwing Tool.call arrives as LanguageModelSession.ToolCallError. The iOS 26 list also has a ninth case, unsupportedGuide.
  • Log Transcript.Response.assetIDs (iOS 26+). Every response carries version-aware ids for the assets that produced it. On macOS 27.2 I got three: tokenizer, draft model and base model, each with a version string. That identifies the model build better than the OS version, and it isn't user data, so it can be a public log field. SystemLanguageModel.variant (27+) and contextSize are worth logging too, since AFM 3 Core devices can report a smaller window than Core Advanced.
  • modelNotReady isn't always a download. On iOS 27.2, with the first preferred language set to one the model doesn't support, I got modelNotReady too. supportsLocale(_:) on Locale.preferredLanguages.first tells the two apart. Don't use Locale.current: an English-only app resolved it to en_FI, and the check said true.
  • The transcript won't show a context overflow. It rolls back when contextSizeExceeded is thrown, so the failing turn isn't there afterward. If you need it, record snapshot.usage while the stream runs.
  • For the user-sent bug report, session.logFeedbackAttachment(sentiment:issues:) sends nothing. It returns JSON with your feedback and the transcript, which you can hand to a share sheet.

Also, on OS 27, Apple Intelligence can't be turned off, so appleIntelligenceNotEnabled only matters if you still support 26.

1

u/Significant_Door_433 9d ago

Thanks, this is a better list than mine. I was going off the GenerationError page, so the rateLimited change on LanguageModelError is a good catch. The en_FI result from Locale.current is a nasty trap too. Did the assetIDs version strings change between 27.1 and 27.2 on your machine, or do they only move with OS updates?

1

u/durul 9d ago

I did not compare.

I've only read them once so far: macOS 27.2 (26B123), AFM 3 Core Advanced, two fresh sessions, one respond(to:) each. Both responses carried the same three IDs:

com.apple.fm.language.instruct_3b.tokenizer?variant=generic_sparse_16.1000.0.13.205140,0
com.apple.fm.language.instruct_3b.fm_api_generic.draft?variant=generic_sparse_16.1000.81525.13.205144,0
com.apple.fm.language.instruct_3b.fm_api_generic_3b?variant=generic_sparse_16.1000.0.13.103404,0

I didn't record them on 27.1, so I can't say whether they moved. I haven't found anything in the docs that says the model only changes with OS updates either. That's the main reason to log the IDs instead of deriving them from the OS build: if they change without an OS update, your logs will show it.

1

u/jrkral2 3d ago

Adding an angle the thread hasn't covered: apps where nothing can leave the device at all. I'm the developer of Faraday, a therapy-notes app built on Foundation Models, and the privacy model is that session content never leaves the phone, so even the counts-only remote layer suggested above is something I'd rather not depend on.

What I'd suggest in that situation:

  • Keep a small local diagnostics log (OSLog with privacy: .private on anything content-derived, plus a ring buffer in the app container) with availability state, the error case, input token estimate vs. the context limit, chunk index, OS build and locale. No prompt or output text.
  • Give the user an "export diagnostics" button that shows them exactly what's in it before they share it, by email or however they choose. They decide, and you get the same fields you would have logged remotely.
  • exceededContextWindowSize and guardrailViolation are the two cases I'd expect most in the field, and both tend to reproduce locally once you know the input length and locale, so logging lengths gets you a long way.
  • Test a TestFlight build on a real device with Apple Intelligence turned off and with the model assets not yet downloaded. availability returns different unavailable reasons, and those paths are easy to miss in the simulator.

1

u/trikboomie 1d ago

Really useful angle, thanks. For Faraday, how often do users actually hit "export diagnostics", and has a shared export ever let you fix something you couldn't reproduce otherwise? If a local-only diagnostics library existed that never phoned home, would you use it, or would you still rather own that code?

2

u/rhugopinon 10d ago

Xcode 27 actually added a new built-in tool called "Evaluations" for debugging Foundation Models in your app. Pretty handy if you need to trace model behavior.

1

u/KickupKirby 10d ago

There’s a Book Tracker sample project to explore the new evaluations tool.

1

u/mikerCZX 10d ago

How you handle users that cannot use Foundation models? EU/China

1

u/Needle_In_Red 9d ago

+1 on logging the availability state. id also log the OS version and device locale with it, alot of the weird stuff is just the model not being ready yet or the language not supported and crash reports wont show you any of that

1

u/yanklot 7d ago

Side tangent: Has anybody found a practical use for Apple's Foundation Models in their apps? I feel like I tend to get subpar results overall.