feat: per-interface data_term override and sparse profile-term identity (streaming P4)

[Jammy2211]

Summary

Streaming visibilities phase 4 (epic streaming-visibilities, Discussion https://github.com/orgs/PyAutoLabs/discussions/13; issue #598). Lets interferometer fits that contain ordinary (non-linear) light profiles run on an array-free dataset (Interferometer.from_stream / from_sparse_terms: no data, noise_map, uv_wavelengths or transformer) by the identity

Σ|d − F i_p|²/σ² = data_term − 2 i_pᵀ d̃ + i_pᵀ W̃ i_p

so the profile visibilities F i_p are never formed. This PR is the PyAutoArray half: the inversion's DatasetInterface gains a per-fit data_term override that fast_chi_squared prefers over the operator's cached (unsubtracted) scalar when data is None; a shared helper computes the subtracted dirty image and data term from one W̃ i_p product; and FitInterferometer gains an overridable sparse_chi_squared hook so light-profile-only fits (no inversion) get chi_squared / log_likelihood / figure_of_merit array-free. Companion PRs: PyAutoGalaxy and PyAutoLens (same branch name).

Why the override matters: without it, removing the array-free raise in the downstream fits makes fast_chi_squared silently fall back to the operator's unsubtracted data_term; the red-check on the companion branches gave a log-evidence wrong by 2.4–3.6 % with no error. With the override the array-free fit matches the in-memory dense fit at rel 1.8e-15.

API Changes

Additive only. aa.DatasetInterface accepts an optional data_term; aa.util.inversion_interferometer.sparse_profile_terms_from is new; aa.FitInterferometer.sparse_chi_squared is a new hook (None by default) and chi_squared consults it on an array-free dataset before raising. In-memory and pixelization-only sparse behaviour is unchanged.
See full details below.

Review

Codex gpt-6-astra (3 independent runs): no defect in this repo's changes (identity sign/factor, override precedence, jit tracing all checked). Its four findings land in the companion PyAutoGalaxy/PyAutoLens PRs (three fixed there, one pre-existing filed in Mind).

Test Plan

• New: identity vs dense Σ|d − F i|²/σ² on interferometer_7 (rel 1e-10); data_term override used when given / operator scalar when absent / ignored when data is given; numpy == jax.jit; fit hook on an array-free fit vs the in-memory fit; maps still raise DatasetException matching "array-free".
• test_autoarray: 1903 passed (main: 1898).
• Companion branches: PyAutoGalaxy 1305 passed, PyAutoLens 793 passed + 1 xfailed.
• CI green on all legs.

Full API Changes (for automation & release notes)

Added

• aa.DatasetInterface(..., data_term=None) — χ² data term of this interface's (possibly profile-subtracted) visibilities; read by the sparse interferometer inversion's fast_chi_squared when data is None, in preference to sparse_operator.data_term.
• aa.util.inversion_interferometer.sparse_profile_terms_from(sparse_operator, image, extent_index_for_masked_pixel, xp=np) → (operated_image, sparse_dirty_image, data_term) = (W̃ i_p, d̃ − W̃ i_p, data_term − 2 i_pᵀd̃ + i_pᵀW̃i_p); data_term is None if the operator has none. jit-safe.
• aa.FitInterferometer.sparse_chi_squared — property hook, None by default; subclasses return the sparse-path χ².

Changed Behaviour

• aa.FitInterferometer.chi_squared — on an array-free dataset (data is None) returns sparse_chi_squared when it is not None; otherwise raises DatasetException as before. Residual / normalized-residual / χ² maps still raise.
• AbstractInversionInterferometer.fast_chi_squared — with dataset.data is None, term 3 is dataset.data_term if set, else sparse_operator.data_term, else InversionException (unchanged fallback order otherwise).

Heart RED development override (human-authorized, 2026-10-01)

Shipped under the AUTONOMY.md "Human override for Heart RED (development only)". Live human authorization in the CLI session (Fable 5.1), quoted: "Yes, ship (push + PR-open)" for task streaming-p4-light-profile-identity (#598). Exact RED reasons at ship time (pyauto-heart readiness):

• release validation FAILED (stage integrate)
• workspace validation not passing (0 failed, 1 timeout, cloud#36404726969: autolens_test scripts/multi_dataset/rectangular.py)
• manifest drift: public front-door organ tables (generated) — 1 mismatch(es) vs PyAutoMind/repos.yaml

None relate to this branch. Branch gates passed: test_autoarray 1903, test_autogalaxy 1305, test_autolens 793 + 1 xfailed; Codex review 4 findings (3 fixed in-branch, 1 pre-existing filed). Scope: push + PR-open only. Merge requires a separate human /prm with every required check green; this is not a release and does not claim Heart is healthy.

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code