feat: per-channel SparseTerms sum and phase-centre shifts in sparse_terms_from_chunks (streaming P5)

[Jammy2211]

Overview

Phase 5 (last) of the streaming-visibilities epic (Discussion https://github.com/orgs/PyAutoLabs/discussions/13). The streamed accumulator sparse_terms_from_chunks already sums per-chunk SparseTerms; this task (a) proves and exposes that MFS terms are the sum of per-channel terms (sum(list_of_terms) via __radd__, parity vs an in-memory MFS dataset at 1e-12), (b) adds chunk-by-chunk phase-centre shifts (phase_centre=(y, x) arcsec; data multiplied by exp(+2πi(u l0 + v m0)), radians; recorded as provenance so terms with different centres cannot be summed), threaded through Interferometer.from_stream, and (c) adds an array-free datacube example to autolens_workspace that runs under the smoke profile. Library first (PyAutoArray), workspace PR follows.

Plan

• Add an optional phase_centre to sparse_terms_from_chunks, applied to each chunk's visibilities as a unit phase before the dirty image is formed; thread it through Interferometer.from_stream.
• Record phase_centre as SparseTerms provenance checked in __add__; add __radd__ so sum(terms_list) works.
• Tests: per-channel sum == one accumulation == in-memory MFS operator/scalars at rel 1e-12 (numpy and jax); phase-centre accumulation == in-memory result on pre-shifted data at 1e-12; DFT point-source test pins the sign and (y, x) order.
• Workspace: new scripts/interferometer/features/datacube/modeling_array_free.py streaming each channel through from_stream into the existing FactorGraph fit, showing the MFS sum and phase_centre; listed in smoke_tests.txt; notebook regenerated.

Detailed implementation plan

Affected Repositories

• PyAutoArray (primary, library)
• autolens_workspace (workspace, after the library PR)

Branch Survey

Repository	Current Branch	Dirty?
array/PyAutoArray	main	clean
lens/autolens_workspace	main	clean

Suggested branch: feature/streaming-p5-cubes-phase-centre (both repos; worktree ~/Code/PyAutoLabs-wt/streaming-p5-cubes-phase-centre/)

Implementation Steps

A. PyAutoArray (array/PyAutoArray) — library PR

1. autoarray/inversion/inversion/interferometer/inversion_interferometer_util.py
   • SparseTerms (L1879): add provenance field phase_centre: Optional[Tuple[float, float]] = None (arcsec, autoarray
     (y, x) order like origin); __add__ (L1944) checks it with the other recorded provenance (mismatch →
     exc.InversionException), _recorded (L2009) carries it; add __radd__(self, other) returning self when
     other == 0 (so sum(terms_list) works) and NotImplemented otherwise.
   • sparse_terms_from_chunks (L2034): new kwarg phase_centre: Optional[Tuple[float, float]] = None (arcsec,
     (y, x)). After _complex_visibilities_from(data) (L2119) and the σ checks, when set:
     l0, m0 = x, y in radians (units.arcsec.to(units.rad) as transformer.py:331 does; note the tuple order), then
     data = data * np.exp(2j*np.pi*(uv[:,0]*l0 + uv[:,1]*m0)). Kernel / beam / scalars untouched. Record
     phase_centre in the chunk provenance. Docstring: what the shift means, the sign, the units, invariance of the
     scalars, and that MFS = sum of per-channel terms (__add__, sum).
2. autoarray/dataset/interferometer/dataset.py from_stream (L328): add phase_centre=None and pass it explicitly
   (the kwargs are listed by hand at L372-382). apply_sparse_operator_from_chunks (L605) needs no change (forwards
   **accumulator_kwargs) — add a docstring line. from_sparse_terms (L252): no change (phase centre is not a mask
   property).
3. Tests test_autoarray/inversion/inversion/interferometer/test_inversion_interferometer_util.py (helpers
   _streaming_inputs(n_visibilities, seed) L1129, _chunks_from L1153; template
   test__sparse_terms_from_chunks__matches_one_shot_… L1161):
   • test__sparse_terms__sum_of_channels_equals_mfs: 3 "channels" from _streaming_inputs with different seeds /
     sizes; sum([terms_c for c]) (exercises __radd__) vs sparse_terms_from_chunks over the concatenated chunks vs
     the in-memory Interferometer(concatenated).apply_sparse_operator() operator kernel, dirty_image, data_term,
     noise_normalization, sum_weights, n_vis at rel 1e-12; repeat with use_jax=True.
   • test__sparse_terms_from_chunks__phase_centre__matches_in_memory_on_shifted_data: accumulate with
     phase_centre=(m0, l0) vs apply_sparse_operator on data * exp(+2πi(u l0 + v m0)) (radians) at 1e-12, NUFFT and
     DFT; kernel / beam / scalars identical to the unshifted run.
   • test__sparse_terms_from_chunks__phase_centre__point_source_recentres (DFT, small mask): visibilities of a point
     source at (y, x) = (m0, l0) via TransformerDFT.visibilities_from; the shifted dirty image's argmax is the
     mask-centre pixel (pins the sign and the (y, x) order).
   • __add__ with differing phase_centre raises; unrecorded-left keeps right (mirror L1404).
   • test_autoarray/dataset/interferometer/test_dataset.py: from_stream(..., phase_centre=…) carries it into
     sparse_terms.phase_centre and equals the pre-shifted in-memory sparse fit's log_evidence-relevant scalars
     (mirror L684); from_sparse_terms(sum(per_channel_terms)) builds an array-free dataset whose fit matches the
     in-memory MFS sparse fit (reuse _random_interferometer).

B. autolens_workspace (lens/autolens_workspace) — workspace PR, after the library PR

1. New scripts/interferometer/features/datacube/modeling_array_free.py (sibling of modeling.py; same mask /
   dataset / FactorGraph structure):
   • For each channel_*: build chunks as a generator over the channel's (uv_wavelengths, data, noise_map) fits
     (one chunk, or sliced into two to show chunking), dataset = al.Interferometer.from_stream(chunks, real_space_mask=mask, transformer_class=al.TransformerNUFFT); then the same AnalysisInterferometer /
     AnalysisFactor / FactorGraphModel fit as modeling.py. Prose explains what array-free means, that no
     data / noise_map / uv_wavelengths are held, what the visualizer shows instead (natural dirty images, from P3),
     and that ordinary light profiles now work (P4).
   • A short section: mfs_terms = sum(dataset_c.sparse_terms for dataset_c in datasets) →
     al.Interferometer.from_sparse_terms(mfs_terms, real_space_mask=mask) with a one-channel-model MFS fit, and a
     sentence on phase_centre= for re-centring (no fit; illustrate the call on one channel).
   • Smoke: no __Env__ override (defaults: TEST_MODE 2, SMALL_DATASETS cap, DISABLE_JAX — from_stream already honours
     disable_jax()); build the terms on the same (possibly capped) mask passed to from_sparse_terms; do not
     hard-code use_jax=True. Add scripts/interferometer/features/datacube/modeling_array_free.py to
     smoke_tests.txt. Reuse simulator.py auto-simulation exactly as modeling.py does.
2. scripts/interferometer/features/datacube/README.md: one row for the new script.
3. Notebook mirror under notebooks/… via the workspace's generator (resolve the exact command in start_workspace —
   generate.py runs from the workspace root with the project key; see memory CWD+key, md≠nb).
4. Run the script under the smoke profile locally (PYAUTO_TEST_MODE=2 SMALL_DATASETS=1 DISABLE_JAX=1 …) and with
   the full profile once; smoke test via /smoke_test before /ship_workspace.

Trade-offs / decisions

• API units and order: phase_centre in arcsec, (y, x) — matches origin, centre and every autoarray tuple;
  converted to radians inside the accumulator. Documented with the sign convention.
• Shift on data, not on the kernel: W̃ and the beam are shift-invariant; shifting the data is one complex multiply
  per chunk and keeps everything else bit-identical (asserted in tests).
• No change to __add__ semantics beyond the new provenance check; __radd__ only for 0 so sum() works.
• Workspace example is additive (new script), leaving modeling.py as the in-memory reference.
• Out of scope: per-channel models beyond the existing FactorGraph; visualization of per-channel dirty beams;
  the profiling campaign (per-chunk JAX recompile) stays in its own draft.

Key Files

• autoarray/inversion/inversion/interferometer/inversion_interferometer_util.py — SparseTerms (phase_centre, __radd__), sparse_terms_from_chunks(phase_centre=)
• autoarray/dataset/interferometer/dataset.py — Interferometer.from_stream(phase_centre=)
• test_autoarray/inversion/inversion/interferometer/test_inversion_interferometer_util.py, test_autoarray/dataset/interferometer/test_dataset.py
• autolens_workspace/scripts/interferometer/features/datacube/modeling_array_free.py (new), README.md, smoke_tests.txt, notebook mirror

Verification

• pytest test_autoarray/inversion/inversion/interferometer/test_inversion_interferometer_util.py test_autoarray/dataset/interferometer/test_dataset.py, then full test_autoarray (baseline after P4: 1903).
• Witness (prompt header): sum of per-channel SparseTerms == MFS terms at rel 1e-12 (numpy and jax);
  phase_centre accumulation == in-memory result on data shifted by exp(2πi(u l0 + v m0)) at 1e-12; the
  autolens_workspace datacube example gains an array-free variant that runs under the smoke profile.
• Workspace: modeling_array_free.py runs under the smoke profile locally and in the workspace smoke CI on the
  same-named branch; notebook regenerated; full-profile run completes once.

Original Prompt

Click to expand starting prompt

Streaming phase 5: per-channel cubes and phase-centre shifts in sparse_terms_from_chunks

Type: feature
Target: PyAutoArray
Repos:

• PyAutoArray
• autolens_workspace
  Themes:
• interferometer
• sparse-operator
• memory
  Autonomy: supervised
  Priority: medium
  Status: draft
  Epic: streaming-visibilities
  Phase: 5
  Difficulty: small
  Consequence: glance
  Witness: summing per-channel SparseTerms equals the MFS terms accumulated over all channels at rel 1e-12 (numpy and jax); sparse_terms_from_chunks(..., phase_centre=(l0, m0)) applied per chunk equals the in-memory result on data shifted by exp(2πi(u l0 + v m0)) at 1e-12; the autolens_workspace datacube example gains an array-free variant that runs under the smoke profile.
  Review-minutes: 4
  Unattended: ready
  Parent: draft/feature/autoarray/interferometer_from_stream_array_free_dataset.md
  Blocked-by: none (phase 3 merged 2026-09-30)

Source: https://github.com/orgs/PyAutoLabs/discussions/13 phase 2, sliced 2026-09-30. No phase-centre code exists today; SparseTerms.__add__ exists and is tested; datacube examples live in autolens_workspace/scripts/interferometer/features/datacube/.

[Jammy2211]

Library PR Created (streaming P5)

• feat: per-channel SparseTerms sum and phase-centre shifts in sparse_terms_from_chunks (streaming P5) #601 — SparseTerms.phase_centre provenance + __radd__; sparse_terms_from_chunks(phase_centre=) / Interferometer.from_stream(phase_centre=); sum of per-channel terms == MFS (1e-12, numpy + JAX brute-force leg); DFT point-source test pins the sign and (y, x) order.
• fix: persist SparseTerms phase_centre through the dataset.fits round trip (streaming P5) PyAutoGalaxy#643 — phase_centre persisted through the dataset.fits round trip (PyAutoLens reuses the ag writer/loader).

Both labelled pending-release; merge order Array → Galaxy (separate human /prm). Next: /start_workspace → the array-free datacube example autolens_workspace/scripts/interferometer/features/datacube/modeling_array_free.py (smoke-listed), shipped under the same grant once its full-profile run and smoke test pass.

Heart RED development override (human-authorized, 2026-10-01)

Live human authorization in the CLI session (Fable 5.1), quoted verbatim: "Yes: library now, workspace when its run passes" for this task. Exact RED reasons at ship time (pyauto-heart readiness):

• release validation FAILED (stage integrate)
• workspace validation not passing (0 failed, 1 timeout, cloud#36404726969: autolens_test scripts/multi_dataset/rectangular.py)
• manifest drift: public front-door organ tables (generated) — 1 mismatch(es) vs PyAutoMind/repos.yaml

None relate to this branch. Scope: push + PR-open only (library now; workspace PR under the same grant on a passing run); merge via separate /prm on green checks; not a release.

Branch gates

• test_autoarray 1917 passed (main 1903); test_autogalaxy 1307 passed (main 1305).
• Codex gpt-6-astra review: F1 (high, introduced — FITS round trip dropped phase_centre, reloaded None summed with anything) → persisted in PyAutoGalaxy#643; F2 (high, introduced — apply_sparse_operator_from_chunks(phase_centre=) shifted the operator but not the retained data; sparse χ² 0 vs residual 2) → typed raise; F4 (medium — data_term from unshifted data while the dirty image used shifted data, Δχ² 1800 at σr≈σi) → shift applied before every data-dependent term; also the JAX test leg made real (spy) and __add__ checks transformer_class_name. F3/F5 pre-existing (NUFFT ignores mask origin; mask-compatibility gap) → draft/bug/autoarray/sparse_terms_nufft_origin_and_mask_compatibility.md.
• Workspace so far: modeling_array_free.py smoke exit 0 (6 s), modeling.py regression exit 0, navigator/catalogue/sizes/black checks OK; full-profile run in progress.

Deviations from the plan

• Unshifted accumulations record phase_centre=(0.0, 0.0) (not None) so shifted + unshifted sums raise; None = not recorded (hand-built / pre-change files).
• PyAutoGalaxy added to the task for the persistence fix (review F1).
• Workspace phase_centre demo re-centres on the dirty-image arc peak (lensed emission), not the source-plane centre.

[Jammy2211]

Workspace PR Created (streaming P5)

• docs: array-free datacube modeling example via from_stream (streaming P5) autolens_workspace#582 — scripts/interferometer/features/datacube/modeling_array_free.py (+ generated notebook, README row, smoke_tests.txt entry, navigator catalogue). Streams each channel through from_stream (2 chunks/channel) into the existing FactorGraph fit; MFS section via sum() of per-channel SparseTerms (n_vis 760 asserted; channel-0 evidence identical to the in-memory dataset at the true model); phase_centre demo re-centres a channel's dirty-image peak on the origin with its value preserved.

Gates: full-profile run exit 0 (~13 min, n_live=50); pyauto-heart smoke autolens --root <worktree>: 42/42 scripts PASS, 2/2 notebooks PASS (modeling_array_free.py 6.4 s); check_navigator / check_search_memory / check_sizes / black OK. Opened under the same human RED override grant ("Yes: library now, workspace when its run passes"); merge via /prm after PyAutoArray#601 and PyAutoGalaxy#643 (library-first), and the phase_centre / sum() sections need the released library (release-gate).

Follow-ups filed: draft/bug/autoarray/interferometer_log_evidence_large_regularization_coefficient_cholesky.md (search's max-likelihood coefficient ≈ 9.7e5 makes log_evidence raise/NaN eagerly on both dataset types); scripts/multi_dataset/plot.py leaves an untracked dataset.fits in the workspace root on every smoke run (not this branch).

[Jammy2211]

Shipped

Merged 2026-10-01 via human /prm with every CI leg green, mergeability CLEAN, library-first:

• PyAutoArray#601 → merge ffd13ba (Tests: unittest 3.12 / 3.13 / nojax)
• PyAutoGalaxy#643 → merge f15570b0 (Tests + Docs)
• autolens_workspace#582 → merge 5721f66a (Smoke 3.12 / 3.13, Script Size Guard, Navigator Check)

The two library PRs remain pending release (labelled pending-release; last tag 2026.9.27.2); the workspace example's phase_centre and sum() sections need that release (release-gate). Mind record: complete/2026/10/streaming-p5-cubes-phase-centre.md. This closes the development phases of the streaming-visibilities epic; the Discussion #13 follow-up post is the remaining item. Follow-ups filed: draft/bug/autoarray/sparse_terms_nufft_origin_and_mask_compatibility.md, draft/bug/autoarray/interferometer_log_evidence_large_regularization_coefficient_cholesky.md.