Skip to content

feat: per-channel SparseTerms sum and phase-centre shifts in sparse_terms_from_chunks (streaming P5) - #601

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/streaming-p5-cubes-phase-centre
Oct 1, 2026
Merged

Jammy2211 merged 2 commits into
mainfrom
feature/streaming-p5-cubes-phase-centre

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

Streaming visibilities phase 5, the last phase of the streaming-visibilities epic (Discussion https://github.com/orgs/PyAutoLabs/discussions/13; issue #600). Two asks from the discussion land here:

  • Per-channel cubes. MFS (multi-frequency synthesis) SparseTerms are the sum of the per-channel terms. SparseTerms.__add__ already summed field-wise; this PR adds __radd__ so sum(list_of_terms) works and proves the identity: three channels accumulated separately and summed equal one accumulation over all chunks and the in-memory MFS apply_sparse_operator operator and scalars at rel 1e-12, in numpy and use_jax=True; an array-free dataset built from the summed terms reproduces the in-memory MFS inversion and log_evidence at 1e-10.
  • Phase-centre shifts, chunk by chunk. sparse_terms_from_chunks(..., phase_centre=(y0, x0)) (arcsec, autoarray (y, x) order) multiplies each chunk's visibilities by exp(+2πi(u l0 + v m0)) (l0 = x0, m0 = y0 in radians) before the dirty image is formed, so a source at (y0, x0) lands at the image origin. Threaded through Interferometer.from_stream. Only dirty_image_native changes; W̃, the dirty beam, sum_weights, data_term and noise_normalization are bit-identical to the unshifted run (asserted). The sign and the (y, x) order are pinned by a DFT point-source test: a source at native pixel (3, 2) of an 11×11 mask peaks at the mask centre (5, 5) after phase_centre=(1.0, −1.5), and at (8, 0) with the opposite sign.
  • phase_centre is recorded as SparseTerms provenance and checked in __add__, so shifted and unshifted channel terms cannot be summed silently. Unshifted accumulations record (0.0, 0.0); None means "not recorded" (hand-built terms) and still adds.

The companion autolens_workspace PR adds the array-free datacube example (scripts/interferometer/features/datacube/modeling_array_free.py, smoke-listed).

API Changes

Additive: SparseTerms.phase_centre (provenance), SparseTerms.__radd__, sparse_terms_from_chunks(phase_centre=), Interferometer.from_stream(phase_centre=). Behaviour change: SparseTerms.__add__ raises on differing recorded phase centres, and accumulations without a shift now record phase_centre=(0.0, 0.0).
See full details below.

Review (Codex gpt-6-astra, 2026-10-01)

Five findings, each reproduced before editing. Fixed in the second commit: F1 (high) the PyAutoGalaxy dataset.fits round trip dropped phase_centre (reloaded None summed with anything) → persisted losslessly in the companion PyAutoGalaxy PR; F2 (high) apply_sparse_operator_from_chunks(phase_centre=) desynchronised the operator from the retained data (sparse χ² 0 vs residual χ² 2 on a one-visibility case) → typed raise; F4 (medium) data_term came from unshifted data while the dirty image used shifted data (Codex case: 1e8 vs 99998200.02, Δχ² 1800 with σr≈σi inside the 1e-5 tolerance) → shift applied before every data-dependent term; plus the jax leg of the MFS test now exercises the JAX kernel builder (spy-asserted) and __add__ checks transformer_class_name. F3/F5 are pre-existing (NUFFT ignores a nonzero mask origin; provenance cannot see masks with the same geometry but different masked pixels) → Mind draft draft/bug/autoarray/sparse_terms_nufft_origin_and_mask_compatibility.md.

Test Plan

  • New (14): sum of channels == MFS (numpy, jax); __radd__ accepts 0 only; phase-centre accumulation == in-memory on pre-shifted data (NUFFT, DFT) with unshifted terms bit-identical; DFT point-source re-centring (sign + order); __add__ phase-centre mismatch raises / unrecorded-left keeps right; from_stream(phase_centre=) matches the pre-shifted in-memory dataset (operator 1e-12, inversion + log_evidence 1e-10); from_sparse_terms(sum(per-channel terms)) matches the in-memory MFS dataset.
  • test_autoarray: 1917 passed (main: 1903).
  • CI green on all legs.
Full API Changes (for automation & release notes)

Added

  • aa.SparseTerms.phase_centre: Optional[Tuple[float, float]] = None — provenance, arcsec, (y, x); (0.0, 0.0) for unshifted accumulations, None = not recorded.
  • aa.SparseTerms.__radd__(other) — returns self for other == 0 so sum(list_of_terms) works; NotImplemented otherwise.
  • aa.util.inversion_interferometer.sparse_terms_from_chunks(..., phase_centre=None) — per-chunk data *= exp(+2πi(u·l0 + v·m0)), l0 = x0, m0 = y0 in radians (arcsec input); dirty image re-centred on (y0, x0); all other terms unchanged.
  • aa.Interferometer.from_stream(..., phase_centre=None) — forwards to the accumulator.

Changed Behaviour

  • aa.SparseTerms.__add__ — raises InversionException when phase_centre is recorded on both operands and differs (as for origin / pixel_scales / shape_native / eps).

  • sparse_terms_from_chunks — records phase_centre=(0.0, 0.0) when no shift is given (was unrecorded).

  • Interferometer.apply_sparse_operator_from_chunks(..., phase_centre=…) — now raises DatasetException (it would shift the operator's dirty image but not the retained data); use from_stream(phase_centre=…).

  • sparse_terms_from_chunks(phase_centre=…) — data_term is formed from the shifted visibilities (equal to the unshifted value to rounding when σr = σi exactly), so every data-dependent term describes the same visibilities.

  • SparseTerms.__add__ — also refuses terms whose recorded transformer_class_name differs.

Heart RED development override (human-authorized, 2026-10-01)

Shipped under the AUTONOMY.md "Human override for Heart RED (development only)". Live human authorization in the CLI session (Fable 5.1), quoted: "Yes: library now, workspace when its run passes" for task streaming-p5-cubes-phase-centre (#600). Exact RED reasons at ship time (pyauto-heart readiness):

  • release validation FAILED (stage integrate)
  • workspace validation not passing (0 failed, 1 timeout, cloud#36404726969: autolens_test scripts/multi_dataset/rectangular.py)
  • manifest drift: public front-door organ tables (generated) — 1 mismatch(es) vs PyAutoMind/repos.yaml

None relate to this branch. Branch gates passed: test_autoarray 1917, test_autogalaxy 1307; Codex review 5 findings (3 introduced → fixed + red-checked, 2 pre-existing → filed). Scope: push + PR-open only. Merge requires a separate human /prm with every required check green; not a release; no claim that Heart is healthy.

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

Jammy2211 and others added 2 commits October 1, 2026 10:20
…erms_from_chunks (streaming P5)

- SparseTerms gains `phase_centre` provenance ((y, x) arcsec), checked in
  __add__ like origin/eps, and __radd__ for 0 so sum(list_of_terms) works
  (MFS terms = sum of per-channel terms).
- sparse_terms_from_chunks(phase_centre=...) multiplies each chunk's
  visibilities by exp(+2 pi i (u l0 + v m0)) before forming the dirty image,
  re-centring a source at (y0, x0) onto the origin; every other term is built
  from the unshifted chunk and is bit-identical. Unshifted accumulations
  record (0.0, 0.0), so shifted and unshifted terms refuse to be summed.
- Interferometer.from_stream forwards phase_centre.
- Tests: channel sum == one accumulation == in-memory MFS (numpy + JAX),
  phase centre == in-memory on pre-shifted data (NUFFT + DFT), DFT
  point-source sign/order pin, provenance checks, array-free inversion parity.

Refs #600

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…a path, check transformer provenance (streaming P5 review)

- sparse_terms_from_chunks applies the phase-centre shift to the data before
  every data-dependent term, so the dirty image and data_term describe the
  same shifted visibilities (with sigma_r ~= sigma_i within the check's
  rtol, the unshifted data_term was off: 1e8 vs 99998200.02 on a
  quarter-turn baseline). data_term is phase-invariant only for exactly
  equal sigmas; the P5 test now compares it at rel 1e-12.
- Interferometer.apply_sparse_operator_from_chunks rejects phase_centre with
  a DatasetException pointing at from_stream: it shifted the operator but
  not the retained data (sparse chi2 0 vs 2 against the retained data).
- SparseTerms.__add__ checks transformer_class_name with the other recorded
  provenance (mismatch raises, unrecorded skips).
- The MFS channel-sum test's JAX leg now runs the JAX brute force
  (method="numpy" + use_jax=True, spied); method="nufft" ignores use_jax.

Refs #600

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Jammy2211

Copy link
Copy Markdown
Collaborator Author

Workspace PR: PyAutoLabs/autolens_workspace#582 (array-free datacube example; merges after this PR under the library-first gate).

@Jammy2211
Jammy2211 merged commit ffd13ba into main Oct 1, 2026
3 checks passed
@Jammy2211
Jammy2211 deleted the feature/streaming-p5-cubes-phase-centre branch October 1, 2026 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant