feat: per-channel SparseTerms sum and phase-centre shifts in sparse_terms_from_chunks (streaming P5) - #601
Merged
Conversation
…erms_from_chunks (streaming P5) - SparseTerms gains `phase_centre` provenance ((y, x) arcsec), checked in __add__ like origin/eps, and __radd__ for 0 so sum(list_of_terms) works (MFS terms = sum of per-channel terms). - sparse_terms_from_chunks(phase_centre=...) multiplies each chunk's visibilities by exp(+2 pi i (u l0 + v m0)) before forming the dirty image, re-centring a source at (y0, x0) onto the origin; every other term is built from the unshifted chunk and is bit-identical. Unshifted accumulations record (0.0, 0.0), so shifted and unshifted terms refuse to be summed. - Interferometer.from_stream forwards phase_centre. - Tests: channel sum == one accumulation == in-memory MFS (numpy + JAX), phase centre == in-memory on pre-shifted data (NUFFT + DFT), DFT point-source sign/order pin, provenance checks, array-free inversion parity. Refs #600 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…a path, check transformer provenance (streaming P5 review) - sparse_terms_from_chunks applies the phase-centre shift to the data before every data-dependent term, so the dirty image and data_term describe the same shifted visibilities (with sigma_r ~= sigma_i within the check's rtol, the unshifted data_term was off: 1e8 vs 99998200.02 on a quarter-turn baseline). data_term is phase-invariant only for exactly equal sigmas; the P5 test now compares it at rel 1e-12. - Interferometer.apply_sparse_operator_from_chunks rejects phase_centre with a DatasetException pointing at from_stream: it shifted the operator but not the retained data (sparse chi2 0 vs 2 against the retained data). - SparseTerms.__add__ checks transformer_class_name with the other recorded provenance (mismatch raises, unrecorded skips). - The MFS channel-sum test's JAX leg now runs the JAX brute force (method="numpy" + use_jax=True, spied); method="nufft" ignores use_jax. Refs #600 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Collaborator
Author
|
Workspace PR: PyAutoLabs/autolens_workspace#582 (array-free datacube example; merges after this PR under the library-first gate). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Streaming visibilities phase 5, the last phase of the
streaming-visibilitiesepic (Discussion https://github.com/orgs/PyAutoLabs/discussions/13; issue #600). Two asks from the discussion land here:SparseTermsare the sum of the per-channel terms.SparseTerms.__add__already summed field-wise; this PR adds__radd__sosum(list_of_terms)works and proves the identity: three channels accumulated separately and summed equal one accumulation over all chunks and the in-memory MFSapply_sparse_operatoroperator and scalars at rel 1e-12, in numpy anduse_jax=True; an array-free dataset built from the summed terms reproduces the in-memory MFS inversion andlog_evidenceat 1e-10.sparse_terms_from_chunks(..., phase_centre=(y0, x0))(arcsec, autoarray(y, x)order) multiplies each chunk's visibilities byexp(+2πi(u l0 + v m0))(l0 = x0, m0 = y0 in radians) before the dirty image is formed, so a source at (y0, x0) lands at the image origin. Threaded throughInterferometer.from_stream. Onlydirty_image_nativechanges; W̃, the dirty beam,sum_weights,data_termandnoise_normalizationare bit-identical to the unshifted run (asserted). The sign and the (y, x) order are pinned by a DFT point-source test: a source at native pixel (3, 2) of an 11×11 mask peaks at the mask centre (5, 5) afterphase_centre=(1.0, −1.5), and at (8, 0) with the opposite sign.phase_centreis recorded asSparseTermsprovenance and checked in__add__, so shifted and unshifted channel terms cannot be summed silently. Unshifted accumulations record(0.0, 0.0);Nonemeans "not recorded" (hand-built terms) and still adds.The companion
autolens_workspacePR adds the array-free datacube example (scripts/interferometer/features/datacube/modeling_array_free.py, smoke-listed).API Changes
Additive:
SparseTerms.phase_centre(provenance),SparseTerms.__radd__,sparse_terms_from_chunks(phase_centre=),Interferometer.from_stream(phase_centre=). Behaviour change:SparseTerms.__add__raises on differing recorded phase centres, and accumulations without a shift now recordphase_centre=(0.0, 0.0).See full details below.
Review (Codex gpt-6-astra, 2026-10-01)
Five findings, each reproduced before editing. Fixed in the second commit: F1 (high) the PyAutoGalaxy
dataset.fitsround trip droppedphase_centre(reloadedNonesummed with anything) → persisted losslessly in the companion PyAutoGalaxy PR; F2 (high)apply_sparse_operator_from_chunks(phase_centre=)desynchronised the operator from the retained data (sparse χ² 0 vs residual χ² 2 on a one-visibility case) → typed raise; F4 (medium)data_termcame from unshifted data while the dirty image used shifted data (Codex case: 1e8 vs 99998200.02, Δχ² 1800 with σr≈σi inside the 1e-5 tolerance) → shift applied before every data-dependent term; plus the jax leg of the MFS test now exercises the JAX kernel builder (spy-asserted) and__add__checkstransformer_class_name. F3/F5 are pre-existing (NUFFT ignores a nonzero mask origin; provenance cannot see masks with the same geometry but different masked pixels) → Mind draftdraft/bug/autoarray/sparse_terms_nufft_origin_and_mask_compatibility.md.Test Plan
__radd__accepts0only; phase-centre accumulation == in-memory on pre-shifted data (NUFFT, DFT) with unshifted terms bit-identical; DFT point-source re-centring (sign + order);__add__phase-centre mismatch raises / unrecorded-left keeps right;from_stream(phase_centre=)matches the pre-shifted in-memory dataset (operator 1e-12, inversion +log_evidence1e-10);from_sparse_terms(sum(per-channel terms))matches the in-memory MFS dataset.test_autoarray: 1917 passed (main: 1903).Full API Changes (for automation & release notes)
Added
aa.SparseTerms.phase_centre: Optional[Tuple[float, float]] = None— provenance, arcsec,(y, x);(0.0, 0.0)for unshifted accumulations,None= not recorded.aa.SparseTerms.__radd__(other)— returnsselfforother == 0sosum(list_of_terms)works;NotImplementedotherwise.aa.util.inversion_interferometer.sparse_terms_from_chunks(..., phase_centre=None)— per-chunkdata *= exp(+2πi(u·l0 + v·m0)), l0 = x0, m0 = y0 in radians (arcsec input); dirty image re-centred on (y0, x0); all other terms unchanged.aa.Interferometer.from_stream(..., phase_centre=None)— forwards to the accumulator.Changed Behaviour
aa.SparseTerms.__add__— raisesInversionExceptionwhenphase_centreis recorded on both operands and differs (as fororigin/pixel_scales/shape_native/eps).sparse_terms_from_chunks— recordsphase_centre=(0.0, 0.0)when no shift is given (was unrecorded).Interferometer.apply_sparse_operator_from_chunks(..., phase_centre=…)— now raisesDatasetException(it would shift the operator's dirty image but not the retaineddata); usefrom_stream(phase_centre=…).sparse_terms_from_chunks(phase_centre=…)—data_termis formed from the shifted visibilities (equal to the unshifted value to rounding when σr = σi exactly), so every data-dependent term describes the same visibilities.SparseTerms.__add__— also refuses terms whose recordedtransformer_class_namediffers.Heart RED development override (human-authorized, 2026-10-01)
Shipped under the AUTONOMY.md "Human override for Heart RED (development only)". Live human authorization in the CLI session (Fable 5.1), quoted: "Yes: library now, workspace when its run passes" for task
streaming-p5-cubes-phase-centre(#600). Exact RED reasons at ship time (pyauto-heart readiness):None relate to this branch. Branch gates passed: test_autoarray 1917, test_autogalaxy 1307; Codex review 5 findings (3 introduced → fixed + red-checked, 2 pre-existing → filed). Scope: push + PR-open only. Merge requires a separate human
/prmwith every required check green; not a release; no claim that Heart is healthy.Generated by the PyAutoLabs agent workflow.
🤖 Generated with Claude Code