feat(interferometer): array-free Interferometer.from_stream / from_sparse_terms (streaming phase 1, Discussion #13)

[Jammy2211]

Overview

Phase 1 of the array-free streamed interferometer dataset (Discussion https://github.com/orgs/PyAutoLabs/discussions/13, phase 2 of the proposal; Mind epic streaming-visibilities). Phase 1 of the discussion (#589 + PyAutoGalaxy#637 + PyAutoLens#757 + #591) made the sparse likelihood array-free, but the process still holds every visibility: Interferometer.__init__ always builds a TransformerNUFFT from uv_wavelengths (~48 B/vis on top of the dataset's 48 B/vis), AbstractDataset.__init__ rejects noise_map=None, and apply_sparse_operator_from_chunks returns a dataset that retains all arrays. This phase adds Interferometer.from_stream / from_sparse_terms that build a dataset with no arrays and no transformer, and makes every consumer either array-free or fail with a typed error.

Plan

• Give SparseTerms provenance (mask shape/pixel_scales/origin, eps, transformer class) and make __add__ check it.
• Let Interferometer be built without uv_wavelengths / arrays / transformer and store sparse_terms; add from_sparse_terms and from_stream.
• Route AbstractInversionInterferometer.mask through the dataset mask instead of the transformer; typed DatasetException from every array/transformer property when absent.
• Add dirty_image_natural and dirty_beam on both in-memory and array-free paths.
• Tests for construction, sparse-inversion parity (numpy + jax), typed errors, provenance; fresh-process RSS witness.

Detailed implementation plan

Affected Repositories

• PyAutoArray (only). PyAutoGalaxy / PyAutoLens suites run for regression; no change there.

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoArray	main	clean

Worktree guard clear. Suggested branch: feature/streaming-p1-array-free-dataset

Design decisions (human-approved 2026-09-30, recorded in the epic ledger)

(a) transformer=None, no stub class; consumers gate and raise exc.DatasetException. (c) in-memory dirty_image unchanged; new dirty_image_natural / dirty_beam on both paths. (e) SparseTerms carries provenance.

Implementation Steps

1. autoarray/inversion/inversion/interferometer/inversion_interferometer_util.py: SparseTerms optional fields shape_native, pixel_scales, origin, eps, transformer_class_name; sparse_terms_from_chunks fills them from real_space_mask / kwargs; __add__ raises on mismatched shape / pixel_scales / eps.
2. autoarray/dataset/interferometer/dataset.py __init__: uv_wavelengths optional; build the transformer only when uv is present and transformer_class is not None, else self.transformer = None; skip the DFT-limit check when uv is None; new sparse_terms=None kwarg stored.
3. autoarray/dataset/abstract/dataset.py __init__: guard the noise_map=None path (no covariance/diag work when both arrays are None); shape_slim guarded.
4. Interferometer.from_sparse_terms(terms, real_space_mask, *, batch_size=128) and from_stream(chunks, real_space_mask, *, transformer_class=TransformerNUFFT, method="nufft", eps=None, chunk_size=None, chunk_k=2048, use_jax=False, show_progress=False, batch_size=128); apply_sparse_operator_from_chunks stores the terms on the returned dataset.
5. New properties dirty_image_natural (dirty_image_native / sum_weights) and dirty_beam from sparse_terms when present, else from the operator's cached native images; amplitudes, phases, uv_distances, dirty_image, dirty_noise_map, signal_to_noise_map, psf_precision_operator_from, apply_sparse_operator raise exc.DatasetException naming from_stream when their input is None.
6. autoarray/inversion/inversion/interferometer/abstract.py mask → dataset mask (real_space_mask on Interferometer, grids.lp.mask on DatasetInterface); check the sparse numba/jax classes and mapped_reconstructed_data_dict follow; mapped_reconstructed_operated_data_dict raises the typed exception when transformer is None.
7. autoarray/fit/fit_interferometer.py: mask, transformer, dirty_*, residual / normalized-residual / chi-squared maps raise the typed exception when arrays are absent.
8. Tests + fresh-process RSS witness (5e5..4e6 vis, 400-pixel image, 4096-vis chunks) reported in the PR body.

Key Files

• autoarray/dataset/interferometer/dataset.py, autoarray/dataset/abstract/dataset.py
• autoarray/inversion/inversion/interferometer/{inversion_interferometer_util.py,abstract.py}
• autoarray/fit/fit_interferometer.py
• test_autoarray/dataset/interferometer/test_dataset.py, test_autoarray/inversion/inversion/interferometer/

Original Prompt

Click to expand starting prompt

Streaming phase 1: array-free Interferometer.from_stream / from_sparse_terms (PyAutoArray)

Type: feature
Target: PyAutoArray
Repos:

• PyAutoArray
  Themes:
• interferometer
• sparse-operator
• memory
  Autonomy: supervised
  Priority: medium
  Status: draft
  Epic: streaming-visibilities
  Phase: 1
  Difficulty: medium
  Consequence: judge
  Witness: aa.Interferometer.from_stream(chunks, real_space_mask, ...) and from_sparse_terms(terms, real_space_mask) return a dataset whose data, noise_map, uv_wavelengths and transformer are all None and which carries sparse_terms; an aa sparse inversion (rectangular mesh, numpy and jax) on it gives log_evidence equal to the in-memory apply_sparse_operator inversion at rel 1e-8; every array/transformer property raises a typed DatasetException naming from_stream; SparseTerms.__add__ refuses mismatched provenance; a fresh-process RSS witness is flat from 5e5 to 4e6 visibilities (≤ ~40 MB over baseline).
  Review-minutes: 8
  Unattended: ready
  Parent: draft/feature/autoarray/interferometer_from_stream_array_free_dataset.md

Source: https://github.com/orgs/PyAutoLabs/discussions/13 phase 2, sliced 2026-09-30 (design decisions (a), (c), (e) in the ledger).

Why

After phase 1 the likelihood is array-free but the process is not: Interferometer.__init__ always builds a
TransformerNUFFT from uv_wavelengths (~48 B/vis on top of the dataset's 48 B/vis), AbstractDataset.__init__
raises on noise_map=None, the DFT-limit check reads uv_wavelengths.shape, and apply_sparse_operator_from_chunks
returns a dataset that still retains every array. The one transformer read on the sparse likelihood path is
AbstractInversionInterferometer.mask (transformer.real_space_mask).

What

1. SparseTerms gains optional provenance fields (shape_native, pixel_scales, origin, eps,
   transformer_class_name); sparse_terms_from_chunks fills them; __add__ requires shape / pixel_scales / eps equal.
2. Interferometer.__init__: uv_wavelengths optional; transformer built only when uv is present (else None);
   DFT-limit check skipped when uv is None; new sparse_terms= kwarg stored. AbstractDataset.__init__ guards
   noise_map=None (no covariance work when both arrays are None); shape_slim guarded.
3. Interferometer.from_sparse_terms(terms, real_space_mask, *, batch_size=128) and
   from_stream(chunks, real_space_mask, *, transformer_class=TransformerNUFFT, method, eps, chunk_size, chunk_k, use_jax, show_progress, batch_size); apply_sparse_operator_from_chunks also stores the terms.
4. New dirty_image_natural (dirty_image_native / sum_weights) and dirty_beam on both paths. amplitudes, phases,
   uv_distances, dirty_image, dirty_noise_map, signal_to_noise_map, psf_precision_operator_from,
   apply_sparse_operator raise exc.DatasetException naming the array-free dataset when their input is None.
5. AbstractInversionInterferometer.mask reads the dataset mask (real_space_mask / grids.lp.mask);
   mapped_reconstructed_operated_data_dict raises the typed exception when transformer is None.
   aa.FitInterferometer mask, transformer, dirty_*, residual/chi-squared maps raise it when arrays are absent.
6. Tests in test_autoarray/dataset/interferometer, inversion/interferometer, test_inversion_interferometer_util.py;
   witness script (fresh process per N_vis) in the PR body. No autogalaxy/autolens change; run their suites for regression.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JZZksyZ8LTA4LLxoZjQMNF

[Jammy2211]

Heart RED override — development only (recorded per AUTONOMY.md)

• Heart verdict at ship: RED, 2026-09-30T12:42:09Z. Exact reason string: release validation FAILED (stage integrate). Unrelated to this branch (a release-integrate leg).
• Authorization: live human, in-session 2026-09-30, for this issue (feat(interferometer): array-free Interferometer.from_stream / from_sparse_terms (streaming phase 1, Discussion #13) #592) and branch feature/streaming-p1-array-free-dataset: "Authorize override for feat(interferometer): array-free Interferometer.from_stream / from_sparse_terms (streaming phase 1, Discussion #13) #592" — commit, push and open the pending-release PR. Merge remains a separate explicit human command with every required check green. No release, no rehearsal, no claim that this branch repairs Heart.
• Branch gates passed before the ask: test_autoarray 1779 passed; test_autogalaxy/interferometer 43 passed and test_autolens/interferometer 29 passed with no edits there; red-checks on the new tests (mask reroute reverted → 3 fail, guard removed → 1 fail, provenance check disabled → 2 fail); independent Codex gpt-6-astra review of the working tree: FINDINGS (4) — Feature/config #1 geometry guard in from_sparse_terms (pixel_scales/origin), Fft #2 origin in SparseTerms.__add__, Fft #3 provenance merge when one operand is unrecorded: all three fixed in-branch with red-checked tests; Feature/clean #4 (AnalysisInterferometer.save_attributes / aggregator cannot handle an array-free dataset) is phase 2 by design, documented.

🤖 Generated with Claude Code

[Jammy2211]

Library PR created

• PR: feat(interferometer): array-free Interferometer.from_stream / from_sparse_terms (streaming phase 1, Discussion #13) #593
• test_autoarray/: 1783 passed
• Shipped under the human-authorized Heart RED development override (see comment above); merge stays a separate human command via /prm once CI is green.

🤖 Generated with Claude Code

[Jammy2211]

Shipped

#593 (merge bd03e09) — array-free Interferometer.from_stream / from_sparse_terms, SparseTerms provenance, dataset-routed inversion mask, typed exceptions on array-dependent properties, dirty_image_natural / dirty_beam. Labelled pending-release. Streaming epic phase 1 of 5; phase 2 (fit + save/reload) is next.

🤖 Generated with Claude Code