Run a job that the test suite with MSan to the CI

[StanFromIreland]

We have to disable extension modules that link against system libraries (which are not built with MSan) since memory those libraries initialise is reported as uninitialised. While this does significantly reduce coverage of some modules, it saves a great amount of CI time. I also had to unpoison a few buffers filled by libc that MSan does not intercept.

A little fix is included, allocate_instrumentation_data() now zeroes the tool_versions of a _PyCoMonitoringData, which update_instrumentation_data() previously read uninitialised. This doesn't have an affect in practice, as the garbage data only decides whether to clear some bits that are already zero, so the outcome is the same either way.

Inspired by #158584.

[StanFromIreland]

Note, test_bytes will be failing till #158584 lands.

[vstinner]

Interesting change.

[vstinner]

test_faulthandler seems to log SEGV from faulthandler_raise_sigsegv and log FPE from faulthandler__sigfpe_impl().

TSan is run with TSAN_OPTIONS="handle_segv=0 (...)".

At least, skip_if_sanitizer_signal() of test_faulthandler can be updated to add memory=True:

    return support.skip_if_sanitizer(f"TSAN/UBSan itercepts {signame}",
                                     thread=True, ub=True, memory=True)

[vstinner]

I merged my bytes.fromhex() fix #158584 so you update the branch to retrieve the fix.

[StanFromIreland]

It passes now! 🎉 And in good time, slightly faster than UBSan.

[vstinner]

LGTM.

I still see faulthandler__sigsegv in "Display logs" of the MSan job. I suggest this change:

diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py
index 82b347c8f8c..2c15cd57455 100644
--- a/Lib/test/test_faulthandler.py
+++ b/Lib/test/test_faulthandler.py
@@ -172,6 +172,7 @@ def check_windows_exception(self, code, line_number, name_regex, **kw):
         self.check_error(code, line_number, fatal_error, **kw)
 
     @skip_segfault_on_android
+    @skip_if_sanitizer_signal("SIGSEGV")
     def test_sigsegv(self):
         self.check_fatal_error("""
             import faulthandler

It may interesting document how to disable extensions in https://docs.python.org/dev/using/configure.html#cmdoption-with-memory-sanitizer documentation. Give an example of Modules/Setup.local file (as shown below).

I tested manually the change:

$ cat Modules/Setup.local
*disabled*
_bz2 _ctypes _curses _curses_panel _dbm _decimal _gdbm _hashlib
_lzma _sqlite3 _ssl _tkinter _uuid _zstd readline zlib

$ ./configure --config-cache OPT="-O2 -g" --with-memory-sanitizer --with-assertions CC=clang LD=clang
$ make clean
$ make
$ export MSAN_OPTIONS="log_path=$PWD/san_log allocator_may_return_null=1"

---

There is another error logged in "Display logs". I suppose that conftest is a C program created by configure. I failed to reproduce the issue locally so far.

==> /home/runner/work/cpython/cpython/san_log.5297 <==
==5297==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55555562618d in main (/home/runner/work/cpython/cpython/conftest+0xd218d) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)
    #1 0x7ffff7c2a8c0  (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2a8c0) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #2 0x7ffff7c2a9d7 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2a9d7) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #3 0x555555587324 in _start (/home/runner/work/cpython/cpython/conftest+0x33324) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)

  Uninitialized value was stored to memory at
    #0 0x555555626126 in main (/home/runner/work/cpython/cpython/conftest+0xd2126) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)
    #1 0x7ffff7c2a8c0  (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2a8c0) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #2 0x7ffff7c2a9d7 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2a9d7) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #3 0x555555587324 in _start (/home/runner/work/cpython/cpython/conftest+0x33324) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)

  Uninitialized value was created by an allocation of 'rc' in the stack frame
    #0 0x555555626071 in main (/home/runner/work/cpython/cpython/conftest+0xd2071) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/runner/work/cpython/cpython/conftest+0xd218d) (BuildId: 96d049a1f86e24824420f04077896a47dfca1208) in main
Exiting

[vstinner]

Maybe Objects/obmalloc.c can be modified to not use debug hooks on memory allocations when Python is built with --with-memory-sanitizer. When I did tests on Valgrind, I set PYTHONMALLOC=malloc environment variable to disable debug hooks and disable pymalloc, to use the generic malloc()/free().

[StanFromIreland]

I will do it as a follow-up to this/#157934. I thought about building with debug here and using PYTHONMALLOC=malloc here, but it does not propagate to subprocesses so I worry we would lose some coverage.

[vstinner]

There is another error logged in "Display logs". I suppose that conftest is a C program created by configure. I failed to reproduce the issue locally so far.

Ah! I found the last error. It's comes from a ffi_call() in configure. I suggest this fix:

diff --git a/configure.ac b/configure.ac
index e1d55a7a3ce..86e12ef671f 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4458,7 +4458,7 @@ int main(void)
     {
         return 2;
     }
-    ffi_arg rc;
+    ffi_arg rc = 0;
     ffi_call(&cif, FFI_FN(z_is_expected), &rc, values);
     return !rc;
 }

I suppose that MSAN complains because my system libffi library was not built with MSAN.

cc @skirpichev who wrote that configure test if I recall correctly.

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34933474 | 📁 Comparing ef1342e against main (9d22a53)

  🔍 Preview build  

1 file changed

± using/configure.html

[vstinner]

LGTM. Thanks for the update.

[pablogsal]

LGTM!

[miss-islington-app]

Thanks @StanFromIreland for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @StanFromIreland and @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker b93fb19a6e3118857d9a1dc4ffcc179b632a88c1 3.15

[bedevere-app]

GH-158832 is a backport of this pull request to the 3.15 branch.

[bedevere-app]

GH-158834 is a backport of this pull request to the 3.14 branch.