Have a look at https://httptoolkit.com/ which works with a lot tools (even cli)

It's not as simple as it sounds, it requires a lot of code to capture Python traffic with charles-proxy. For example, you might modify your python code to use a Proxy and accept a self-signed Charles's certificate.

If you need a 1-click solution, no dependencies, and no code's required, check out Proxyman with Auto-Setup: https://docs.proxyman.io/automatic-setup/automatic-setup

Works with all popular Python libs: request, aiohttp, http.client, urllib3, etc

* Disclaimer: I'm Noah, creator of Proxyman. I know a pain point when using Charles, and decided to build a new one, to make life easier. Hope it helps you.

That’s fine if you can, but say you want to trace deployed on stage, or shudders even production. Or you code tests against some CI and can only add Python.

I'm really surprised this was downvoted. Running e.g. mitmproxy and pointing your Python process at it is absolutely the way to do this.

It is important to check checksums (and signatures, if there are any) of downloaded packages prior to installing them; especially when resuming interrupted downloads.

Pip has a hash-checking mode, but it only works if the hashes are listed in the requirements.txt file, and they're the hashes for the target platform. Pipfile.lock supports storeing hashes for multiple platforms, but requirements.txt does not.

If the package hashes are retrieved over the same channel as the package, they can be MITM'd too.

You can store PyPi package hashes in sigstore.

There should be a way for package uploaders to sign their package before uploading. (This is what .asc signatures on PyPi were for. But if they are retrieved over the same channel, cryptographic signatures can also be MITM'd).

IMHO (1) twine should prompt to sign the package (with a DID) before uploading the package to PyPi, and (2) after uploading packages, twine should download the package(s) it has uploaded to verify the signature.

; TCP RESET and Content-Range doesn't hash resources.

I really like vrcpy. I used it a lot with pytest in my previous job. httpdbg isn’t exactly the same; the idea is more about seeing HTTP requests in real-time and being able to easily study them.

The inspection and debugging features this offers are great additions though. I've stared at VCR yaml enough times to not want to ever do it again.

You can do that with django debug toolbar. If you have an endpoint that doesn't return HTML, and hence wouldn't render debug toolbar, you can go to django admin (or any other endpoint that would render ddt) and go to the history pane, check other requests and switch to them.

It seems to intercept calls for some popular http client libs:

https://github.com/cle-b/httpdbg/tree/main/httpdbg/hooks

It works only with Python code because it intercepts HTTP requests by hooking into certain Python functions.

It supports any HTTP library based on Python’s standard socket library. Specifically, it works with libraries like requests, httpx, aiohttp, and urllib3, as well as pytest, providing more detailed information about the initiator of the requests.

If you want to trace the HTTP requests in a notebook, you must install the package notebook-httpdbg.

This is documented here: https://httpdbg.readthedocs.io/en/latest/notebook/

CPython since 3.8 already has built-in audit events, including open, so you don't need to patch anything or use anything external. Just add an audit hook with sys.addaudithook().

Quick example:

    import inspect
    import pathlib
    import sys


    def callsite():
        try:
            pathlib.Path("/tmp/file").open()
        except:
            pass


    def audit_hook(event, args):
        if event == "open":
            path, mode, flags = args
            print(f"audit: open({path!r}, {mode!r}, 0o{flags:o})")
            # Not using traceback here because traceback will attempt to read the
            # source file, causing an infinite recursion of audit events.
            f = inspect.currentframe()
            while f := f.f_back:
                print(
                    f'File "{f.f_code.co_filename}", line {f.f_lineno}, in {f.f_code.co_name}'
                )


    def main():
        sys.addaudithook(audit_hook)
        callsite()


    if __name__ == "__main__":
        main()

    audit: open('/tmp/file', 'r', 0o100000000)
    File "/path/to/python/lib/python3.12/pathlib.py", line 1013, in open
    File "/tmp/audit.py", line 10, in callsite
    File "/tmp/audit.py", line 26, in main
    File "/tmp/audit.py", line 30, in <module>

If on Linux or windows you can use Procmon or Instruments on macos.

https://github.com/Sysinternals/ProcMon-for-Linux

Went spelunking through the source. I think you absolutely could!

There's actually not a whole lot I found that's really http-library specific. It uses the traceback module in a decorator that ends up being manually wrapped around all of the functions of the specific libraries the author cared about.

https://github.com/cle-b/httpdbg/blob/main/httpdbg/hooks

Should be easy enough to extend this to other libraries.

Super cool tool thanks for sharing @dmurray!

You might find the syscall tracing functionality of Cirron useful: https://github.com/s7nfo/Cirron

Unlike other tools such as proxies that allow you to trace HTTP requests, httpdbg makes it possible to link the HTTP request to the Python code that initiated it. This is why it is specific to Python and does not work with other languages.

In the old days we'd use tcpdump and wireshark for this, but nowadays everything is encrypted up in the application layer so you need this kind of thing. Or tricky key dumping hacks.

I think the nice thing about HTTP for this is different parts of the stack can introduce default headers etc and it's helpful to be able to see the actual request after all that processing's been done.

Is there an example?