bin/python size exploded 2000 times larger

[gfrankliu]

Is there a reason why the binary bin/python3.10 in latest python 3.10.19 has size 43963640 which is more than 2000 times larger than bin/python3.10 in python 3.10.15 (size 20944)?

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    is_default = True,
    python_version = "3.10.15",
)

[shayanhoshyari]

I saw something similar that 3.12 bundle on macos ~10MB is much smaller than linux ~100MB.

[aignas]

These are questions to the upstream project https://github.com/astral-sh/python-build-standalone/releases

We do not repackage anything and we are not testing for the size of the interpreter.

[gfrankliu]

Maybe we should re-open this.
Based on upstream ticket, they now statically link the libpython into the python binary, so the binary is much bigger. This also means the libpython is no longer needed unless rule_python needs the lib for something else. The above ticket suggested distributions (such as "rules_python") could remove the libpython and that should reduce the total size same as before.

gfrankliu-ws ➜  ls -l bin lib
bin:
total 42968
lrwxrwxrwx 1 gfrankliu gfrankliu        9 Jan 21 18:09 2to3 -> 2to3-3.10
-rwxrwxr-x 1 gfrankliu gfrankliu      158 Jan  1  2024 2to3-3.10
lrwxrwxrwx 1 gfrankliu gfrankliu        8 Jan 21 18:09 idle3 -> idle3.10
-rwxrwxr-x 1 gfrankliu gfrankliu      156 Jan  1  2024 idle3.10
-rwxrwxr-x 1 gfrankliu gfrankliu      259 Jan  1  2024 pip
-rwxrwxr-x 1 gfrankliu gfrankliu      259 Jan  1  2024 pip3
-rwxrwxr-x 1 gfrankliu gfrankliu      259 Jan  1  2024 pip3.10
lrwxrwxrwx 1 gfrankliu gfrankliu        9 Jan 21 18:09 pydoc3 -> pydoc3.10
-rwxrwxr-x 1 gfrankliu gfrankliu      141 Jan  1  2024 pydoc3.10
lrwxrwxrwx 1 gfrankliu gfrankliu       10 Jan 21 18:09 python -> python3.10
lrwxrwxrwx 1 gfrankliu gfrankliu       10 Jan 21 18:09 python3 -> python3.10
lrwxrwxrwx 1 gfrankliu gfrankliu       17 Jan 21 18:09 python3-config -> python3.10-config
-rwxrwxr-x 1 gfrankliu gfrankliu 43963640 Jan  1  2024 python3.10
-rwxrwxr-x 1 gfrankliu gfrankliu     3092 Jan  1  2024 python3.10-config

lib:
total 48288
drwxr-xr-x  2 gfrankliu gfrankliu     4096 Jan 21 18:09 itcl4.2.4
lrwxrwxrwx  1 gfrankliu gfrankliu       20 Jan 21 18:09 libpython3.10.so -> libpython3.10.so.1.0
-rwxrwxr-x  1 gfrankliu gfrankliu 44339056 Jan  1  2024 libpython3.10.so.1.0
-rwxrwxr-x  1 gfrankliu gfrankliu    16800 Jan  1  2024 libpython3.so
-rwxrwxr-x  1 gfrankliu gfrankliu  1945712 Jan  1  2024 libtcl8.6.so
-rwxrwxr-x  1 gfrankliu gfrankliu  3102232 Jan  1  2024 libtk8.6.so
drwxr-xr-x  2 gfrankliu gfrankliu     4096 Jan 21 18:09 pkgconfig
drwxr-xr-x 35 gfrankliu gfrankliu     4096 Jan 21 18:09 python3.10
drwxr-xr-x  5 gfrankliu gfrankliu     4096 Jan 21 18:09 tcl8
drwxr-xr-x  6 gfrankliu gfrankliu     4096 Jan 21 18:09 tcl8.6
drwxr-xr-x  2 gfrankliu gfrankliu     4096 Jan 21 18:09 thread2.8.9
drwxr-xr-x  6 gfrankliu gfrankliu     4096 Jan 21 18:09 tk8.6
gfrankliu-ws ➜  

[aignas]

Reopening for discussion/planning and implementing, thank you for the reference.

[rickeylev]

Thanks for bringing this up. It's nice to have a concrete reason to look at the runtime size of the Python runtime. There's many things in the python runtime that many apps probably don't need at runtime (bundled pip and headers come to mind; within Google we strip more out by default).

remove libpython ... unless needed for something else

There's two something elses:

(1) If a C extension is being built: it needs libpython to link against. This is a build-time only need.

(2) If something is dynamically linking at runtime, e.g. an embedded python C program, then it needs libpython at runtime.

In either case, libpython has to stick around in the extracted repo. That doesn't mean it has to go into the runfiles, though.

The issue, though, is, given an arbitrary binary, we don't know what it needs from the Python runtime it needs at runtime.

I'm +1 on breaking up the glob for the runtime files into some separate targets, and having flags and/or PyRuntime fields carry them.