The embeddable package from python.org [1] is 20 MB unpacked. To install different sets of packages per workbook, just have Excel automatically set the module path as appropriate.

[1] https://www.python.org/downloads/release/python-3114/