You mention Python and Node, which are programming language that unusually require end users to have the text of your program and all its dependencies on their own machine, and the languages store some parts of your program in /usr/lib and other parts of your program in your source directory. (npm does a little better here and at least puts the dependencies in your project directory.) Those constraints make development and packaging hard no matter what.
Python and Node both need a way to compile the code down to a single statically-linked binary like more modern languages (Go, Rust), solving the distribution problem once and for all.
There are module systems that aren't insane, like Go's module system. It uses semantic versioning to mediate version conflicts. Programs can import multiple major versions of the same module. The module requirements files ensure that every checkout of the code gets the exact same bytes of the code's dependencies. The compiler is aware of modules and can fetch them, so on a fresh workstation, "go test ./..." or "go install ./cmd/cool-thing" in a module-aware project works without running any other command first. It is actually so pleasant to use that I always think twice "do I want to deal with modules" before using a language like Javascript or Python, and usually decide "no".
npm and pip are the DARK AGES. That's why you're struggling. The community has been unable to fix these fundamental flaws for decades, despite trying in 100 incompatible ways. I personally have given up on the languages as a result. The standard library is never enough. You HAVE to solve the module problem on day 1.
I haven't used Perl for 15 years, but it was pretty miserable back then. Obviously I had my workflow and didn't run into too many problems, but I was certainly nervous about how I could share my work with other people. (Putting it into prod wasn't that bad. I don't know why, but it always went OK. CPANPLUS helped at the time.) I used to teach Perl trainings as a side job, and people would literally be in tears over @INC. It was bad.
I don't use Python much these days, but it's not as bad as Perl 15 years ago. I see blog posts like "how to set up the perfect Python dev environment with Docker" and it makes me very sad, but at least teams are getting their work done. The edge cases of Python packaging, though, are really really bad. For example, the C compiler that Python was compiled with (and C library; musl vs. glibc) affects installability of modules through pip. This really forces your Linux distribution to be the arbiter of what modules you can use, which is always too out of date for development. Also, the exact requirements (what are the sha256s and URLs of the code files that this code depends on) depends on architecture, compiler, etc. As far as I can tell, you have to run code to find the whole dependency tree. That is the fatal flaw I see with Python's packaging system.
I spent a lot of time trying to use Bazel to unify all the work my company does across languages. Python was the killer here. We do depend on a lot of C extensions, and I have a C toolchain built with Bazel (so that arm64 mac users can cross-compile to produce Linux releases; maybe a dumb requirement, but it would be too unpopular to discriminate against certain developer devices), but getting Python built with that toolchain, and using that Python to evaluate requirements.txt didn't work. (There are bugs open; they all end with "Python has to fix this".) As a result, you don't get compatible versions of, say, numpy with this setup. Dealbreaker. Partially Bazel's fault, partially Pyton's fault.
(It pained me to use Bazel for Go, but it did all work. While the workflow wasn't as nice as what Go has built in, easy things were hard and hard things were possible. I had working binaries within a few hours, buildable on any supported workstation type, with object and test caching in the cloud.)
Python and Node both need a way to compile the code down to a single statically-linked binary like more modern languages (Go, Rust), solving the distribution problem once and for all.
There are module systems that aren't insane, like Go's module system. It uses semantic versioning to mediate version conflicts. Programs can import multiple major versions of the same module. The module requirements files ensure that every checkout of the code gets the exact same bytes of the code's dependencies. The compiler is aware of modules and can fetch them, so on a fresh workstation, "go test ./..." or "go install ./cmd/cool-thing" in a module-aware project works without running any other command first. It is actually so pleasant to use that I always think twice "do I want to deal with modules" before using a language like Javascript or Python, and usually decide "no".
npm and pip are the DARK AGES. That's why you're struggling. The community has been unable to fix these fundamental flaws for decades, despite trying in 100 incompatible ways. I personally have given up on the languages as a result. The standard library is never enough. You HAVE to solve the module problem on day 1.