The Python logging module is one of the worst modules in the stdlib. You can see it’s Java inspiration all over it, as being one of the oldest modules it was created before anyone really knew what “pythonic” code was.
Things I really dislike about it is the lack of “context”. Usually log messages are about something - often a request. You can’t attach a request ID or something else a nested set of function calls that log.
The whole library is just really rubbish and has not aged well. There’s no standard way of outputting JSON logs, but there’s a built in way to spawn a socket server that allows the logging configuration to be updated remotely via a bespoke protocol? And everything is built mostly around logging to files.
It also hits that perfect sweet spot of being both overly engineered and totally inflexible. I wanted to funnel our log messages to s3 via the “s3fs” module, which supports a “file like” object that gets written to s3. I had to hack around the various handlers to support this because it assumes it’s a “real” file.
I've been writing Python daily for 15 years but when I have to do any configuration beyond `basicConfig()`, I still groan. In many instances I have wanted to do something like temporarily turn on logging of HTTP requests, which turns into a much more involved effort than it ought to be.
I just recently discovered that there's no documented way to dump the current configuration (https://stackoverflow.com/q/72624883) so you can see what loggers exist and where their output is going.
It's definitely sad that there isn't built-in context support, but if you're looking for a solution to this, it is actually very easy to attach context to logs using threadlocal and a logging filter.
When running a webserver though, an even simpler trick is to add middleware that sets the current request's info as the current thread's name, and then including threadName in your log format.
The contextual support is lacking but it can be done. See this comment from a previous discussion where I briefly described how I’ve used the record factory to help: https://news.ycombinator.com/item?id=31484527
Agreed. One of the most confusing libraries with a ton of magic going on behind the scenes. I also detest PyTest, although not a std lib, pytest is one of the worst in terms of explicitness. It’s very useful but that’s orthogonal. I’ve seen newcomers to python struggle to grasp how pytest just manipulates a billion things behind the scenes.
Overall, I prefer less magic and more explicitness.
I wish I could agree. Every aspect of pytest that I like (fixtures, plugins, etc.) is marred by a boneheaded move to make them autoloaded by default. Loading should be explicit/opt-in. It's dumv that if you pip install something that adds a new pytest plugin it'll automatically modify your runtime behavior. Some of the plugins are great (shoutout to pytest-socket), but it has the same issue flake8 has with the plugin ecosystem where the quality varies wildly.
I might be in the minority but I also find python's mock implementation awful. Mockito in Java is way better where you explicitly state which calls you expect and if you get an unexpected call it errors out.
The implicit magic makes writing pytest tests painless.
Also with improvement of type hinting, tooling like PyCharm starts to understand pytest better and make it easier to read and manage and a lot of the magic can be dispellt by written out explicitly. Though this need some discipline from the dev team.
PSL unittest is very explicit and has no magic. Ironically it also has Java written all over it. You must be the first person I met who prefer unittest to pytest.
camelCase is idiomatic in Smalltalk, which as a language and a community was a pretty big influence on other dynamic OO languages like Python and Ruby, and from whence SUnit comes, and thus the whole xUnit world, derives.
So, it's maybe not the best evidence that the Python version is particularly a clone of the Java port.
Pytest is very opinionated about how and when it's going to be run. If you want to put a test in say, Jupyter or a Markdown block, unittest is much more amenable to just doing what you want without having to read through documentation to break a whole bunch of magic behavior.
What do you mean? Pytest rules! Yes there is some magic, but if you let it take control, it is very convenient to write and run tests with it. Just... you need to trust it and it will do its job.
Strong agree on all points! The number of times I had to debug into the pytest source code because developers (the ones using pytest) thought it would be cool to use some decorator black magic to manipulate global state behind the scenes which pytest not just makes very easy but actually encourages and which was obviously going to break sooner or later…
And regarding the sibling comment: Yes, I prefer unittest as well.
>> You can see it’s Java inspiration all over it, as being one of the oldest modules it was created before anyone really knew what “pythonic” code was.
The irony is Java logging moved on and with newer libraries offers better supports for lazy evaluation, structured logging and multiple output formats, while Python logging seems stuck with an ancient copy of log4j 1.x
> there’s a built in way to spawn a socket server that allows the logging configuration to be updated remotely via a bespoke protocol
That sounds like there's a Log4shell-like vulnerability waiting to be found. But I couldn't even find it, I found the SocketHandler but that's just a log destination.
A few months ago I built a project and wanted to add logging. After searching around, I ended up selecting loguru and it definitely is a nicer user experience. But I also didn't really love it, for reasons that I don't recall now. I would look to it again, simply for lack of a better choice.
Sadly, my social media is subscribed to by pretty heavy hitters in the Python community, but when I asked about the best logging for python it was all crickets. Maybe that says something. :-)
One missing feature of loguru is filter by logging levels. You can still get around it by modifying an internal variable that you can specifically not to
It’s rock solid, at least. I’ve also had the displeasure of working in the logging package. Totally outdated. I created a JSON formatter (also support for context). Huge pain, but always works.
Also logging can be set up remotely, but it's easier to do like that from the container and then configure the hos to send everything to a single log machine.
You should try the loguru library. I was able to roll a rolling-upload-to-s3 adapter in under an hour. Switching to json logs is one bool flag away. Plus it's gorgeous
If you're talking about console.log() and console.error(), I don't think it's trying to be a logging system. The name of that first function is unfortunate but not fooling me.
I can't agree with you more. I switched to Python's logger off of my janky thing about 6 months ago, and I'm hating every second of it. I still have no clue how to turn it on, turn it off, get it to fork output (stdout, stderr, file), nor how to override its behavior.
It's literally easier to just have an IOBuffer laying around.
There are valid complaints in the grandparent, but if you can’t figure out how to fork output to stdout, stderr and file in six months, that’s really on you. You should know how to register multiple handlers if you spent five minutes reading the documentation.
true that. but I've never figured out a good way to do this and have a single test runner module log when invoked directly: python3 test_foo.py and indirectly python3 -m unittest test_foo
The Python logging module is one of the worst modules in the stdlib.
One has to remember that logging was never intended to be some infinitely generalizable metaprotocol for massively streaming parallel notifications to the cloud or whatever. It was meant as a replacement for "if (DEBUG > 4): print(...)". For which I think it does quite well, thank you.
The whole library is just really rubbish and has not aged well. Hate it.
So what have you written that has made people's (not just your bosses' or your clients') lives easier, for nearly 20 years now (if only incrementally)? Do share.
I see and grant your points, but I find this strong emotional rebuke of something that was never intended to be anything more than a simple convenience library to be well, strange.
It clearly is supposed to be more than a convenience library. It’s the blessed, stdlib way to do logging with Python. The module isn’t called “print_with_levels”.
So now everything needs to work with it, it’s internals and it’s way of doing things. And to top it off it’s near impossible to refactor or replace.
Things I really dislike about it is the lack of “context”. Usually log messages are about something - often a request. You can’t attach a request ID or something else a nested set of function calls that log.
The whole library is just really rubbish and has not aged well. There’s no standard way of outputting JSON logs, but there’s a built in way to spawn a socket server that allows the logging configuration to be updated remotely via a bespoke protocol? And everything is built mostly around logging to files.
It also hits that perfect sweet spot of being both overly engineered and totally inflexible. I wanted to funnel our log messages to s3 via the “s3fs” module, which supports a “file like” object that gets written to s3. I had to hack around the various handlers to support this because it assumes it’s a “real” file.
Hate it.