Amusingly that links to https://peps.python.org/pep-0206/ from 14th July 2000 (25 years ago!) which, even back then, described the cgi package as "designed poorly and are now near-impossible to fix".

Looks like the https://github.com/jackrosenthal/legacy-cgi package provides a drop-in replacement for the standard library module.

There are certainly some suboptimal design choices in the cgi module's calling interface, things you did a much better job of in Django, but what made them "near-impossible to fix" was that at the time everyone reading and writing PEPs considered backwards compatibility to be not a bad thing, or even a mildly good thing, but an essential thing that was worth putting up with pain for. Fixing a badly designed interface is easy if you know what it should look like and aren't constrained by backwards compatibility.

As a side note, the most popular databases are getting only a tiny fraction of the available performance on current hardware. I wrote a couple of comments with more details about this a week ago: https://news.ycombinator.com/item?id=44408654

In the manycore world, Python's GIL makes some approaches to scaling across cores unavailable, though that is changing. But I don't think those are usually relevant to web server throughput, just (potentially) latency.

I haven't tried it yet but I do wonder about the feasibility of writing code in Python and then having an LLM transcode it to something like C, especially since I know C well enough to do what I want in that directly so I could check the resultant code by hand.

I've had much better luck with LLMs translating code from one language to another than with writing it from scratch.

I have bash CGI scripts too, though Shellshock and bash's general bug-proneness make me doubt that this was wise.

There are some advantages of having the CGI protocol implemented in a library. There are common input-handling bugs the library can avoid, it means that simple CGI programs can be really simple, and it lets you switch away from CGI when desired.

That said, XSS was a huge problem with really simple CGI programs, and an HTML output library that avoids that by default is more important than the input parsing—another thing absent from Python's standard library but done right by Django.

As mentioned elsewhere in the thread, the query parameter parsing is still in the Python standard library, just invoked differently.

That policy, and the heinous character assassination the PSF carried out against Tim Peters, mean I can no longer recommend in good conscience that anyone adopt Python.

But I also understand that the world is not perfect. We all need to prioritize all the time. As they write in the rationale: "The team has limited resources, reduced maintenance cost frees development time for other improvements". And the cgi module is apparently even unmaintained.

I guess a "batteries included" philosophy sooner or later is caught up by reality.

What do you mean by "character assassination" carried out against Tim Peters? Not anything in the linked article I presume?

https://www.theregister.com/2024/08/09/core_python_developer...

https://tim-one.github.io/psf/ban

https://chrismcdonough.substack.com/p/the-shameful-defenestr...

It does make me wonder whether Python is still the best choice for what I use it for, and whether I should be moving to something else.

So the remaining people periodically launch some deprecation PEPs or other bureaucratic things in order to give the appearance of active development.

As for prioritizing, I think the right choice is to deprioritize Python.

No, not everyone. I've been using Python as my primary language since 2000 (that's 1.5.2 days). It has been the least troublesome language that I work with, and I work with (or have worked with) a bunch (shell, perl, python, ruby, lua, tcl, c, objective-c, swift, java, javascript, groovy, go and probably others I'm forgetting).

Even all the complaints about the Python packaging ecosystem over the years... I just don't get it. Like, have you ever tried working with CPAN or Maven or Gradle or even, FFS, Ruby Gems/bundler? The Python virtual environment concept is easy to understand and pip mostly does its job just fine, and these days, uv makes all that even faster and easier.

Anywho, just dropping a contrarian comment here because maybe I'm part of the generally silent majority that is just able to use Python day in and day out to get their job done.

I have not yet had major problems with breaking changes, but they do happen more often than the used to and it makes me nervous.

I've used CPAN, Maven, gem, and bundler, so I'm also always a little puzzled when people complain about Python's packaging system. However, I've also used npm, so I can kind of understand it.

Python was great in 02000, but some of the things that made it great then are gone now. Stability was one of those; simplicity another; reasonable performance a third; but the biggest issue is really social rather than technical. And I feel like we have alternatives now that we didn't have then.

The maintenance burden of Python projects is just so much higher than it has any right to be. The language is neat, but not that neat. I think too often we think of performance as a sort of "tradeoff" for having a bad, unergonomic language, but that's not necessarily true. Plenty of languages have poor performance and are also a pain in the ass. We no longer live in a world where our options are C++ or scripting languages. We have mature environments with fantastic tooling. We have fast compilers with amazing error messages. We have great runtimes with more than adequate performance.

I do think there are some inherent tradeoffs in the space.

My main issue with prototyping as a concept is that it doesn't exist in most workplaces. Prototypes quickly devolve into applications. Discarding code is risky. Your best bet IMO is choosing a language that's ergonomic in the long run, because odds are you're in for the long run.

dotnet is another great choice because of the tooling and batteries included, although you do have to deal with a fairly slow compiler. Java is okay too, but Java is very restrictive and high-friction, which might not lend itself to prototyping.

In the world of scripting languages, ironically PHP is a decent choice. It has better progressive typing than Python and it's reasonably safe these days. We've sort of come full circle on PHP. The downside is that PHP programmers tend to throw everything in an array, especially when going fast. That hurts readability and the IDE a lot.

And then, of course, typescript and node. I don't like typescript. There's something about scripting languages with build steps that pisses me off. But, it's got a wide developer pool and it's not the worst language ever. Although there's a bit too much teeth-pulling IMO with typescript.

You guys are all really getting worked up over very little.

This is a bit like Apple firing Steve Jobs for wearing sneakers to work because it violates some dress code.

Also I used Python way before JS, and I still like JS's syntax better. Especially not using whitespace for scope, which makes even less sense in a scripting language since it's hard to type that into a REPL.

What Node.js had from the start was concurrency via asynchronous IO. And before Node.js was around to be JavaScript's async IO framework, there was a robust async IO framework for Python called Twisted. Node.js was influenced by Twisted[0], and this is particularly evident in the design of its Promise abstraction (which you had to use directly when it was first released, because JavaScript didn't add the async/await keywords until years later).

[0] https://nodejs.org/en/about

But that's beside the point. Performant web backends are way easier to deal with in NodeJS than in Python. I'm not comparing to Twisted because, even though it looks good, every Python backend I've ever seen was either plain Python or Django, which was also a mess compared to Express.

Not saying that Python is great, but Node is even worse.

Jupyter fixes the REPL problem, and it's a major advance in REPLs in a number of other ways, but it has real problems of its own.

I do not think JS got it right. Node did, by doing async, but the reason for that was that JS did not do threads! It was making a virtue of a deficiency.

I love whitespace for scope.

JS didn't do threads because threads are an error-prone way to write concurrent software. Crockford was a huge influence on its development in the early 02000s, and he had been at Electric Communities; he was part of the capabilities cabal, centered around Mark S. Miller, who literally wrote his dissertation on why you shouldn't use threads and how to structure async code comprehensibly. Promises came from his work, by way of Twisted. Unfortunately, a lot of that work didn't get into JS until well after Node had defined a lot of its APIs.

But this wasn't "making a virtue of a deficiency". JS was intentionally exploring a different approach to structuring concurrent software. You might argue that that approach didn't pay off, but it wasn't some kind of an accident.

TCL, which promoted the same approach for the same reasons long before Node, eventually added threading.

We clearly need some way to take advantage of manycore, but I'm still not convinced that threading with implicitly shared mutable state is the right default. It isn't even what the hardware implements! Every core has its own cache! It's a better fit to the hardware than a single giant single-threaded event loop is, and I think that accounts for its curent dominance, but there are a lot of other possibilities out there, like transactional memory, explicit asynchronous message-passing interfaces, or (similarly) lots of tiny single-threaded event loops like Erlang (or, maybe, like Web Workers).

Greenthreading like in Golang is even better cause you get the advantages of OS threads, but that requires a runtime, which is why Rust didn't do that. And I guess it's hard to implement, cause Java didn't have it until very recently.

Depends what you are doing, how you are doing it, and how careful you need to be with resources.

The article is about the fact that it is often OK even when done in a particularly inefficient way.

https://github.com/python/cpython/commits/3.12/Lib/cgi.py

Turns out most maintenance this thing received is the various attempts of removing it.

All that was in the cgi module was a few functions for parsing HTML form data.

As a side note, though, CGIHTTPRequestHandler is for launching CGI programs (perhaps written in Rust) from a Python web server, not for writing CGI programs in Python, which is what the cgi module is for. And CGIHTTPRequestHandler is slated for removal in Python 3.15.

The problem is gratuitous changes that break existing code, so you have to debug your code base and fix the new problems introduced by each new Python release. It's usually fairly straightforward and quick, but it means you can't ship the code to someone who has Python installed but doesn't know it (they're dependent on you for continued fixes), and you can't count on being able to run code you wrote yourself on an earlier Python version without a half-hour interruption to fix it. Which may break it on the older Python version.

The support for writing CGI programs in Python is in wsgiref.handlers.CGIHandler .

You can't carry everything to very long term horizons, especially for categories of "everything" whose user base is 2 people and 1 squirrel.

People who want otherwise should volunteer to maintain what they want kept.

{ Oh, and before anyone jumps on me, this is only an analogy as it relates to freshman moral philosophy courses, not an attempt by me to over-dramatize - that is more the fault of said courses trying to engage 18 year olds. :-) I'm mostly interested in the active-passive details of the pledge campaign. }

Secondly, you have to find a reliable maintainer or several.

A lot of people want stuff to be maintained indefinitely for them by unspecified "others".

Not updating the system is usually a solution to such problems.

At best there is a nginx or an API in front that acts a reverse proxy to clean-up/normalize the incoming requests and prevent directly exposing the service.

Example: banks, airlines, hospitals, air traffic controllers, electricity companies, etc

All critical services that nobody wants to touch, as it works +/-

a) make the system air gapped

b) pay a Python consulting company to back port security fixes

c) hire a Python core dev to do the system, directly

OOOOR, they can just update to Python 3.13 and migrate to the equivalent Python package that's not part of the core. For sure they already use other Python packages already.

We're making a mountain out of a molehill, also on behalf of places that have plenty of money to spend if push comes to shove.

Jython no longer works with basically any current Python libraries because it never made the leap to Python 3, and the Python community stigmatizes maintaining Python 2 compatibility in your libraries. This basically killed Jython, and from my point of view, Jython was one of the best things about Java.

EDIT: So, you get threads like this https://stackoverflow.com/questions/65651040/what-is-the-rec... and so on

That is very much a self-inflicted wound, though. If you insist on not using the standard packaging solution for the language, you have to own the complications of that.

Python has no standard packaging. They even deprecated and removed distutils (another terrible idea that caused a lot of busywork). The only way that python supports packages is via 3rd party external solutions.

I mean, I understand the desire to remove distutils. It sucked. It was the least Pythonic package in the whole Python standard library. But removing it was even worse, because it means you can't use old versions of most Python libraries with recent versions of Python.

Personally… I don't use pip. Why? apt is there.

Also I'm part of the python team in debian so I can package what's missing or update out of date things if I need.

Lua barely has any stdlib to speak of, most notably in terms of OS interfaces. I'm not even talking about chmod or sockets; there's no setenv or readdir.

You have to install C modules for any of that, which kinda kills it for having a simple language for CGI or scripting.

Don't get me wrong, I love Lua, but you won't get far without scaffolding.

But my concern is mostly not about needing to bring my own batteries; it's about instability of interfaces resulting from evaporating batteries.

LuaJIT, release-wise, has been stuck in a very weird spot for a long time, before officially announcing it's now a "rolling release" - which was making a lot of package maintainers anxious about shipping newer versions.

It also seems like it's going to be forever stuck on the 5.1 revision of the language, while continuing to pick a few cherries from 5.2 and 5.3. It's nice to have a "boring" language, but most distros (certainly Alpine, Debian, NixOS) just ship each release branch between 5.1 and 5.4 anyway. No "whatever was master 3 years ago" lottery.

That said these days I'd rather use Go.

Admittedly Python is not great at this either (reload has interacted buggily with isinstance since the beginning), but it does attempt it.

I agree its not a rapid prototyping kind of language. AI assistance can help though.

Your app could add almost no latency beyond storage if you try.

Since I learnt Python starting in version 1.6, it has mostly been for OS scripting stuff.

Too many hard learnt lessons with using Tcl in Apache and IIS modules, continuously rewriting modules in C, back in 1999 - 2003.

edit: Looks like yes for Node JS. I can't tell for PHP as I keep getting results for optcache which is different and in memory.

I still miss PHP's simple deployment, execution and parallelization model, in these over-engineered asyncy JavaScripty days.

Also lets see the impact of Microsoft's Python team layoffs on it, given that CPython developers only started caring about performance due to Facebook and Microsoft, so far the JITs in Python have been largely ignored by the community.

At the time Perl was the thing I used in the way I use Python now. I spent a couple of years after that working on a mod_perl codebase using an in-house ORM. I still occasionally reach for Perl for shell one-liners. So, it's not that I haven't considered it.

Lua is in a sense absolutely stable unless your C compiler changes under it, because projects just bundle whatever version of Lua they use. That's because new versions of Lua don't attempt backwards compatibility at all. But there isn't the kind of public shaming problem that the Python community has where people criticize you for using an old version.

JS is mostly very good at backwards compatibility, retaining compatibility with even very bad ideas like dynamically-typed `with` statements. I don't know if that will continue; browser vendors also seem to think that backwards compatibility with boring technology like FTP is harmful.

- `perl -de 0` provides a REPL. With a readline wrapper, it gives you history and command editing. (I use comint-mode forn this, but there are other alternatives.)

- syscalls can automatically raise exceptions if you `use autodie`.

Why is this not the default? Because Perl maintainers value backward compatible. Improvements will always sit behind a line of config, preventing your scripts from breaking if you accidentally rely on functionality that later turns out to be a mistake.

https://entropicthoughts.com/you-want-technology-with-warts

https://entropicthoughts.com/why-perl

Perl feels clumsy and bug-prone to me these days. I do miss things like autovivification from time to time, but it's definitely bug-prone, and there are a lot of DWIM features in Perl that usually do the wrong thing, and then I waste time debugging a bug that would have been automatically detected in Python. If the default Python traceback doesn't make the problem obvious, I use cgitb.enable(format='text') to get a verbose stack dump, which does. cgitb is being removed from the Python standard library, though, because the maintainers don't know it can do that.

Three years ago, a friend told me that a Perl CGI script I wrote last millennium was broken: http://canonical.org/~kragen/sw/rfc-index.cgi. I hadn't looked at the code in, I think, 20 years. I forget what the problem was, but in half an hour I fixed it and updated its parser to be able to use the updated format IETF uses for its source file. I was surprised that it was so easy, because I was worse at writing maintainable code then.

Maybe we could do a better job of designing a prototyping language today than Larry did in 01994, though? We have an additional 31 years of experience with Perl, Python, JS, Lua, Java, C#, R, Excel, Haskell, OCaml, TensorFlow, Tcl, Groovy, and HTML to draw lessons from.

One benefit Perl had that I think not many of the other languages do was being designed by a linguist. That makes it different -- hard to understand at first glance -- but also unusually suitable for prototyping.

Basically something like "use strict" in JS.

For now, the relevant committees think some more experimentation and deprecation needs to happen before locking in the set of features to be considered modern.

Come to think of it, that's a nice extensibility scheme, too - whenever you want to update it, just add another "very". ~

Only one benchmark on one system, but over in day before yesterday's HN thread on this (https://news.ycombinator.com/item?id=44464272), I report a rather significant slowdown in Perl start up overhead: https://news.ycombinator.com/item?id=44467268 . Of course, at least for me, Python3 is worse than Python2 by an even larger factor and Python2 worse than Perl today by an even larger factor.

FWIW, in Nim, you can get a CGI that probably runs faster than the Go of this article with simply:

    import std/cgi                  # By default both ..
    for (key, val) in decodeData(): #.. $QUERY_STRING & POST
      if key == "something":
         do_something(val)

Python has Werkzeug, Flask, or at the heavier end Django. With Werkzeug, you can translate your CGI business logic one small step at a time - it's pretty close to speaking raw HTTP, but has optional components like a router or debugger.

then that endpoint will have at least 400ms response times, not great

If I remember correctly that is about half of what StackExchange served on daily average over 8 servers. I am sure using Go or Crystal would have scale this at least 10x if not 20x.

The problem I see is that memory cost isn't dropping which means somewhere along the graph the memory cost per process together will outweight whatever advantage this has.

Still, sounds like a fun thing to do. At least for those of us who lived through CGI-Bin and Perl era.

I agree that tens of milliseconds of latency is significant to the user experience, but it's not always the single most important consideration. My ping time to news.ycombinator.com is 162–164ms because I'm in Argentina, and I do unfortunately regularly have the experience of web page loads taking 10 seconds or more because of client-side JS.

[1] https://totalrealreturns.com

I agree that the number I gave is erroneous, but not in the direction you imply. 400ms is extremely conservative, and it's not just starting an executable. I took my 100-millisecond estimate for the time to start up Python and `import cgi`, which loads 126 modules in the version I have here on my cellphone, and multiplied it by a safety factor of 4. Even on my cellphone it doesn't take that long, although `python3 -m cgi` does, because it loads all those modules and then runs a debug dump script. The time I measured for `python3 -m cgi` on my laptop is 90–150ms.

If all you are doing is starting a per-request executable, that will typically take more like 1 millisecond than 100 milliseconds.

Perhaps you meant to suggest that the actual logic triggered by the request would increase this 400ms number significantly.

Consider first the case where it would, for example because you do 200ms of computation in response to each request. In this case the best you could do by switching from CGI to a more efficient mechanism like FastCGI is a 3× increase in req/s. If that allows you to increase from 10 million requests per day on one server to 30 million, that could be worthwhile if you actually have tens of millions of requests per day. But it's kind of a marginal improvement; it saves you from needing two or three servers, probably.

Now consider the case—far more common, I think—where your computation per request is small compared to our hypothetical 400 ms Python CGI overhead. Maybe you do 20ms of computation: 80 million machine instructions or so, enough to tie an IBM PC up for a few minutes. In such a case, the total one-core request time has gone from 400ms to 420ms, so 400ms is a very good estimate, and your criticism is invalid.

Even on HN#1, you get like two requests per second for maybe 18 hours. That's three orders of magnitude below 200M/day. Shifting the argument to say that no hobby project needs this is easy. PHP being in common use already proves that starting up and tearing down all context is fine for >99% of websites. But the context we're in is an article saying one can do 200M requests per day with CGI-bin, not whether 99% of sites need that. CGI-bin is simply wasteful if the process takes 400ms before it starts doing useful work, not to mention a noticeable amount of lag for users. (Thankfully TFA specifies they're not doing that but are speaking of compiled languages)

> your computation per request is [commonly] small compared to our hypothetical 400 ms Python CGI overhead. Maybe [that makes it go] from 400ms to 420ms

At "only" 420ms total request time, I'd still want to take that prototype code out of this situation. Might be as easy as wrapping the code so that the former process entrypoint (__main__ in python) gets, instead, called by some webserver within the same process (Flask as the first python example that comes to mind). Going from 420ms request handling time to slightly over 20ms seems rather worth an hour of development time to me if you have this sort of traffic volume

You can measure this easily, get a copy of Windows busybox and write a shell script that forks off a process a few thousand times. The performance difference is stark.

Are you really saying that it takes 70ms on Microsoft Windows? I don't have an installed copy here to test.

Even if it does, that would still be about 15% of the time required for `python3 -m cgi`, so it seems unlikely to be an overriding concern for CGI programs written in Python, at least on manycore servers serving less than tens of millions of hits per day. Or does it also fail to scale across cores?

  $ cat winforktest.sh
  #!/home/busybox-1.35 sh

  i=1; while [ $i -lt 10000 ]; do (echo $i); i=$((i+1)); done

  $  cat /etc/redhat-release
  Red Hat Enterprise Linux release 8.10 (Ootpa)

  $ time ./winforktest.sh > /dev/null
  real    0m0.792s
  user    0m0.490s
  sys     0m0.361s

  c:\users\x>ver

  Microsoft Windows [Version 10.0.22631.5472]

  c:\users\x>busybox | busybox head -2
  BusyBox v1.37.0-FRP-5236-g7dff7f376 (2023-12-06 10:31:32 GMT)
  (mingw64-gcc 13.2.1-5.fc39; mingw64-crt 11.0.0-2.fc39; glob)

  c:\users\x>busybox sh

  ~ $ time ./winforktest > /dev/null
  real    3m 44.44s
  user    0m 0.32s
  sys     0m 4.90s

I understand that WSLv1 presents a more efficient fork(), which is not really available to shims like Cygwin or mingw.

So, that does 19999 fork()s in 224 seconds, which works out to about 11 milliseconds per fork(). (On my Linux box, it's actually doing clone(), and also does a bunch of wait4(), rt_sigaction(), exit_group(), etc., but let's assume the fork() is the bottleneck.)

This is pretty slow, but it's still about 6× faster than the 70 milliseconds I had inferred from your "100× slower than any other POSIX".

Also note that your Red Hat machine is evidently forking in 39μs, which is several times faster than I've ever seen Linux fork.

...and then you're wasting a 64-core server at 100% CPU load on just starting up and tearing down script instances, not yet doing any useful work. This is doing 160 startups per second, not requests per second

Would be a pretty big program for it to require 400ms on a fast CPU though, but the python interpreter is big and if you have one slow import it's probably already not far off

why lua?

Shortly after that, we had a lengthy thread about "how Lua compares to Python": https://news.ycombinator.com/item?id=42655158

Two years ago I commented https://news.ycombinator.com/item?id=38862372 listing Lua's worst flaws from my point of view.

haha, keep writing, you're good