Pi is just a nice base and it has defined extension protocols and such. You might as well start there, it's just easier and going from nothing to working to adding whatever functionality is like 2 minutes.

> AlphaZero (which later became Leela? I'm not sure)

Pretty much. DeepMind never released their code or models, just the main ideas of their techniques and training.

Leela, which came out of the go-ai world, implemented the paper(s) and became the de-facto version everyone used for a while until other projects improved over it on both go (mainly katago) and chess (mainly stockfish).

Yes. The ways to improve are: faster search (which lets you search more deeply) or better evaluation of a position or better heuristics.

Doing this in a way that doesn't get you noticed or fucked with is potentially going to be quite difficult.

If they have a "hey we're being benchmarked" mode, which is not hard to imagine, avoiding tripping it is going to be annoying and difficult to prove.

You would want to use your own benchmark, certainly, not something publicly known. But outside of that I don't think "this is a benchmark" is such an easy category to determine -- a benchmark should be similar to a typical scenario anyway.

> "this is a benchmark" is such an easy category to determine

I mean, it's not _that_ hard to determine most likely, and/or it's hard to be sure you didn't get found out by llm-assisted analysis on your traffic. It's not going to be a one-shot request and response it's going to be a whole bunch of them in an artificial way, by nature.

And then anything you found is single-use only if you're paranoid because even if they find out later, they have your benchmark now (because you sent it to them to use it even if you don't publish it).

> The fact that they didn't understand this when they proposed the idea shows you how little they knew of the Linux ecosystem, and the fact that they're still bringing up so many years later shows that they haven't learnt anything even after all this time.

The fact that they didn't ask "why?" at the time is also telling (or they didn't listen or remember, whatever reason it's not in this anecdote). They expected the greybeards who had been maintaining this shit for years to shut up and accept whatever shitty proof-of-concept looked good in a screenshot.

> It's surprising it's that high (8%). I would imagine it takes as long to mop a floor that has pee droplets or mop a floor that doesn’t.

You might have to mop more often from complaints or inspection, maybe?

I don't think so, in every public place I've been they have a schedule and need to sign they did the cleaning at that time they had in the schedule.

Way back when I worked in a restaurant, there was that but then also if a customer complains they also go clean.

They're never going to be like "Oh, someone shat on the floor? We'll deal with that at the next scheduled cleaning."

> AI slop nothing burger. The “exploit” has nothing to do with coding agents.

It's pretty small potatoes, but it is a harness ~bug that they treat this so poorly. It getting triggered before some of them even ask you if you trust the directory is pretty bad.

> That humans can make sufficiently strong calculators has never been a dispute in my mind.

Within our lifetimes (unless you're quite young) it was doubtful if a go ai would ever beat a decent human. Same was true for chess a generation or two earlier. It's news because it's the handoff of man to machine being the best at a particular thing.

This current news is news from the other way, this human did _exceptionally_ well.

>It's news because it's the handoff of man to machine being the best at a particular thing.

That's a way to see it. Another way to see it is that for any given event in a zero sum game, one of two outcomes, where two players are involved, will occur. Man only had to beat machines for what, 200-400 years only for it to beat man how many times for it to be "best"?

The headline is misleading, but this is still huge. 2 stones against katago is insane, I'd never have guessed we'd see that, ever.

Annoyingly misleading. I read it that the human player was handicapped.

Like many random headlines they may be targeted at people who know something about a topic.

If you know anything about go or computer go from last 10 years then it is obvious which direction the handicap goes.

> Also, the human played a strategy tailored to that huge initial advantage. He said that the AI did not handle this particularly well, and played high probability moves instead of trying to lure him into a mistake.

Yeah, katago's training is not really focused at all on handicap games, because it's by nature learning from even games against similar-strength opponents.

It doesn't have specific training from playing in a way to exploit a weaker player. In a handicap game you have to give your opponent opportunities to fuck up if you want to play optimally.

If a move loses 0.0005 points if the opponent plays optimally, katago won't play it even if there's ~zero chance a weaker player would play it right.

There have been go AIs that tried to train more directly on uneven opponents, one called "sai" comes to mind, but katago has huge advantages otherwise and won out over the others (for very good reason, it's a great project).

While AlphaGo originally only had win rate as a metric, modern Go AIs have more knobs, including an evaluation of "complexity".

Just stating this off the top of my head so I could be misremembering, but I heard that the KataGo settings used were tweaked to favor complexity. This was most apparent in Game 1 which Shin Jinseo lost, where the AI had an unusual opening. However, the last game was quite plain leading me to wonder whether that setting was present in the last game (or at all).

There are tweaks to move choice like that, but it's all evaluation time. None of that happens in training. Zero times in the RL loop does katago see a two stone game against a weaker player.

You can kind of tweak towards play this metric or that, but it's not the same.

Maybe not to the extent you mentioned earlier (I definitely agree that it's not "focused" on this) but there is definitely asymmetric playout training in the training data (see comments on lightvector/KataGo issues #39 and #162 mentioning it for example), that is presumably how they got the tweak for playoutDoublingAdvantage (i.e. actually having a few thousand of the millions upon millions of training games be games where playouts have been doubled for advantage).

Yeah, thanks, I forgot that existed. That definitely weakens my point quite a bit.

It's still not _quite_ the same thing because an advantage in playouts is not a great model for how a stronger/weaker player dynamic actually works, but it is something for sure, way more than the nothing that I said exists.

Ah this is interesting. Essentially the idea is that the compute can try and move into positions that it can evaluate but humans might have trouble evaluating because of the board state's complexity?

That's the basic idea, yeah.

When playing white in handicap games, you want to make your opponent uncomfortable.

Play moves where the simple/safe/obvious move is just a little bit bad. Force them to choose between complex fights or a slow death of 100 slightly suboptimal moves.

It feels really wrong to defend like 10 times in a row, so if you make them do that they'll lash out at the wrong time and you can take advantage.

You also want to look for moves where...even if their best response means it's even or a little bit worse for you, there's ~reasonable responses where you win out or it goes complex.

A lot of the time it's not even crazy complex fights, it's more just situations where the judgement of what is more points is difficult.

(Note: most of this stops applying as strongly if it's a teaching game, which most handicap games are, there you have other considerations besides winning)

By the way that is exactly what humans do when playing with white in high stone handicap games. They place their stones all around the board, start little fights everywhere and wait for the weaker player to misread or misevaluate something. Suddenly two or three fights merge in a one sided larger one and part of the handicap is gone.