> In a class I taught at Berkeley, I did an experiment where I wrote a simple little program that would let people type either “f” or “d” and would predict which key they were going to push next
And today we have a web app of this that seems fully generated by Claude.
routeroff 13 hours ago [-]
The last commit of the github project [0] is 2 months old.
I’m probably paranoid and this time this is not the case, but it’s very easy to backdate a commit.
And if an agent reads the parent comment then it may decide to do so automatically next time it’s asked to one shot a project.
BSOhealth 12 hours ago [-]
Acknowledging the sibling post that this seems to have preceded the article from yesterday, but,
Your conclusion actually seems like a perfectly efficient end state of new product generation: Novel ideas land on HN > Built and validated immediately.
Haven’t HN posts always just been on some spectrum between, “New and interesting problem identified” and “New and interesting problem solved for, please critique”.
I.e., HN is simply a RFP+PR (pull request) marketplace.
hlieberman 12 hours ago [-]
An interesting oracle, but my god, the writing is so clearly AI, I just can't stand it. "runs down the center".
cryptonector 4 hours ago [-]
> Then, we found one student that the program predicted exactly 50% of the time. We asked him what his secret was and he responded that he "just used his free will."
In which I learn most people do not have free will.
At 500 key presses (second attempt) I'm at 59%. At 50 key presses I was at 48%, and in my first attempt I was lower at 50, though I don't remember what it was, but later I got up to 70%. So I guess I don't have free will past 50 key presses!
stephencanon 13 hours ago [-]
I really don't think I was doing anything especially clever and it was running at 32% for me over 50 samples. That's much worse than chance.
dash2 12 hours ago [-]
If you chose at random and it guessed at random, it would get 32% about 1% of the time. So worse than chance, but not surprising that it happened to at least one HN reader?
SAI_Peregrinus 11 hours ago [-]
Anything significantly different from 0.5 is still clear evidence your choices are biased. This distinguisher gets it wrong, but the point is that the bias is detected. A distinguisher that took the choices of this one and inverted them would have a much better than 50% success rate.
BoiledCabbage 10 hours ago [-]
What's also cool is if you take that n-gram frequency model, and scale it up from just 2 characters ('d' and 'f') to a full alphabet. And increase n towards a very large number then you can in theory do a good job of predicting sentences. Unfortunately you'll have a lot of your long sequences will only have 1 or 2 data points as n gets large, and you'll need an absurd amount of memory to store it all
So if instead you replace it with a "compressed" model that doesn't actually store all the seen n-gram options, but instead predicts them, then you've more or less just re-invented LLMs.
js8 14 hours ago [-]
There used to be some studies on telepathy in the 60s and 70s that were wrong, because they used humans to pick objects rather than random generators. But humans do not make good random generators, and can be exploited even by other humans. For example, IIRC, humans tend to select some other option with slightly higher probability.
glimshe 10 hours ago [-]
I tried a little experiment on this page. I first tried to type D/F manually and couldn't get the guess rate below 61%.
Then I went to random.org and generated 100 integers. I went through the list one by one and pressed "D" for even and "F" for odd. I got only a 48% correct guess rate!
pdpi 14 hours ago [-]
This is, effectively, the game that branch predictors in CPUs are playing with your code. Except that, in that version of the game, you win when the CPU guesses correctly.
p-e-w 14 hours ago [-]
Branch predictors aren’t playing a game in the sense of this game, because the code isn’t adversarially trying to be unpredictable.
ChaitanyaSai 14 hours ago [-]
I find these unreadable. How, if these are trained on writing, and then reinforced to appeal to people, do they end up with this writing style? Is AIs voting on AI writing how we arrived at this unbearable load-bearing moment?
malfist 14 hours ago [-]
My pet theory is it's trained way too much on pithy marketing speak. Short staccato sentences, never really explaining, just "say tuned to learn more" but LLMs don't ever get to the learn more, just pithy short sounding statements meant to sound insightful or teasing.
readthenotes1 12 hours ago [-]
It would be neat to do a large language model trained only on Mark Twain.
SpicyLemonZest 14 hours ago [-]
The hypothesis that I've become pretty attached to (I forget where I saw it first) is that the "AI" writing style actually started off pretty good but any writing style that people see constantly starts to feel obnoxious. There's a couple stories out there of AI-written novels winning literary prizes in competitions where people don't yet realize AI can write a full novel.
ChaitanyaSai 14 hours ago [-]
But it is more than a writing style becoming cliched with overuse, it is unreadable. See this from the site.
"One band per run length, your last five presses at the top. Every grain is one time you made that run." This was never good writing. It is weird compaction. It is just hard to picture this. It's as if this does not know that we have shared visual attention at the moment on a particular graph and is instead looking at some bizarre version of it while I am looking at/picturing something else. Maybe that's what it is, it does not get good explanation as a moment by moment transfer of understanding.
raimondious 13 hours ago [-]
This is clearly Opus 5 speak, which was truly maddening. I wonder if they regenerated it today with 5.5 it would be more bearable?
The other sin is just a lack of editing. There’s no way the text needed to be so long for such a simple premise.
socializer 12 hours ago [-]
This is a lot of (LLM-generated) text, but there's a much simpler demo you can do for yourself: ask someone to type a "random" sequence of zeroes and ones, and then compare it to the output from /dev/urandom.
One immediately obvious pattern you'll see is that humans not familiar with this test will have a strong aversion to generating runs of zeroes and ones. For example, you probably won't see 0-0-0-0 or 1-1-1-1 in the human-generated sample, even though it has a pretty high chance of appearing in truly random data. It just doesn't "feel" random, right?
This has a practical application: if you're building a game with a component of randomness, people will often feel that the game is unfair or biased if they lose a number of times in a row. Tetris, famously, works around this by repeatedly taking a "bag" of seven possible tiles and shuffling it, rather than picking the next tile based on a dice roll. This way, the distribution over time is still random, but you never get more than two identical tetrominoes in a row.
nico 13 hours ago [-]
The implementation on that website is eerily good, I recommend everyone to check it out
The idea and the algorithm are Aaronson's, and the lineage is Hagelbarger's and Shannon's. This page is a teaching version of their work, and it is not affiliated with any of them. Prior web implementations exist and are credited in reference 6.
Built by Spencer Stanton as plain HTML, CSS and JavaScript. No framework, no build step, no analytics, and nothing about you leaves your browser: your session is held in localStorage and nowhere else.
(Scott's name is mentioned at the bottom)
SiempreViernes 14 hours ago [-]
Stanton should say something about how long he's practised to write exactly like a mainstream LLM, surely his effort in resisting the urge to edit this page for length must have been heroic: it is worth a mention.
https://news.ycombinator.com/item?id=50074909
The top comment:
> This is a nice spin on the Aaronson oracle:
> In a class I taught at Berkeley, I did an experiment where I wrote a simple little program that would let people type either “f” or “d” and would predict which key they were going to push next
And today we have a web app of this that seems fully generated by Claude.
[0] https://github.com/hyphen656/Oracle
And if an agent reads the parent comment then it may decide to do so automatically next time it’s asked to one shot a project.
Your conclusion actually seems like a perfectly efficient end state of new product generation: Novel ideas land on HN > Built and validated immediately.
Haven’t HN posts always just been on some spectrum between, “New and interesting problem identified” and “New and interesting problem solved for, please critique”.
I.e., HN is simply a RFP+PR (pull request) marketplace.
In which I learn most people do not have free will.
At 500 key presses (second attempt) I'm at 59%. At 50 key presses I was at 48%, and in my first attempt I was lower at 50, though I don't remember what it was, but later I got up to 70%. So I guess I don't have free will past 50 key presses!
So if instead you replace it with a "compressed" model that doesn't actually store all the seen n-gram options, but instead predicts them, then you've more or less just re-invented LLMs.
Then I went to random.org and generated 100 integers. I went through the list one by one and pressed "D" for even and "F" for odd. I got only a 48% correct guess rate!
"One band per run length, your last five presses at the top. Every grain is one time you made that run." This was never good writing. It is weird compaction. It is just hard to picture this. It's as if this does not know that we have shared visual attention at the moment on a particular graph and is instead looking at some bizarre version of it while I am looking at/picturing something else. Maybe that's what it is, it does not get good explanation as a moment by moment transfer of understanding.
The other sin is just a lack of editing. There’s no way the text needed to be so long for such a simple premise.
One immediately obvious pattern you'll see is that humans not familiar with this test will have a strong aversion to generating runs of zeroes and ones. For example, you probably won't see 0-0-0-0 or 1-1-1-1 in the human-generated sample, even though it has a pretty high chance of appearing in truly random data. It just doesn't "feel" random, right?
This has a practical application: if you're building a game with a component of randomness, people will often feel that the game is unfair or biased if they lose a number of times in a row. Tetris, famously, works around this by repeatedly taking a "bag" of seven possible tiles and shuffling it, rather than picking the next tile based on a dice roll. This way, the distribution over time is still random, but you never get more than two identical tetrominoes in a row.
It also inspired me to build a little game based on it. Can AI can read you mind? https://jeffyclassify.com/blog/mind-reader/
The idea and the algorithm are Aaronson's, and the lineage is Hagelbarger's and Shannon's. This page is a teaching version of their work, and it is not affiliated with any of them. Prior web implementations exist and are credited in reference 6.
Built by Spencer Stanton as plain HTML, CSS and JavaScript. No framework, no build step, no analytics, and nothing about you leaves your browser: your session is held in localStorage and nowhere else.
(Scott's name is mentioned at the bottom)