About a month ago I wrote a perf tool for my application to measure its network message speed. It showed I could send 3,250,000 WebSocket messages per second in my TypeScript application. If I moved each test of an experiment into its own call stack I could send 3,250,000 messages as fast as 0.002 seconds. Claude confirmed the number.
It was about two weeks later I learned the number is wrong. The number would be correct if this were just a linear instruction processing problem. Actually the processing time, CPU and network handling, is about 0.08%. The other 99.92% of processing time is in memory. What was happening is the perf tool would push messages into a queue on the socket from a JavaScript loop, but the speed to process these messages and drain the network buffer was less than the loops execution speed.
Correcting for that dropped my transmission speed to a more realistic 100,000 messages per second. Modified queue management then allowed a rate of 800,000 messages per second. Pushing each message send into a separate call stack then allowed for a rate of 6,000,000 messages per second on a single socket but it sacrifices stability to reach that rate of transmission speed.
My wrong number was confirmed by Claude for the same reason I believed it, a confirmation bias that was well reasoned from the logic. Claude came to confirm the number for the same reason I did but came to that conclusion a completely different way.
Seems like persistent models like OpenAI's highly persistent internal model can become really effective over time. Those are the ones that drove most of the HF-OAI incident.
This training technique does not relate to how persistent a model is, at all really. They sample more parallel attempts at hard problems, to increase their chances of having at least one success to learn from.
About a month ago I wrote a perf tool for my application to measure its network message speed. It showed I could send 3,250,000 WebSocket messages per second in my TypeScript application. If I moved each test of an experiment into its own call stack I could send 3,250,000 messages as fast as 0.002 seconds. Claude confirmed the number.
It was about two weeks later I learned the number is wrong. The number would be correct if this were just a linear instruction processing problem. Actually the processing time, CPU and network handling, is about 0.08%. The other 99.92% of processing time is in memory. What was happening is the perf tool would push messages into a queue on the socket from a JavaScript loop, but the speed to process these messages and drain the network buffer was less than the loops execution speed.
Correcting for that dropped my transmission speed to a more realistic 100,000 messages per second. Modified queue management then allowed a rate of 800,000 messages per second. Pushing each message send into a separate call stack then allowed for a rate of 6,000,000 messages per second on a single socket but it sacrifices stability to reach that rate of transmission speed.
My wrong number was confirmed by Claude for the same reason I believed it, a confirmation bias that was well reasoned from the logic. Claude came to confirm the number for the same reason I did but came to that conclusion a completely different way.
These links at the top point to the same page, not the paper or the code!
I've seen this happen a lot recently, when the page still has relatively few comments. Insightful comments are dead leaving behind trivial ones.
What's going on?