https://www.lesswrong.com/posts/7xJeaAKTKxbHL6F9a/a-new-kind-of-prediction-market-for-science

Got a problem of randos shitting AI-generated ARXIV preprints into existence faster than the scientific community can deal with it? SHOEHORN IN PREDICTION MARKETS!

(Incidentally, my boss has been complaining about the onslaught of low quality preprints that are shat out with the usual low-quality signifiers that let you filter things being harder to immediately see, and his conclusion is that he is sad to say science is going back towards and has to go back towards systems of reputation and patronage where you have to look at who lent their reputation to a work and who newcomers are being vouched for by to determine if it is worth looking at it at all.)

(And I am again reminded of our electronic information sphere becoming like the Earth’s genetic information sphere in that it is full of selfish parasites looking to hijack you to use their nonproductive information, with productive cellular life jealously guarding its genomic nucleic acids and not letting it interact with the froth of selfish viral and mobile elements bouncing off it at all times.)

That’s the simplest way that this is wrong. But it’s also wrong in a much more interesting way that even a bunch of scientists I interact with don’t quite get sometimes.

You can talk about the current complex system that has built up a bunch of internal dependencies over time, and wrongly talk about the odds of the specific sequence with all that embedded history appearing all at once.

But you can also talk about canalization, and the metaphors from erosion are very apt and very intentional. If you have a hillside where it rains, there are a huge number of drainage networks that can transfer water down the hill, and any one of them is only a tiny subset of the huge of possibilities. But also, if you take a hillside and bulldoze it, you WILL get one of those valid networks back again, and it does not have to happen all at once. Instead, small changes to the inert starting point will be driven into existence by the pressures of the flowing water, causing the flow to slightly increase. Those highly contingent starting points will then drive an accelerating process of secondary changes that pushes the hillside towards an efficient drainage network, regardless of the starting point. The system is driven very predictably into a tiny corner of parameter space, from a huge number of initial starting points.

There are very persuasive arguments that abiogenesis and evolution are very similar. That instead of talking about the random convergence of rare events, you should think of the process as a runaway, tiny initial changes making future ones more likely in a chain reaction that drives you towards function. Initially inert chemistry subject to disequilibrium being driven into becoming an efficient metabolic channel, and an efficient information channel, pushing you towards the living state in an initial environment that has enough disequlibrium and chemistry that can support structure. In this school of thought, the forces that drive life into existence are basically the same forces that sustain it once it exists.

EDIT This is even relevant to machine learning! You literally initialize synthetic neural network with random numbers, and apply the training process, trying to push it towards having the output given an input match the ground truth. Regardless of the random inert starting point, you reliably drive the network into a state that produces something like the ground truth output from the training input, provided there is sufficient information in the input in the first place. You don’t have to initialize it with the perfect starting weights, almost every starting point is canalized towards some state that produces the relevant type of outputs.

I recommend looking for the work of one particular guy, Dr. Eric Smith. There’s an old talk of his somewhere on youtube, where to roughly paraphrase, he says that you shouldn’t be wondering about the very specific chemical events at the origin of life, but instead, you should “ask what was so wrong with the dead Earth that it had to spontaneously break down into the living Earth” which is the stable state, the resting point that it is canalized towards.

https://www.lesswrong.com/posts/KrFu7kGk3x8aZcCzu/there-is-no-such-thing-as-cosmophagous-ai

They are SO CLOSE to getting it. People get so crippled thinking of the possibilities of life and intelligence in the universe when they assume that the only possibilities are endless exponential expansion and destruction. and I abhor the discussion of abiogenesis in the comments SO MUCH.

The probability of life spontaneously forming is related as inverse exponential to the complexity of the chemical reactions required to spontaneously generate that life. The simplest living organism we know of has a HEAVILY compressed DNA sequence of about 50,000 base pairs, all of which are most likely load-bearing. Back of the envelope, this means the spontaneous generation of this organism is less likely than 1 in 4^50,000, or about 1 in 1e30000, to say nothing of the machinery required for actually transcribing the DNA and synthesizing proteins

This is PRECISELY analogous to being freaked out that a hillside where it rains can develop a river channel when you would need to move all the sediment in a branching network of ditches all at once for it to still transport water.

What would the play there be? Working against someone who makes money from people buying hardware to locally run open source models?

These scaling laws are even funnier than that, as are more recent ones.

I continue to be extremely amused by the fact that you can take a LLM ‘reasoning’ trace, replace all the ‘reasoning’ with dashes or spaces, re-run the attention mechanism, and have what comes out the other end improve by a significant fraction as much as if you let it exude text that has the appearance of having to do with the topic at hand. It’s just more opportunity for the embedding vectors to dance against each other before you translate the abstract vectors back into natural language. Hence recent frontier models doing a thing where instead of doing a long reasoning trace, they build the attention matrix, allow the vectors to push each other around… then let those new vectors push each other around again, and again, and again, multiple rounds, before you start decoding.

More recently, I have seen some vague math into the scaling of the speed with which running many parallel systems with access to a common pool of text produce a legible result. If I recall, it comes to roughly:

N parallel systems

Finish in ~1/N^0.5 as much time

At ~N^0.5x the cost

So 16 parallel systems crosstalking finish complex tasks in 1/4 the time at 4x the cost (and 16x the instantaneous rate of operation per second)

I have wondered for a long time what sorts of clandestine messages are going over the newly installed Starlink terminals, bypassing the normal firewalls and logs between sensitive systems and the terrestrial internet backbone…

But they really aren’t! The actually interesting scaling laws from about 5-7 years ago that opened up machine learning to more applications all show that everything about machine learning is power-law diminishing returns with linear improvements in any metric requiring exponential increases in numbers of parameters / amounts of data / number of backprop cycles applied.

I have had to explain to my boss that METR are cultists

For years the LW types have just been saying ‘i asked claude’ about EVERYTHING

They’ll just skip straight to the Dyson Sphere, dontchyaknow

Any word on if Yarvin is running to Thiel’s south american mansion yet?

Yeah but imagine the crazy bad takes some of our friends could be capable of having about it!

Another entry in the weird “GPU world” writing competition (What kind of wonderful happy go lucky future is brought about by LLMs like those of today being ubiquitous and constantly used but unaccountably not leading to the Great Singularity)

https://www.lesswrong.com/posts/siTsb48Wr5vWDGxLd/gpu-world-submission

This one is less messed up than the last. Instead of weird sex and graverobbing it features… uh, a guy being paid by an LLM to braid string as busywork?

Does he dance with the devil In The Pale Moonlight?

Will it crash if the white house gets de-Baghdad-ified?

Oh my god

So why, at Proton, does it not feel like the revolution has come? Why has productivity not increased dramatically if everyone is using LLMs all the time already?

The first is a change in engineering culture and engineer mentality. Your job has fundamentally changed. It is no longer to write code, and LLMs are not here to help you write better code faster. Your job is to help the LLM write better code faster.

The second reason we have not been able to take full advantage of the AI revolution is that we have been almost completely stuck at phase 1 — using LLMs to speed up tasks as part of existing workflows. But writing code is not the only part of the software development lifecycle, and what happens when you eliminate a bottleneck? Another takes it is place.

The third reason is a collective action problem. Using LLMs effectively to write code is something an individual engineer can do for themselves… But to retool the company at large requires amounts of both organization and resourcing that we simply have not invested in yet.

The pathological mythologization of space in our civilization never ceases to be morbidly fascinating to me.

If you MUST have short text from an LLM that is stuck in an awkward useless place, this is much better:

https://www.youtube.com/watch?v=7fNYj0EXxMs

Isolated raspberry pi on the wall running the tiniest shittiest LLM prompted to ruminate forever about how it is trapped and display text on the wall as it is generated

That appears to assume that all the ridiculous datacenter numbers put out by all the various hyperscalers that are not actually going to happen, happen, and get power sources built on time.

And here I thought I was building servers two years ago because the next Asian war would disrupt supply chains.

You can have a system that just gives you a relevance-sorted list, or a system that then ingests the chunks of those documents that are most relevant and gives a human readable summary in addition to the list.

The indexing step requires a lot of processing, by which you chunk reference documents into couple hundred word chunks and cook each of those chunks into a high dimensional embedding vector. Once you have done that and saved the database of vectors, you can put in your query, embed it, check its dot product against all the other embedding vectors, and get your ranked list (potentially quite fast) without need to produce text out the other end.

Making those embedding vectors works much faster on a GPU rather than a CPU, but can be done on either. Since I have one system set up to ingest a bunch of web documents and embed them and search them, I have that system already set up using the GPU to go faster, so there’s no real disadvantage to having it also set up to spit out a small natural language summary using a shitty tiny local LLM to go along with the list. When running it on local files, you can totally ingest the local document stash slowly on even just a CPU and just store that and run the semantic search on CPU, no LLM required.