Rendered at 21:20:42 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
farfatched 21 minutes ago [-]
This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University.
> Why unify information theory and machine learning? Because they are
two sides of the same coin. In the 1960s, a single field, cybernetics, was
populated by information theorists, computer scientists, and neuroscientists,
all studying common problems. Information theory and machine learning still
belong together. Brains are the ultimate compression and communication
systems. And the state-of-the-art algorithms for both data compression and
error-correcting codes use the same tools as machine learning.
I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.
22 minutes ago [-]
cyanydeez 46 minutes ago [-]
it was vaguely in my understanding of information & intelligence with compression; it was also brought up in several of the initial trials against AI companies where they discussed how the AI is akin to compression.
So they're both sourcing a bit broader zeitgeist.
ssivark 52 minutes ago [-]
Nope; there is a bit more nuance and the distinction is important.
Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support! Eg: you observe a rare edge case in your training data and (lossy) compression could simply ignore it. But if you wanted generalization in that particular part of the space -- either because an adversary was testing you, or for design freedom where you choose to build in that specific corner -- then you don't just want data compression, but good prediction performance on a test distribution which peaks in that corner.
Assuming that the training data distribution is exactly the distribution you will ever care for is implicitly doing a lot of the heavy lifting in the claim that compression = prediction, and I'm peeved at how much this statement is unthinkingly repeated like a manifesto.
There is nothing natural about the training data distribution, especially if the data generation process is exploratory while the downstream usage will be exploitative.
porphyra 9 minutes ago [-]
How does that invalidate the "compression is prediction"? If the future data is different and you failed to generalize, then the failure to predict means you got worse at compressing and have to spend more bits storing the new information. Conversely, if the future data is the same as that you've seen previously, you could predict it very well, and compress better as a result.
gr_norm 2 minutes ago [-]
A maximally efficient compressor for the existing data distribution is not in general (and often will not be) maximally efficient for future data. The former may only be enabled by convenient local optima of the input distribution that a compressor accounting for the latter could not take advantage of.
tcgv 38 seconds ago [-]
Compression can be prediction would be more accurate.
jbs789 23 minutes ago [-]
That’s interesting.
Also sparked the thought that the assumption only holds if the future looks like the present.
schopra909 24 minutes ago [-]
100% agreed.
woliveirajr 23 minutes ago [-]
There is Compression done by Prediction by partial matching [0]
There is the Kolmogorov Complexity [1], Normalized Information Distance [2] and Normalized compression distance [3] that correlates those.
Finally, there's the Pre-Big Bang Informational Compression and the Delayed Release of Antimatter [4]
All big {rabbit/black} holes to lose some time, if you have any.
I stumbled across a connection between LLMs and compression when researching N-dim polytope emergence in neural networks. Toy Models of Superposition (Anthropic, 2022) suggests that gradient descent can independently discover efficient geometric packing arrangements for sparse features. LVQ compression uses regular lattice structures, including some based on 4D lattices.
I found this interesting and wonder whether LLMs have a higher density ceiling, since training and inference don't rely on a fixed lattice and can instead learn their own representational geometry.
throwaway_7274 1 hours ago [-]
This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!
nonameiguess 2 minutes ago [-]
There's another element to this that I almost never see discussed. Ideas are not facts. Neither LLMs nor humans can generate new knowledge, as opposed to ideas, by thinking alone. Physical investigation and experimentation is necessary.
The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.
glial 50 minutes ago [-]
> it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process
This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit?
Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.
throwaway_7274 5 minutes ago [-]
Wow, I didn’t even notice how badly I underexplained that! Yeah, thanks, your edit is what I had in mind. jbay808 explains it well, too.
redhed 38 minutes ago [-]
How I see it, is if the human brain does lossy compression/prediction of the natural world that learns from its "training set" (sensory inputs) and we have been able to come up with new ideas, then it seems like AI would be able to as well.
jbay808 18 minutes ago [-]
Imagine you're curve-fitting a bunch of data points on, say, the orbital motion of planets and asteroids. You get tons and tons of data on these orbital motions, and then put it into a huge black-box optimization algorithm that compresses the heck out of it. It compresess and compresses until it can't find a more compact representation, no matter how much more effort it applies. The output is a function, where you can provide an initial condition, and it gives you the predicted future orbital position at any requested time.
Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time.
But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data?
The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well.
A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them.
(A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data).
The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
cyanydeez 43 minutes ago [-]
Once MP3s were invented, I had the idea for the Apple IPOD; but obviously I didn't have a giant manufacturing wing, the ability to make small hard drives, or anything else.
I don't think Apple invented the ipod anymore than I invented it; LLMs likely would have also come to the same conclusion about an ipod like device.
Original ideas either dont exist or have a functionally irrelevent definition in comparison with inputing tokens to LLMs to get novel ideas out.
throwaway_7274 1 hours ago [-]
Incidentally, the relationship is bidirectional. You can try it out just for fun. zstd is a pretty crappy language model :)
variadix 1 hours ago [-]
This is a lot less surprising when you learn how non-LZ compressors work, that is, by modeling a probability distribution and using those probabilities to encode information in the minimum number of bits required to transmit the data. A less obvious conclusion is that LZ compressors do this to implicitly, the length of each symbol they could emit (literal or match, etc.) can be converted to the probability distribution the LZ compressor induces, since the number of bits to encode the symbol is related to its probability by the information content.
duskwuff 34 minutes ago [-]
A common design in compressors is to use LZ as a first step, but to then represent the constant data and/or offset-length pairs from LZ using an entropy coder.
Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio.
Bzip2 is an interesting aversion of this pattern - it uses the Burrows-Wheeler transform as a first pass instead of LZ. Unfortunately, this is one of the major reasons why it's so slow.
jparishy 42 minutes ago [-]
Cool visuals and breakdown. I wrote something in early 2025 about how LLMs seem to be an emergent behavior of lossy compression, but did not have the knowledge or verbiage at the time to get this detailed. In retrospect my writing seems naive and I'm happy to have found this and the Google paper linked inside. To be a fly on the wall in some of the labs, man.
Another thought that came from the same post is that, insofar as we see LLMs as human-style intelligence, they're more like stream of consciousness devices. Essentially incessant talking and buying enough time until you get to a usable answer. I think I associate some subset of intelligence with what you don't say, which is impossible with the SOC-style outputs, so this is something I think about a fair bit.
What could maybe differentiate current gen models from next gen is the ability to call tools modeled within the layers themselves, not externally. I think as far as I understand it, model trainers expect the model to do this itself in a way we don't understand or control, like a version of the bitter lesson. But I posit we can model many determinate tools as NNs themselves and figure out how to get the internal states of the LLM to make use of them during inference, e.g. calculators, indexes, citations. Just an enthusiast though, so grain of salt and all.
pjankiewicz 1 hours ago [-]
I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.
davmre 47 minutes ago [-]
Any compressor actually can be used, trivially, as an autoregressive language model.
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
Legend2440 49 minutes ago [-]
>Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
You can, actually! Any compressor can be losslessly converted into a generator, and vice versa.
Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.
pjankiewicz 42 minutes ago [-]
I agree that technically they are doing the same thing but in practice LLMs are better compressors than PNGs (learned this while I was researching this topic). That was quite surprising to me.
aaroninsf 46 minutes ago [-]
That sounds like boostrapping the weights involved in early layers, to obviate the need for those layers to learn (optimize) for the distribution in the training set.
Makes me wonder idly,
- is this conceptually akin in some sense to a "universal grammar," and if so
- with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training,
- does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics
pjankiewicz 26 minutes ago [-]
That was my thinking as well mainly to increase the speed of training. But it may turn out that the simple statistics that you can capture like this may account for 1% of the training and are likely to be captured as the first thing during the training.
But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc.
I always feel like people leave out the third case of the analogy: indexing
The article itself has decision trees for the compression explanation, which is also a lookup index.
In each case you try to recognise (re)usable structure.
Self-indexing succinct data-structures are a good example of the third side of the coin.
So it's a trinity: compression, prediction, indexing
deepsun 1 hours ago [-]
> compressors and LLMs
Why only LLMs? All statistical models are compressor. You can say "model" and "compressor" are synonyms.
Article does not mention "embeddings" at all, even though it's commonly viewed as a compression method. Also "encoder" part on "auto-encoders".
hmokiguess 17 minutes ago [-]
I often wonder how would language fare if we didn't have redundancy in abstractions, why do things get different terms, and if there is such a smaller set that contains everything in a lossless way (english-wise)
adamgordonbell 50 minutes ago [-]
Small world. I just did a podcast on this same topic, but coming at it from a different direction, ie. me and my neighbor trying to beat the hutter prize for compression.
Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights.
A cool Claude Shannon story:
Shannon wanted to measure how much information is actually contained in ordinary
English text. His 1948 theory said such a number must exist, but he had no way to
calculate it, because the patterns in English reach across dozens of letters and no
equation or frequency table captures all of them at once.
So instead of calculating it, he ran an experiment on a person.
He took a passage from a novel that the subject had not read, and covered it with a
card so only the text already guessed was visible. He asked the subject to name
the first letter. If the guess was wrong, he asked again, and kept asking until the
subject named the correct letter. He wrote down how many guesses it had taken,
revealed the letter, and moved the card one position to the right. Then he repeated
the process for the next letter, and the next, through the whole passage.
What this produced was not a sequence of letters but a sequence of numbers — one
number per letter, recording how many guesses that letter required. Most of the
numbers were 1, because someone fluent in English, seeing the preceding text,
usually names the next letter correctly on the first attempt.
Shannon then argued that this sequence of numbers contains exactly as much
information as the original passage.
Unrelated to the content: I was really pleased to see that this site defaults to the bare minimum for cookie consent. I reflexively clicked "Reject all" only to see that it was already the default, which threw me off.
jubilanti 22 minutes ago [-]
But all modeling is compression of a dataset? This is how I learned it in stats for CS majors 101.
sethev 53 minutes ago [-]
This immediately reminded me of the Hutter Prize (http://prize.hutter1.net/) - a contest that has run since 2005(?) based on the premise that compression is closely related to intelligence.
> Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning.
Book (creative commons): https://www.inference.org.uk/mackay/itila/book.html
Lectures: https://m.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWo...
Any rigorous CS program should cover this in depth.
[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
So they're both sourcing a bit broader zeitgeist.
Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support! Eg: you observe a rare edge case in your training data and (lossy) compression could simply ignore it. But if you wanted generalization in that particular part of the space -- either because an adversary was testing you, or for design freedom where you choose to build in that specific corner -- then you don't just want data compression, but good prediction performance on a test distribution which peaks in that corner.
Assuming that the training data distribution is exactly the distribution you will ever care for is implicitly doing a lot of the heavy lifting in the claim that compression = prediction, and I'm peeved at how much this statement is unthinkingly repeated like a manifesto.
There is nothing natural about the training data distribution, especially if the data generation process is exploratory while the downstream usage will be exploitative.
Also sparked the thought that the assumption only holds if the future looks like the present.
There is the Kolmogorov Complexity [1], Normalized Information Distance [2] and Normalized compression distance [3] that correlates those.
Finally, there's the Pre-Big Bang Informational Compression and the Delayed Release of Antimatter [4]
All big {rabbit/black} holes to lose some time, if you have any.
[0] https://en.wikipedia.org/wiki/Prediction_by_partial_matching
[1] https://en.wikipedia.org/wiki/Kolmogorov_complexity
[2] https://homepages.cwi.nl/~paulv/papers/chapter08.pdf
[3] https://en.wikipedia.org/wiki/Normalized_compression_distanc...
[4] https://philarchive.org/rec/GREPBI
I found this interesting and wonder whether LLMs have a higher density ceiling, since training and inference don't rely on a fixed lattice and can instead learn their own representational geometry.
The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.
This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit?
Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.
Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time.
But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data?
The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well.
A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them.
(A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data).
The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
I don't think Apple invented the ipod anymore than I invented it; LLMs likely would have also come to the same conclusion about an ipod like device.
Original ideas either dont exist or have a functionally irrelevent definition in comparison with inputing tokens to LLMs to get novel ideas out.
Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio.
Bzip2 is an interesting aversion of this pattern - it uses the Burrows-Wheeler transform as a first pass instead of LZ. Unfortunately, this is one of the major reasons why it's so slow.
Another thought that came from the same post is that, insofar as we see LLMs as human-style intelligence, they're more like stream of consciousness devices. Essentially incessant talking and buying enough time until you get to a usable answer. I think I associate some subset of intelligence with what you don't say, which is impossible with the SOC-style outputs, so this is something I think about a fair bit.
What could maybe differentiate current gen models from next gen is the ability to call tools modeled within the layers themselves, not externally. I think as far as I understand it, model trainers expect the model to do this itself in a way we don't understand or control, like a version of the bitter lesson. But I posit we can model many determinate tools as NNs themselves and figure out how to get the internal states of the LLM to make use of them during inference, e.g. calculators, indexes, citations. Just an enthusiast though, so grain of salt and all.
Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
You can, actually! Any compressor can be losslessly converted into a generator, and vice versa.
Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.
Makes me wonder idly, - is this conceptually akin in some sense to a "universal grammar," and if so - with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training, - does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics
But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc.
This paper proposes something similar where you can inject memory without training https://arxiv.org/abs/2605.16893
The article itself has decision trees for the compression explanation, which is also a lookup index.
In each case you try to recognise (re)usable structure.
Self-indexing succinct data-structures are a good example of the third side of the coin.
So it's a trinity: compression, prediction, indexing
Why only LLMs? All statistical models are compressor. You can say "model" and "compressor" are synonyms.
Article does not mention "embeddings" at all, even though it's commonly viewed as a compression method. Also "encoder" part on "auto-encoders".
Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights.
A cool Claude Shannon story:
Sounds a lot like next token prediction to me.https://corecursive.com/the-hutter-prize/
http://prize.hutter1.net/
https://github.com/hkust-nlp/llm-compression-intelligence
https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
https://news.ycombinator.com/item?id=19589848
https://news.ycombinator.com/item?id=27244004
https://bellard.org/ts_zip/
https://arxiv.org/abs/2306.04050
https://www.youtube.com/watch?v=B6u-FPskfAE