Shannon airport

 Charlie Snell et al., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. The famous scaling laws paper from Kaplan et al. is about how model performance improves with the number of model parameters (the calculation of which is ‘train-time compute’). But what happens when you scale compute for inference?11

I said this before, but I find it amusing how much papers about scaling read like economics papers. We’re basically setting up an optimisation problem, and trying to understand the “exchange rate” between a marginal FLOP of pretraining and a marginal FLOP of inference. And then a small number of firms can use this information to optimise their production in a model of oligopolistic competition.. . . "

Before we can understand von Neumann entropy and its relevance to quantum informa-tion, we should discuss Shannon entropy and its relevance to classical information