A Nature Methods paper found that even containerized ML code can produce different results on different hardware. Same code + same data + different GPU = different outcome.

They are even seeing this with containers. Because hardware, drivers, and libraries in actual machine differ.

See people arguing over determinism in code gen. I wonder how much hardware matters.

I am trying to see the point in your post.

But if the summary is; that they don't know what they're doing, that's a fact. LLM's do not deliver guaranteed results at all. It's after a training still a blackbox. So, yeah, for a nerd to use for some mayhem coding, sure. But if you would rely on it communicating with customers or whatsoever, they're still not reliable enough. At this point, while i love all the LLM's. I doubt the AI factor. Is this really AI? Who defines!?

Edit: haha, i swear, also the AGI shit is bullshit, won't happen. The only thing that would happen is that they inventied it. Not gonna happen.

@retoor people are doing AI research of some sort. Spend years and thought they controlled the variables via containers. It is turning out the underlying hardware/software is affecting the result. Could invalidate a lot of research.

I think its interesting in a growing pains sort of way. Wonder if it will result in some sort of compute standardization. But imagine you put up a paper of some discovery. Only to find its bullshit hardware side effect. Frustrating and years of work lost. I would be pissed as a researcher.

The importance of this depends on what "ML code" is.

If it's an already-trained model with temperature at some constant, this is slightly interesting because AFAIK an LLM tuned with 0/ constant temp should produce the same results. It's just a program reading the weights of the model.

If it's a container that is training the model twice and comparing results, this is painfully obvious. There's a lot of stochasticity in model training.

@AlgoRythm so is there differences in training the same model on the same setup?

is this the one you're referencing?

https://onlinelibrary.wiley.com/doi...

"Various studies have demonstrated that hardware differences, such as different GPUs or CPUs, and compiler settings can lead to different computational outcomes (Hong et al. 2013). Additionally, a comparison between the same ML algorithm with fixed random seeds executed using PyTorch and TensorFlow resulted in different performances "

@YourMom of course, models are generally initialized with random weights, not zero, because zero weight wrongfully indicates a negative relationship between two neurons. If the weight is random, it can be smoothed out as you train into the correct value as you optimize for your loss function.

@qwwerty the most boring thing of all time is a paper comparing the training of 2 copies of the same model using 2 different implementations of ML. Yawnnn

Hmm... I am reminded of a thing a friend once said to me.

Apparently at some point in the past there was this quirk in how macs or something else from Apple would calculate things.

`10 + 2 * 3` calculated on that hardware would result in the wrong answer, given how it failed to properly factor in the correct order of the calculation.

You had to be explicit w/ it `10 + ( 2 * 3 )`.

...or something like that. I heard it so long ago, I only remember the gist of it.

I've never actually needed to verify this story, so... take that as you will.

@D-4got10-01 aren't a lot of "calculators" doing that unless they have options for parenthesis? You enter 10, then hit +, then enter 5, then hit *, then enter 2. It does each operation at each step. But I think a spreadsheet would take into account of order of operations.

I wonder what application it was that apple had. I don't ever recall using a calculator on an apple or mac.

@YourMom The topic came up when speaking about programming games... maybe even when we had been porting one of the games. So, it posed a problem if the code was tailored to usual PCs && had to be ported.

...something along that line.

@D-4got10-01 oh, you mean the compiler did that? Wild.

@YourMom Yeah. The built game on that platform would behave differently than on PCs due to that quirk. So e.g. your score would be wildly different on each platform, when doing the same thing.