I'm finetuning llama 3.1 8b locally on a 32gb 5090. It took weeks to learn enough to know where to start, hours to get my environment sane, and it's going to take 5.5 hours to finish training.

I started on my own model but the lack of data and acres of server farms led me down the finetuning path.

I honestly don't have any idea if any of it's going to be worth the time or CONSIDERABLE financial investment I made building my computer, but I refuse to stay ignorant.

Cool.

Would be doing the same if I had the resources.

You finding the heat and power-usage profile on the 5090 okay or is the thing melting?

Can I keep you as a contact/backpocket reference to run some new architecture code if thats something that would interest you?

Tell me more, I've been on a similar grind and never know where to start (and what to start it for)

@Wisecrack I have to keep the side of my case off to keep the thing operational even with 6 fans and an AIO cooler. I recently limited the power to 70% max for very minimal performance loss, so I might not need to do that anymore.

The power cable I was using started to melt, but I replaced it with the one that came with it and I don't have any problems with that anymore.

I'm open to collaborating!

@BordedDev I started by asking Grok, and just about every agent that Pycharm Professional has to offer, and I've had generally positive progress. The information cutoff happened sometime before a lot of function args changed, so that has been a bit of an annoyance, but I eventually found myself with a working finetuning setup.

I'm now struggling with the realization that LLMs really are just fancy autocomplete engines. Finetuning doesn't impart any new abilities or knowledge that the model doesn't already possess.

LLMs cannot do exact lookups of a dataset and no amount of finetuning will enable it to do so. You need a RAG system to recall facts that aren't overwhelmingly represented in the training data. A finetuned model is the search engine for a RAG system with a personality you shaped.

Pretraining establishes an underlying relationship between words and concepts and finetuning establishes its "personality". Any facts it learns in the process is truly accidental and is based on the statistical relationship between words. That is where hallucination comes from.

That means that the dataset is the Word of God to an LLM, and does not have to be based in reality. If you train a model on a dataset full of falsehoods, it will return text that is factually wrong, but statistically correct.

Not realizing that LLMs can't do exact lookups of its dataset cost me a couple days, even though every AI agent I used said it was possible. It may technically be kind of possible with overfitting, but it only works on small datasets.

I have to be missing something. It seems so mundane and boring.

The difference between something like Grok and llama and a model I created from scratch can't just be the amount of data I have to feed it and the hardware available to train and run it, can it?