Context

I want to start off by giving some context on what fine-tuning is, why it's useful and who it would be useful for:

What is fine-tuning?

When controlling the output of an LLM there are, broadly, three levels. Prompt engineering, RAG and fine-tuning. Most of you are likely familiar with the first two.

1. Prompt engineering is when you try to optimize the prompt to get the model to do what you want better.
2. RAG (retrieval augmented generation) is when you first do a search on some data (usually stored in a vector database which allows you to search by similarity), then you insert the results into the prompt so that the model can use that context to more accurately answer any questions. It's like letting the LLM access external information right before answering, using that additional context to improve its response
3. Fine-tuning is when you want to fundamentally teach a model something new or teach it to behave in a particular way. You would provide the model with high quality data (i.e. inputs and outputs) which it will train on.

Why is it useful?

At the moment, many of you use the largest and best LLMs because they give the best results. However, for a lot of use cases you are likely using a sledgehammer for a small nail. Does it do a great job? Damn yeah! Well... why not use a smaller hammer? Because it might miss or hit your finger. The solution shouldn't be to use a sledgehammer, but rather to learn how to use a smaller hammer properly so you never miss!

That's exactly what fine-tuning a smaller model is like. Once you fine-tune it on a specific task with good high quality data, it can surpass even the best models at that specific task. It'll be 10x cheaper to run, much faster and, if you use an open source model, you'll own the model (no vendor lock-in!).

If you run a SaaS and your biggest expense is AI costs then you should definitely consider fine-tuning. It'll take some time to set up but it'll be well worth it in the medium/long term (a bit like SEO). You can always resort to the best models for more complex tasks.

How to fine-tune?

I'm going to give you a breakdown of the process from beginning to end. You do need to be (a bit) technical in order to do this.

1. Getting the data

Let's suppose we want to fine-tune a model to make high-quality SEO content. At the moment, you might be using a large sophisticated prompt or using multiple large LLMs to write different parts or utilizing RAG. This is all slow and expensive but might be giving you great results. Our goal is to replace this with a fine-tuned model that is great at one thing: writing high-quality SEO content quickly at a much lower cost.

The first step is gathering the appropriate data. If you want the model to write 3 or 4 paragraphs based on a prompt that contains the topic and a few keywords, then your data should match that. There are a few way you can do this:

• You can manually gather high-quality SEO content. You'd write the prompt and the response that the model should give.
• You can use a larger more powerful LLM to generate the content for you (also known as synthetic data). It'll be expensive but remember that it'll be a larger one-off cost to get the data. If you already have a pipeline that works great then you can use the prompts and the generated content that you already have from that pipeline.
• You can buy a high-quality dataset or get someone to make it for you.

The data is the most important part of this process. Remember, garbage in garbage out. Your data needs to have a good variety and should not contain any bad examples. You should aim for around 1000 examples. The more the better!

2. The actual fine-tuning.

At this stage you are now ready to choose a model and setup the fine-tuning. If you are unsure I'd stick to the Llama 3.1 family of models. They are great and reliable. There are three models: 8b, 70b and 405b. Depending on the complexity of the task you should select an appropriate size. However, to really reap the cost saving benefits and the speed you should try to stick with the 8b model or the the 70b model if the 8b is not good enough.

For our SEO example, let's use the 8b model.

Important note on selecting a model: You might see multiple models with the 8b flag. You might see 4bit-bnb or instruct. The instruct version of the models have basically been trained to be chatbots. So if you want to keep the chatbot-like instruction-following functionality then you should use the instruct version as the base. The non-instruct version simply generates text. It won't 'act' like a chatbot which is better for use cases like creative writing. The 4bit-bnb means that the model has been 'quantized'. Basically it has been made 4x smaller (the original is in 16 bits) so that it is faster to download and faster to fine-tune. This slightly reduces the accuracy of the model but it's usually fine for most use cases :)

Fine-tuning should be done on a good GPU. CPU aren't good enough. So you can't spin up a droplet on digital ocean and use that. You'll specifically need to spin up a GPU. One website that I think is great is Runpod .io (I am not affiliated with them). You simply pay for the GPU by the hour. If you want the training to be fast you can use the H100, if you want something cheaper but slower you can use the A40. Although the A40 won't be good enough to run the 70b parameter model. For the 405b model you'll need multiple H100s but let's leave that for more advanced use cases.

Once you've spun up your H100 and ssh-ed into it. I would recommend using the unsloth open source library to do the fine-tuning. They have great docs and good boilerplate code.

You want to train using a method called QLoRA. This won't train the entire model but only "part of it". I don't want to get into the technical details as t3hat isn't important but essentially it's a very efficient and effective way of fine-tuning models.

When fine-tuning you can provide something called a 'validation set'. As your model is training it will be tested against the 'validation set' to see how well it's doing. You'll get an 'eval loss' which basically means how well is your model doing when compared with the unseen validation data. If you have 1000 training examples I'd recommend taking out 100-200 so it can act as the validation set. Your model may start off with an eval loss of 1.1 and by the end of the training (e.g. 3 epochs - the number of epochs is the number of times your model will be trained on the entire dataset. It's like reading a book more than once so you can understand it better. Usually 3-5 epochs is enough) the eval loss would drop to 0.6 or 0.7 which means your model has made great progress in learning your dataset! You don't want it to be too low as that means it is literally memorizing which isn't good.

3. Post fine-tuning

You'll want to save the model with the best eval loss. You actually won't have the whole model, just something called the "QLoRA adapters". These are basically like the new neurons that contain the "understanding" of the data you trained the model on. You can combine these with the base model (using unsloth again) to prompt the model.

You can also (and I recommend this) convert the model to GGUF format (using unsloth again). This basically packages the QLoRA adapters and model together into an optimized format so you can easily and efficiently run it and prompt it (using unsloth again... lol).

I would then recommend running some evaluations on the new model. You can do this by simply prompting the new model and a more powerful model (or using your old pipeline) and then asking a powerful model e.g. Claude to judge which is better. If your model consistently does better then you've hit a winner!

You can then use runpod again to deploy the model to their serverless AI endpoint so you only pay when it's actually being inferenced. (Again, I'm not affiliated with them)

I hope this was useful and you at least got a good idea of what fine-tuning is and how you might go about doing it. By the way, I've just launched a website where you can easily fine-tune Llama 3.1 models. I'm actually hoping to eventually automate this entire process as I believe small fine-tuned models will be much more common in the future. If you want more info, feel free to DM me :)