How to choose a custom LLM for your AI project

What to ask when picking a model

Here's a good list of questions to be able to answer when picking a model:

• Can I use a cloud-based LLM or do I need to host one myself?
• How fast do I need the responses to be?
• How accurate do I need the responses to be?
• How much $$ will my project save and/or generate? Then, what price should it fall below?
• How long do I need my responses to be?

Generally speaking, it's difficult to speed up a powerful model or reduce its costs, and it’s easier to improve a less accurate model. 

However, it's much faster to get started with a powerful model, and if it fulfills your project's needs, you may not need as much engineering effort (plus, it’s easier to maintain).

Starting from a model

The first thing you should do is pick a starting model. There are plenty of leaderboards online that compare the different models.

For example:

• Hugging Face maintains a leaderboard for open source models.
• Vellum has an excellent one for the more popular models.

If your company has an in-house model, consider using it to work with your budget and keep data private. If you need to host the model yourself, consider an open-source model.

Fine-tuning 

Fine-tuning involves providing examples to your model so that it learns how to do a certain task well. If you want it to excel in speaking about your product, you might provide a swath of examples of your company’s best sales calls.

If the model is open source, ask yourself if your team has enough engineering capacity to fine-tune a model.

If the model is closed source and provided as a service – GPT-4 or Claude – then you can usually have your engineers fine-tune custom models using APIs. The price usually increases substantially through this method, but there is little to no maintenance. 

But for many use cases, fine-tuning is not the first step towards optimizing your model. 

A great case for fine-tuning is building a knowledge bot for static knowledge. By giving examples of questions and answers, it should be able to answer them in the future without looking up the answer. But it’s not a practical solution for real-time information. 

Retrieval-augmented generation

RAG is a fancy name for a simple thing that we've all done in ChatGPT: pasting some text into ChatGPT and asking a question about it. 

A typical example is asking if a certain product is in stock on an e-commerce site, and a chatbot looking up the information in a product catalog (instead of the wider internet). 

In terms of speed of development, and getting real-time information, RAG is a must-have. 

It doesn't usually affect which model you will pick, however nothing stops you from creating an LLM API endpoint that queries information and answers and using this endpoint as though it were its own LLM. 

Using RAG for a knowledge-based chatbot is often easier to maintain, as you don't need to fine-tune a model and keep it up to date – which can also reduce costs.

N-shot learning 

The fastest way to get started in improving the quality of responses is to provide examples in a single LLM API call. 

Zero-shot – giving zero examples of what you're looking for in an answer – is how most of us use ChatGPT. Adding one example (or one-shot) is usually enough to see a substantial improvement in the response quality. 

More than one example is considered n-shot. N-shot does not change the model, unlike fine-tuning. You're simply giving examples just before asking for a response, every time you ask a question. 

But this strategy can’t be overused: LLM models have a maximum context size, and are priced according to the size of the message. Fine-tuning can remove the need for n-shot examples, but takes more time to get right. 

Other prompt engineering techniques

There are other prompt engineering techniques, like chain-of-thought, which force models to think out loud before coming up with an answer. 

This increases the quality of response, but at the cost of response length, cost and speed. 

My recommendation

While every project will have its own unique needs, I’ll give my two cents on a strong approach. 

A good place to start is using an off-the-shelf model that balances speed and quality, like GPT-4o Mini. Start by looking at the quality of the responses, response speed, cost, context window needs, and decide what needs to be improved from there. 

Then, with a narrow use case, you can try some simple prompt engineering, followed by RAG, and finally fine-tuning. Every model that goes through these will have performance gains, so it can be tricky to figure out what to use.

Offload engineering to save $

Keeping a lean budget is critical in making your project a reality. One of the ways you can do that is reducing engineering time by decoupling requirements. 

Nowadays we have access to low-code solutions like Flutterflow, Shopify, which can be used by traditionally non-technical roles like Product Managers. Chatbots are no exception, and some AI automation platforms even allow you to use your own LLM. 

You can instruct engineers to focus on hosting the LLM and setting up with the automation platform. That frees up the business analysts, product managers, and other related roles to build AI agents that satisfy business requirements. 

When something additional is required, these platforms generally have a way for engineers to add some code. This way, you keep the advantages of a custom model, and gain flexibility, speed and affordability.

Provide engineering freedom to solve business problems

On the other hand, sometimes business problems are just very hard to solve. 

We're talking about fully network-gapped LLM applications, on-device apps, or projects requiring giving chatbots extremely advanced capabilities that are more than syncing data between two platforms. 

In those cases, allowing engineers the freedom to use whatever tools they are most comfortable makes sense. This is usually just writing code, and stakeholders simply act as project managers.

1. How do I evaluate the ROI of investing in a custom LLM for my business?

Start by comparing the upfront and ongoing costs (engineering time, infrastructure, licensing) with the expected gains like time saved, increased customer satisfaction, or higher conversion rates. If the model automates a high-cost or high-impact task, ROI becomes easier to justify.

2. What KPIs should I track to measure the effectiveness of a custom LLM?

Look at metrics like response accuracy, latency, user satisfaction (CSAT or NPS), resolution rate, and cost per interaction. These help you gauge both the performance and business impact of your LLM.

3. How can I estimate the long-term maintenance cost of a custom LLM solution?

Factor in recurring costs like server hosting, model retraining or fine-tuning, engineering hours for updates, and monitoring tools. Also consider how frequently your data changes, as that affects how often you’ll need to re-optimize.

4. How can I benchmark different LLMs for my specific industry or domain?

Test models with real, domain-specific queries and evaluate their responses for accuracy, tone, and relevance. You can also use industry datasets or simulate user tasks to see how well each model performs under real-world conditions.

5. What compliance standards should a custom LLM meet if I’m handling user data?

At a minimum, look for GDPR, SOC 2, and HIPAA (if applicable). Ensure the model provider supports data encryption, audit logs, and has clear data handling and deletion policies.