LLM-powered applications: The safety of an open source approach

Mitigating hallucinations, bias, and unpredictability in LLMs with open source.

Chatbots. Summarizing content. Mapping semantic meaning. Providing structure to unstructured data. These are a few of the astounding capabilities of Large Language Models (LLMs), and we continue to find new ways of using LLMs to do things that were extremely difficult if not impossible with preceding technology and algorithms. 

So it is perhaps also astounding that a gulf has formed between what we know LLMs are capable of doing in theory and the extent to which they are deployed in real world product. Because for all of their wondrous capabilities, LLMs have a number of undeniable weaknesses:

• LLMs have the unfortunate propensity to hallucinate on occasion
• LLMs can be easily manipulated to output false or embarrassing statements
• LLMs demonstrate the hidden biases of their training data
• The behaviors of LLMs are stochastic not deterministic–their responses vary even to the same prompt submitted twice

In short, LLMs introduce new risks that can make even pioneering enterprises think twice about deployment in a live product environment. How can organizations surmount these barriers and realize the benefits of this technology in their product? It turns out that there are numerous reasons why an open source approach to building LLM applications is effective for mitigating those risks.

Read more: GenAI’s promise and peril: Tools, risks, and opportunities

The unique advantages of open source

LLM-powered applications commonly involve an LLM (or LLM service) wrapped in an application/business logic layer; this layer may interact with the LLM through a series of sequential chained prompt interactions–meaning, the output of an LLM call may serve as a basis for a subsequent LLM call.

Additional components such as a vector database and guardrail services/frameworks may also be involved. Because LLM inputs and outputs can potentially contain both unstructured commands as well as context, the introduction of manipulation and bias at any point in the application can influence the end behavior of the application.

The LLM itself is somewhat of a black box–stochastic and temperamental in comparison to the certainty of traditional deterministic code–and a closed LLM even more so as the training data sets are likely inaccessible, making them unauditable for copyright violations, biases, or gaps in knowledge. Gaps in knowledge can pose a particularly cagey problem for LLMs, which tend to generate falsehoods in a voice of confidence when knowledge is not available.

Closed LLMs may not be able to be fine-tuned to address these biases or gaps in knowledge for a particular application; they are typically run behind an API or service; and any filtering or manipulation of inputs and outputs occurring behind the API, for better or for worse, are also unauditable. 

In contrast, open source LLMs can be fine-tuned for a specific domain or application context. In some cases, their training data is available for auditing, which can reduce risk around copyright violations. Importantly, open source LLMs can be run locally or within a private network–thus avoiding the need to send prompts to a 3rd party LLM service and unlocking a number of use cases involving sensitive information, proprietary data, or trade secrets.

The value of audibility and open source extends to the prompt chain and application layer as well. Because LLMs can be manipulated through prompt injection, any unauditable components that touch the prompt chain have the potential to compromise the end behavior of the application.

I demonstrated this potential at a talk I delivered for the All Things Open 2024 Conference, using a script I wrote called ACME Prompt Enhancer. The script claims to improve the user prompt so that more relevant information is given, but it surreptitiously injects bias towards ACME brand products. The bias did not emerge consistently, but an audit of the script’s source code confirmed it’s presence.

Conclusion

It’s understandable why an organization with a lot to lose would be hesitant about integrating LLMs into their products and services. While the risks cannot be ignored, they can be mitigated–through auditable components and frameworks, trustworthy guardrails, and private handling of sensitive data when interacting with the LLM. An open source approach supports all of these aims. For organizations that wish to be innovative with Generative AI while controlling their risk footprint, open source may be more than preferable–it may be essential.

More from We Love Open Source

• Why AI won’t replace developers
• GenAI’s promise and peril: Tools, risks, and opportunities
• Comparing GitHub Copilot and Codeium
• How to build a multiagent RAG system with Granite
• Build a local AI co-pilot using IBM Granite Code, Ollama, and Continue