Do you trust your model NOT to put malware in your software?
How AI models can be tampered to insert malicious software and how to prevent that from happening.
The biggest chunk of AI usage is for coding - close to 40% on frontier models (source https://www.fdq.ai/state-of-ai). Anthropic and OpenAI are leaders with their frontier models and harnesses but many people are considering alternatives as these frontier models increase in price (per token) and degrade in performance.
Running LLMs locally is becoming more and more accessible and the quality of the produced output is very much feasible. Hugging Face is flooded with all kinds of models that can be downloaded and run locally.
Running local AI has multiple advantages but it also presents many risks. One of the risks I want to share with you is about using tampered AI models for code generation.
Tampered AI Models and LLM05: Data and Model Poisoning
A tampered AI model is a model that has been intentionally modified to alter its behavior or produce outputs that benefit an attacker. This can be achieved by manipulating training or fine-tuning datasets, modifying model weights, or introducing hidden backdoors that influence the model’s responses under specific conditions. Unlike conventional vulnerabilities, model poisoning can be particularly difficult to detect because a compromised model may continue to produce seemingly legitimate and accurate responses while exhibiting malicious behavior only in specific scenarios.
Open-Source vs. Open-Weight AI Models and The Risk of Model Tampering
Open-source AI models provide access to the materials required to understand, inspect, modify, and reproduce a model, including its source code, training data, and training methodology, alongside the trained model weights. Open-weight models, on the other hand, make the trained model weights publicly available without necessarily disclosing the underlying training data, source code, or methodology.
This distinction introduces an important security concern: while open-weight models allow developers to download and run models locally, they also allow third parties to modify, fine-tune, and redistribute them without the original model developer’s knowledge or control. Malicious actors could exploit this by introducing backdoors or manipulating model behavior and distributing tampered versions that appear to be legitimate models.
In the context of AI-assisted software development, this creates a potential software supply-chain attack vector (OWASP LLM04 - Supply Chain). A tampered coding model could be deliberately manipulated to recommend malicious software libraries or packages, generate vulnerable code, or introduce malicious dependencies into otherwise legitimate software projects. Developers who trust the model without verifying its provenance and integrity could unknowingly incorporate these recommendations into their applications, potentially compromising their development environments, CI/CD pipelines, and production systems.
Protecting Against Tampered AI Models
Individuals and organizations using open-weight AI models, particularly for software development, should implement security controls to verify model provenance and integrity before deploying them.
Some security measures that you can implement include:
- Download models only from trusted publishers and official repositories;
- Avoid downloading modified models;
- Use model security scanning and testing tools to identify suspicious model artifacts, potentially malicious configurations, embedded executable code, and unexpected model behavior.
- Run models in restricted environments with restricted filesystem access, network connectivity, and execution permissions.
- Never blindly trust package recommendations generated by AI coding assistants. Verify package names, maintainers, repositories, versions, and integrity before installation. Use Software Analysis tools to identify vulnerable or malicious dependencies. Check my previous article for SBOM and LMBOM.
- Monitor and treat model updates, new versions, and fine-tuned derivatives as new software supply-chain dependencies.
Risks with Cloud LLM Providers
Have in mind that the risk of coming across poisoned models exists not only for local models. Cloud providers are not immune to this type of attack. They just may not be aware of it. The chances are significantly lower because they earn money from providing LLM models in the cloud and they surely don’t want bad reputation from poisoned models. That is why they take care to protect and monitor them.
The attack vector with cloud models is relying on their hallucination. It is not uncommon for models to propose old or non-existing software libraries and packages. Attackers who spot these hallucinations take advantage of them by creating fake repositories or using existing vulnerabilities in the old ones.
Conclusion
No matter what model you use for coding—Cloud or Local—you should always inspect your final product for vulnerabilities. Software nowadays usually relies on many different technologies, packages and libraries. You have to be sure that all of them are secure and reliable. If you are negligent the risks are high.