HuiShan Lai
← Research archive

Information Security Practice and Experience (ISPEC 2025) · Published

Assessing the Security of Vibe Coding: Baseline Vs. Security-Oriented Prompts in LLM Code Generation

A controlled evaluation of whether security-oriented prompts reduce vulnerabilities in LLM-generated Python code across tasks, random seeds, and model families.

Runtong He, Huishan Lai, Jingxue Chen, Chunhua Su

University of Aizu & UESTC · Japan & China

DOI: 10.1007/978-981-95-9284-5_27

Paper Springer Code
Severity-weighted vulnerability counts for baseline and security-oriented prompts across ten Python programming tasks
Security-oriented prompts lowered vulnerability severity for several tasks, but did not remove every weakness.

Abstract

Vibe coding makes software prototyping more accessible, but code that satisfies a functional request can still contain exploitable weaknesses.

This study compares baseline and security-oriented prompts on ten Python tasks inspired by OWASP Top 10 and CWE categories. It evaluates GPT-OSS 20B across ten random seeds and compares GPT-OSS 20B with Gemma-3 27B in a three-seed cross-model study.

Bandit static analysis and lightweight runtime probes show that security-informed prompts can reduce vulnerability prevalence and severity, particularly for command execution and unsafe deserialization. Persistent failures in file upload and authentication demonstrate that prompting mitigates risk rather than eliminating it.

Methods

Controlled security evaluation

Two prompt conditions are evaluated across ten security-sensitive coding tasks, multiple random seeds, and two locally served language models.

Experiment pipeline from ten Python tasks through baseline and security-oriented prompts, code generation, Bandit analysis, runtime probes, and aggregated security metrics
The pipeline measures vulnerability presence, issue count, severity-weighted count, and runtime-probe results for each generated sample.

Results

Security prompts help unevenly

Across ten GPT-OSS 20B seeds, the strongest change appears in command execution: vulnerability presence falls from 60% to 10%. File upload and JWT authentication remain vulnerable in every sample.

Bar chart comparing vulnerability presence for baseline and improved prompts across ten GPT-OSS 20B coding tasks
Vulnerability presence by task for GPT-OSS 20B across ten random seeds. Empty bars indicate no Bandit or runtime-probe finding for that condition.

Lower prevalence does not always mean complete remediation. Security-oriented prompts reduced the severity-weighted count for unsafe deserialization from 4.6 to 1.8, while file-upload vulnerability presence stayed at 100% and email validation became less reliable in this sample.

Cross-model study

Improvements do not transfer uniformly

Under the improved prompt condition, GPT-OSS 20B produced fewer vulnerable samples than Gemma-3 27B in unpickling and secrets management, while both models remained vulnerable in SQL and file-upload tasks.

Bar chart comparing vulnerability presence under improved prompts for Gemma-3 27B and GPT-OSS 20B across ten tasks
Cross-model vulnerability presence under security-oriented prompts, based on three random seeds per model and task.

The evidence supports a layered workflow: security-aware prompts should be combined with static analysis, runtime checks, review, and verification. The study reports descriptive trends across two models and a limited task set, not a guarantee that a particular prompt or model is secure.

Authors

Runtong HeUniversity of Aizu
Huishan LaiUniversity of Aizu
Jingxue ChenUniversity of Electronic Science and Technology of China
Chunhua SuUniversity of Aizu

Citation

He, R., Lai, H., Chen, J., and Su, C. (2026). Assessing the Security of Vibe Coding: Baseline Vs. Security-Oriented Prompts in LLM Code Generation. In H. Xiong, L. Zhou, and M. Atiquzzaman (Eds.), Information Security Practice and Experience (ISPEC 2025), Lecture Notes in Computer Science, vol. 16418, pp. 470–488. Springer, Singapore. https://doi.org/10.1007/978-981-95-9284-5_27

@inproceedings{he2026assessing,
  author    = {He, Runtong and Lai, Huishan and
               Chen, Jingxue and Su, Chunhua},
  editor    = {Xiong, Hu and Zhou, Lu and Atiquzzaman, Mohammed},
  title     = {Assessing the Security of Vibe Coding: Baseline Vs.
               Security-Oriented Prompts in LLM Code Generation},
  booktitle = {Information Security Practice and Experience},
  series    = {Lecture Notes in Computer Science},
  volume    = {16418},
  pages     = {470--488},
  publisher = {Springer Nature Singapore},
  address   = {Singapore},
  year      = {2026},
  doi       = {10.1007/978-981-95-9284-5_27}
}