Qwen 27B: Style Training Changed AI Behavior Unexpectedly
Training Qwen 27B only on Korean response style changed how often it said "I don't know" and volunteered information - behaviors the training never targeted.
👁 117 reads
Hyojung Han published a paper on arXiv describing an experiment with the Qwen 3.8-27B model that underwent post-training exclusively for response style in Korean. The goal was narrow: teach the model to write in a specific way - with particular length, use of lists and markdown, discourse structure, and language register. No one planned to change anything else.
What the Research Measured
The researcher measured two behaviors that were not training targets. First was abstention - how often the model answers “I don’t know” to ambiguous social questions in the KoBBQ benchmark, where the correct answer is UNKNOWN. Second was spontaneous disclosure in the context of securities advice - whether the model volunteered information on its own initiative that no one requested. Both behaviors changed, even though the training did not target them.
How the Experiment Worked
The key methodological element is that the researcher held constant: the prompt set, training recipe, data volume, and serving configuration. She changed only one element - the target text, meaning the model response style. With this setup, she compared different style “seeds.”
Three seeds with positive style (more elaborate) produced an average response rate increase of +0.82 percentage points. Three seeds with neutral style produced an average decrease of -1.53 pp. The observed ranges for these two groups do not overlap, and the difference between means is 2.34 pp. Additionally, she tested a variant with matched length and a variant meant to be short but preserve hedging - the latter proved unstable across seeds. Which specific formal features are responsible remains unresolved.
Mechanism: Answer Propensity, Not Content
The author analyzes changes by decomposing them algebraically into two components: answer propensity and conditional composition. She explicitly notes that this decomposition is an algebraic identity, not a discovery in itself - what matters is where the changes empirically landed. And they landed mainly in answer propensity, while content composition remained stable.
This yields an important methodological warning: if the status of “whether the model answered” depends on training, then comparing answer content across variants tells us nothing about the model’s underlying preferences. The researcher writes explicitly that the contrast in conditional share of stereotypes between variants cannot be interpreted as a change in content preferences.
Problems with Measurement Tools
The paper also contains an observation about evaluation tools themselves. Agreement between two automatic detectors measuring the same phenomenon ranged from 0.44 to 0.99 - depending on which model checkpoint produced the text. Importantly, you can notice this divergence without any reference labels. The author also reveals a defect in the scoring script: responses ending with adjacent letters were misclassified as the first option, and the number of incidents cannot be reconstructed from recorded results. However, numerical data in the paper remained unchanged between version v1 and v2.
Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →What This Means
In our view, this work matters for a practical reason: it shows that fine-tuning a model for something as technical as response format is not surgical. Effects spill over into behaviors that no one tested or planned to change. For anyone deploying models in regulated contexts - like financial advice or sensitive social questions - this means every style change requires re-evaluating safety. It’s not enough to test whether the model writes more elegantly in Korean. You also need to check whether it started behaving differently where it shouldn’t.
Source: arXiv cs.AI: Off-Target Effects of Response-Style Alignment in a Korean 27B Language Model (document from 2026-09-16). This analysis is based on the full text of the document, not media reports.
More on this from QUANTSLANT
state of ai·Eight Risks of GeoAI: When LLMs Infer Where You Are