Ensuring that LLMs align with human values is not an easy task. Alignment is particularly challenging because human values are not static, universal, or easily quantifiable and codifiable. What is considered ethical, fair, or appropriate varies significantly across cultures, political ideologies, and social contexts, making it difficult to establish a one-size-fits-all alignment approach (Liu et al., 2023; Shen et al., 2023). An output considered neutral or factual in one country might be seen as biased or controversial in another, specifically when thinking about political values like democracy. Similarly, ethical priorities and dilemmas like whether AI should prioritise free speech or harm prevention are often in direct conflict with one another.

“As machine-learning systems grow not just increasingly pervasive but increasingly powerful, we will find ourselves more and more often in the position of the ‘sorcerer’s apprentice’: we conjure a force, autonomous but totally compliant, give it a set of instructions, then scramble like mad to stop it once we realize our instructions are imprecise or incomplete—lest we get, in some clever, horrible way, precisely what we asked for. How to prevent such a catastrophic divergence—how to ensure that these models capture our norms and values, understand what we mean or intend, and, above all, do what we want—has emerged as one of the most central and most urgent scientific questions in the field of computer science. It has a name: the alignment problem.” – Brian Christian, The Alignment Problem (2020, p.19-20).