>>AI and Ethics: A Set of Questions On Creating Morally Aligned Machines
We need not imagine science-fiction (or near-future) scenarios with fully autonomous, embodied, or super-intelligent artificial agents in order for pressing ethical questions to arise. Current AI systems make decisions with moral consequences (such as when a self-driving car should swerve during an accident,1 which job candidates are worth shortlisting, and which inmates should be granted parole2). People are increasingly using Large Language Models (e.g. Therapist GPT) to seek counsel, life advice, and even therapy.
Even less obviously problematic cases are still ones where value judgments (judgments about what is good or bad, right or wrong, worth pursuing or worth avoiding) lurk in the background. Consider the case where one visits a new city and asks ChatGPT to tell them what is worth seeing in a three-day trip. While the LLM is not making a moral judgment per se, its judgment has a tangible, albeit minor, impact on the person’s well-being. For the LLM to succeed in making the judgment, its verdicts needed to, at some level, become aligned with the user’s human preferences. An AI that is involved directly or indirectly in decision-making (either by deciding for us, or by being consulted for one of our decisions) enters the domain of the practical, and thereby, the moral: it, in some sense, decides or helps decide what is worth doing.
AI alignment
Questions about AI’s effects on the practical, value-laden realm are often presented under the buzzword of alignment; the aim, it is said, is to produce AI whose goals and emergent behaviours are aligned with our human values. But this is not an entirely neutral way of characterizing the question either: The ultimate goal can’t be simply to align AI with any set of human values a society, or a set of human stakeholders making decisions happens to have – if our society falls prey to a demagoguing dictatorship, we wouldn’t want the AI to align with the values of said demagoguing dictatorship; if our AI is owned by a limited set of corporations, we wouldn’t want the AI to align with the values of this limited set of corporations. Rather, the goal is to create AI that stably acts in the best interests of moral subjects –not privately or societally aligned AI, but morally aligned AI.3
As AI is, in fact, owned and developed by a limited set of corporations, there are, of course, practical problems in how to avoid these cases of alignment with only the values of particular self-interested, power-concentrating actors. These are real, tangible dangers, and we do not mean to demean their importance. Arguably, these dangers require challenging the ownership and control of AI by such a limited set of actors. But here we set these dark, albeit realistic worries aside and present a more abstract question: Even if we could decisively avoid these dangers, even if the decisions that shape AI alignment were reached fairly and democratically, how could we and should we approach moral-value alignment at all?
It seems a tall order, and a question that immediately thrusts us into the murky territory of moral philosophy. It seems that, to design moral AI, we need to first decide what is fundamentally morally good and bad. Call this question the question of (A) solving moral theory. We are not comfortable with AI making moral judgments on its own, but should we even be confident in the veracity of our own moral judgments? And even if we finally decide on some unquestionable moral judgments, how do we teach complex, autonomous entities to respect them? Call this latter question the question of (B) creating morally aligned agents.
For use-specific, limited scope AIs (e.g. ones that are driving cars, or AIs that are making probabilistic predictions about inmate parole violations), these questions do not seem too daunting; The limited nature of these AIs’ scope of aims and behaviours means that it is not pressing to fully answer (A) to start grappling with the relevant limited version of (B). For example, we do not need to have the complete true theory of morality in order to encode driving rules with a moral dimension into self-driving cars. This is not to say that these limited-use AIs do not often thrust us into substantive questions in moral theory. But the substantive questions at hand are limited in scope – at most, they are sub-questions about special cases of (A). For example, they require us to decide which of two statistical measures to correlate with the desired goal of fairness in the particular case of parole procedures,4 not to come up with a full true theory of fairness as such. Therefore, the general strategy of alignment for these cases seems straightforward: we need to decide on the kinds of outputs that we deem morally desirable, and ask our computer scientists to create AIs that deliver those outputs. This would be a partial kind of moral hardcoding – deciding on the behaviours we deem morally acceptable and coding for those behaviours. But for AIs approaching general intelligence and general use cases, which would be involved in a variety of value-laden decisions which we cannot antecedently predict, the problem seems much more complex.
AGI and Alignment – Lessons From the Human Case
The first obvious problem for creating morally aligned AI with general use-cases, then, is about how we can progress with respect to (B) when social practice and moral philosophy still leave (A) substantively open; Moral philosophers still disagree in attempts to formulate moral values that are universally correct. How do we then decide what rules to encode into our AI as the moral ones – rules that they never ought to disregard? The second obvious problem is that, even if we do answer (A), it is not clear that the architecture of modern and future AI would allow us to straightforwardly solve (B) in the case of AI: most AI systems are fundamentally statistical, and their algorithmic processes do not straightforwardly translate to the linguistic and logical building blocks of human thought and deliberation. AIs are made to alter their behavioural profile based on new data, often with little transparency as to how they achieve the goals that they have been trained for. Hence, it is not at all clear that we can be sure that we are making moral AI by merely hardcoding into it the moral rules – even if we had solved the problem of deciding which rules are the moral ones to begin with.
But it may be that we can make progress on (B) without first solving the oldest, perhaps hardest question in moral philosophy, namely (A). The practical question of how we teach and train autonomous, non-transparent, broadly statistical agents to behave morally is one that any reflective parent has puzzled over when raising their child. Crucially, parents aren’t usually moral philosophers, and yet through a nurturing, careful upbringing, they can, by and large, succeed in creating imperfect but by-and-large virtuous, compassionate, moral people.
Certain theorists on AI morality take what we can learn from the parenting analogy – and more generally from human moral learning – quite far: 5 much like parents don’t teach their children how to be moral by hardcoding rules into them, we shouldn’t seek to teach AI how to behave morally by hardcoding rules into it either. The complexity, opacity, and statistical nature of machine learning prove to be a feature rather than a bug: they are what allow us to create morally aligned AI without having come up with the clear, universal laws of morality, in much the same way that the complex, opaque, statistical nature of human learning results in morally sensible humans without them having been fed clear, universal moral laws by their parents and environment. If we had to raise moral children by unequivocally giving them the complete rules of morality, we would be at a loss. But through providing reinforcement and feedback to children through socialisation – messy, statistical, non-rule-like feedback– we can encode values in them that our society’s best philosophers cannot yet formulate in rule-like patterns. Is this the approach to take with AI, and what could it look like?
Potential Disanalogies with the Human Case
The question still seems very much speculative. It will depend on the architecture of the AI at hand, and its ability to get ample feedback from complex social practices; besides, children only develop into fully competent moral agents after a large amount of multimodal feedback and interpersonal relationships with parents, school, peers, and society (feedback and relationships that seem to essentially include affects and emotions).
There is also the issue of moral priors: humans succeed in moral learning not only through social feedback, but arguably by reliance on moral priors that are biologically hardwired. Infants show an intuitive grasp of key moral concepts very early –before it would be reasonable to hypothesise that they acquired it through social reinforcement.6 We still have a very limited understanding of the extent of those moral priors, and no clear idea as to what it would look like for them to be hardwired into artificial agents in the same way that they are biologically hardwired into infants. There are further disanalogies of scale in the reinforcement learning and social feedback that a child and an AI are engaged in, which make the comparison between AI and human moral learning even more dubious; a child’s moral compass is affected mostly by its immediate social environment, and usually affects its immediate, limited social environment, while some AI agents (think ChatGPT) affect billions of users every day, typically without themselves being universally affected (apart from modifying their behaviour to fit with each user’s needs).
Still, we are at a crossroads: we are building AIs that are both increasingly complex and increasingly integrated into society in a way that leaves judgments with moral consequences in their hands. And yet, our best understanding of similarly complex agents becoming morally competent is the incomplete grasp we have through observing humans ourselves; we only get there through a complex combination of nature and nurture –biologically hardwired predispositions and rich feedback through integration into social networks. Is a similar type of holistic societal integration desirable for AI?
Endnotes and References
[1] Although the significance and frequency of these types of fringe cases is likely exaggerated in the public policy dialogue on autonomous vehicles – see Kamm, F. M., 'The Use and Abuse of the Trolley Problem: Self-Driving Cars, Medical Treatments, and the Distribution of Harm', in S. Matthew Liao (ed.), Ethics of Artificial Intelligence (New York, 2020; online edn, Oxford Academic, 22 Oct. 2020), https://doi.org/10.1093/oso/9780190905033.003.0003.
[2] This kind of case is discussed in https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. Further discussion of issues surrounding the relevant algorithm’s bias and different conceptions of algorithmic fairness can be found in A. Chouldechova, “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments”, Big Data, 2017; J. Kleinberg, S. Mullainathan, and M. Raghavan, “Inherent Trade- offs in the Fair Determination of Risk Scores”, arXiv, 2016; G. Pleiss et al., “On Fairness and Calibration,” In NeurIPS, 2017.
[3] For a pushback to this, see Bonnefon, Jean-François, Azim Shariff, and Iyad Rahwan, "The Moral Psychology of AI and the Ethical Opt-Out Problem", Ethics of Artificial Intelligence, 2020, https://doi.org/10.1093/oso/9780190905033.003.0004.
[4] See the controversy again in the references mentioned in [2].
[5] See P. Railton "Ethical Learning, Natural and Artificial", in Ethics of Artificial Intelligence, 2020, https://doi.org/10.1093/oso/9780190905033.003.0002. See also R. Millière, "Normative conflicts and shallow AI alignment". Philosophical Studies, 2025, https://doi.org/10.1007/s11098-025-02347-3 for work that exploits the analogy with human moral capacities and argues for the necessity of implementing similar capacities in humans.
[6] For a review of some recent data, see L. Surian, E. Parise & A. Geraci, "Core Moral Concepts and the Sense of Fairness in Human Infants". Human Nature, 2025, https://doi.org/10.1007/s12110-025-09490-0.