What is value, and why does it keep slipping through our fingers?
andre
Value is one of the most fundamental things in human society, in human life, and perhaps in the universe itself for anything that thinks. Yet it is seldom discussed explicitly or directly. Value, in its most primitive form, is simply what tells us whether something is good or bad, or whether one option is better than another.
Its importance is obvious. Every day we make decisions: go left or right, ask or stay silent, say yes or no, eat or refrain. Each choice produces an outcome of higher or lower value for ourselves, for people we care about, for people we dislike, or for strangers we barely register. The trouble is that we often cannot assess that value with any reliability. The factors that matter may be unknown to us, or known only in outline, or known in detail yet still misjudged. Even when all the relevant data are in front of us, we can still assign value wrongly through ordinary human failings: stupidity, emotion, distraction, or a particularly vicious toothache. Humans are not perfect.
The difficulty grows worse with time. There are more of us, and we possess more powerful tools. Nuclear weapons, if ever used, will inflict enormous pain even if they do not end civilisation. Laboratory accidents involving dangerous pathogens, oil spills, and greenhouse gases already demonstrate decisions whose value looks low for large numbers of people. The pattern is familiar.
Consider a mundane illustration. John and James both want the same promotion. If John receives it, the immediate value is high for him and his family, low for James and his. Time, however, often rearranges the picture. The promotion may demand long hours; promised bonuses may be delayed by factors outside John’s control; his marriage may fray under the strain; his children may see less of him. James, passed over, leaves the firm and finds better pay, better balance, and a happier home. What looked like high value becomes low, and what looked like low value becomes high. The same reversal appears across countless ordinary choices.
The problem becomes sharper still with artificial intelligence. The leading laboratories say they want systems guided by value functions that keep the technology responsible. The hard question is who decides what counts as value. Some prohibitions are easy to state: the system should not help users kill or harm people, nor supply instructions for murder, bioweapons, or other clear harms. Yet the boundary blurs at once. How does one distinguish a request for knowledge of a lethal pathogen made in the service of legitimate research from the same request made with destructive intent? Hacking skills illustrate the dual-use problem neatly. Building defences against intrusion is widely regarded as good; the underlying techniques overlap heavily with those used for attack. Many other cases sit in the grey zone between these extremes.
The difficulty intensifies with the rise of sovereign AI. Different countries will pursue their own capabilities, guided by national interest rather than any shared conception of value. This is not merely a contest between America and China; it involves India, several states in the Middle East, Russia, and others. The result resembles the John-and-James contest, only with far higher stakes.
At present, explicit talk of value is largely confined to theology and philosophy. Scriptural commandments and centuries of commentary offer one set of answers; philosophers offer others. Neither has produced a systematic, widely agreed framework that can be rendered into mathematics without losing contact with ordinary human ethics. Different societies already assign different weights to different goods. That divergence is unlikely to shrink. In the decades ahead we should expect more disagreement about value, not less.