When the World Refuses to Stay Still: From Restless Bandits to Adaptive Intelligence
Adaptive Intelligence, Part 4
10 min.
Abstract
Artificial intelligence is usually trained on the past and deployed into the future. That works only while the future remains sufficiently similar to the world represented in the training data. Real environments, however, do not stand still. Patients deteriorate, machines age, markets change regime, users alter their preferences, and scientific knowledge advances.
This final article in the Adaptive Intelligence series examines what happens when the alternatives themselves continue changing while decisions are being made. It moves from concept drift to the Restless Multi-Armed Bandit, from the exact elegance of the Gittins Index to the practical approximation of the Whittle Index, and finally toward a broader idea of adaptive intelligence. It also proposes a possible role for large language models as semantic sensors capable of detecting contextual evidence that the world—and not merely the data—has changed.
Key Takeaway: Intelligence in a changing world requires more than making good decisions from existing knowledge. It requires recognizing when that knowledge has begun to expire.
Keywords
Adaptive Intelligence, Artificial Intelligence, Restless Multi-Armed Bandit, Whittle Index, Gittins Index, Concept Drift, Non-Stationarity, Reinforcement Learning, Continual Learning, Large Language Models, Decision-Making Under Uncertainty
The Series
This article concludes the Adaptive Intelligence series.
Adaptive Intelligence, Part 1 — Why Every Intelligent Decision Begins Before Certainty
Read Part 1 on AI-Talks.org
Adaptive Intelligence, Part 2 — Classical Multi-Armed Bandit: Learning Under Uncertainty
Read Part 2 on AI-Talks.org
Adaptive Intelligence, Part 3 — The Gittins Index: One of the Most Elegant Algorithms Ever Created
Read Part 3 on AI-Talks.org
Now one final assumption must be removed.
The world does not wait.
Introduction: The Last Assumption
The Gittins Index offers an extraordinarily elegant rule: under the classical assumptions of the problem, simply choose the alternative with the highest index. Yet that elegance depends on a rather peculiar condition: an alternative that is not selected remains, in effect, frozen.
If a machine is ignored, it does not deteriorate. If an experiment is postponed, the opportunity does not disappear. If a patient is not treated, the patient’s condition waits until we return. This mathematical world is extremely useful because it allows each alternative to be analyzed relatively independently, but it resembles only a limited part of reality.
Machines age while we inspect other machines. Patients change while clinicians treat someone else. Markets continue moving while analysts study previous data. Users alter their preferences while recommendation systems show them content, products, or information.
The question therefore stops being simply:
Which alternative should I choose?
and becomes:
Which alternative deserves attention now, knowing that all the others may continue changing while I focus on something else?
This apparently small change transforms the problem profoundly. It takes us from static models to concept drift, from classical bandits to restless bandits, from Gittins to Whittle, and ultimately toward a more demanding conception of artificial intelligence.
The central thesis of this article is simple: intelligence in a changing world cannot be defined only by the ability to optimize. It must also include the ability to recognize that the problem being optimized has changed.
When the World Refuses to Stay Still
Machine learning requires some degree of regularity. A model can learn from past observations because it assumes that at least some of the relationships identified during training will remain useful in the future. This assumption is closely related to the idea of stationarity: not everything needs to remain perfectly constant, but the world encountered after deployment must preserve enough similarity to the one that generated the training data.
The problem begins when this necessary approximation is mistaken for a permanent property of reality. Real systems have histories: machines wear out, populations change, pathogens evolve, regulations are modified, and technologies become obsolete. Even human preferences respond to culture, incentives, experience, competition, and to the recommendations produced by algorithms themselves.
A model can therefore continue executing exactly what it learned and still become progressively less useful. In machine learning, an important part of this phenomenon is known as concept drift: the relationship between what a model observes and what it is trying to predict can change over time.
Consider a hospital AI trained to estimate the probability that a patient’s condition will deteriorate significantly within the next 24 hours. For some time, the system performs very well. Then the hospital changes: a new treatment protocol is introduced, a new pathogen begins circulating, the average patient population becomes older, a laboratory replaces one assay with another, and certain medications that once indicated severe illness begin to be administered earlier.
None of these changes necessarily causes the AI system to stop functioning. The software still runs, the model still produces risk scores, and the interface continues to operate normally. The problem is more subtle: the relationship between the measurements and the clinical outcomes may have changed.
The model can therefore remain computationally correct while becoming clinically wrong.
This is one of the most important dangers of static intelligence. Failure does not always appear as a system crash or an error message; it can take the far more dangerous form of apparently confident answers produced after the assumptions that once made those answers reliable have quietly expired.
An adaptive system must therefore ask an additional question:
Does what I learned still describe the world in which I am about to act?
The Restless Multi-Armed Bandit
The classical Multi-Armed Bandit represents a fundamental decision problem: when several uncertain alternatives are available, should we exploit what currently appears best, or explore something less familiar that may ultimately prove superior?
The Restless Multi-Armed Bandit adds a crucial complication: the alternatives continue changing even when we are not paying attention to them.
Return to the hospital. Suppose twenty patients need monitoring, but only five advanced monitoring systems are available. The hospital must decide which five patients will receive intensive attention during the next hour, knowing that the other fifteen will not remain frozen while they wait. Some may improve, others may remain stable, and others may deteriorate; a patient who appears relatively safe now may become the most urgent case before the next decision.
The same structure appears in many other domains. Machines continue degrading while maintenance teams work on other equipment; financial opportunities disappear while attention is focused on different assets; communication channels fluctuate, users lose interest, and scientific opportunities may vanish because another laboratory publishes first.
The essential distinction can be summarized in one sentence:
In a rested bandit, waiting postpones an opportunity. In a restless bandit, waiting can change the opportunity itself.
Each alternative has a current state and some tendency to evolve, and its future may depend on whether we act or wait. A patient receiving intensive treatment may stabilize, while the same patient without intervention may deteriorate. A machine that receives maintenance may remain reliable, while that same machine may continue degrading if maintenance is postponed.
In other words, doing nothing does not mean that nothing happens.
Scarcity is what turns this dynamic into a decision problem. If every patient, machine, experiment, or opportunity could receive unlimited attention, there would be little need to choose. Intelligent systems, however, operate under limited resources: time, computation, money, energy, personnel, and attention.
Choosing one alternative necessarily means postponing another, and the postponed alternative continues changing. As the number of alternatives grows, so do the possible combinations of states, actions, and futures, making the calculation of a globally perfect sequence of decisions quickly impractical.
This is precisely where the extraordinarily elegant structure of the Gittins Index ceases to be sufficient.
From Gittins to Whittle
The Gittins Index works so well because an unselected alternative effectively waits. Once ignored alternatives continue evolving, that separation disappears: choosing Patient A now is no longer merely a decision about Patient A, because during the same interval Patients B, C, D, and all the others continue changing as well.
The value of choosing one option therefore begins to depend on the consequences of not choosing the others.
Peter Whittle approached this problem through an ingenious idea. Instead of attempting to solve the entire coupled system at once, we can imagine considering each alternative separately and asking how much compensation would be required for us to accept leaving it unattended.
Suppose there is a hypothetical reward for keeping a particular option passive. For an option that is not urgent, a small reward may be enough to make waiting acceptable. For an option whose condition may worsen rapidly, a much larger compensation would be required. The point at which acting and waiting become equally attractive provides the intuition behind the Whittle Index.
A high index therefore indicates that postponing that alternative carries a high cost. Under appropriate conditions, known as indexability, the alternatives can be ranked according to these indices and scarce resources allocated to those with the highest priorities.
Unlike the classical Gittins result, this strategy is not universally exact. Its value lies precisely in its practicality: it transforms an extremely complex global problem into several much smaller and potentially tractable local problems.
Consider three patients competing for a single specialized monitoring system. Patient A is stable; Patient B is in worse condition but usually changes slowly; Patient C appears only moderately ill, but experience suggests that deterioration, once it begins, can become severe very quickly.
A purely static ranking might select B because B looks worse at that moment. A restless policy may instead prioritize C because the cost of waiting is greater for C. An hour later, C may have stabilized while B has deteriorated, and the monitoring resource is then reassigned to B.
Priority changes because the patients change. From this emerges one of the most important ideas in the entire article:
priority is a property of a state at a particular moment, not a permanent property of an option.
From Restless Bandits to Adaptive AI
The restless-bandit perspective extends far beyond hospitals. A recommendation system must decide what deserves the limited space on a screen while user preferences continue changing. A financial system allocates attention and capital while markets shift between regimes. An autonomous vehicle distributes computational resources among pedestrians, other vehicles, road conditions, and potential hazards that continue moving while other parts of the environment are being analyzed.
Despite their differences, all these systems share the same structure: attention is scarce, and unattended reality does not freeze.
Reinforcement learning adds another dimension by recognizing that an agent’s own actions can alter future states. An intelligent system is therefore not merely observing a changing world; in many cases, it participates in changing that world.
Continual learning adds another requirement: the system must keep learning after deployment without destroying what it has already learned. This gives rise to the familiar stability–plasticity dilemma. Too much stability produces rigidity and prevents adaptation to new conditions; too much plasticity can cause the system to abandon useful knowledge whenever new information appears.
The deeper problem therefore ceases to be simply how to learn and becomes a set of three related decisions: what should be preserved, what should be revised, and what should be forgotten.
Greater computational power or representational capacity does not solve this problem automatically. A massive neural network can still be extraordinarily well adapted to yesterday’s world. A system may possess an enormous amount of knowledge and still fail because it cannot recognize that some of that knowledge has lost its validity.
Adaptive intelligence therefore requires something resembling metacognition: the system must not only maintain a model of the world, but also continuously evaluate whether that model still deserves to be trusted.
Large Language Models as Semantic Sensors
There is, however, another important difficulty: not every meaningful change first appears as a number.
Return once more to the hospital. The monitoring system receives heart rate, oxygen saturation, blood pressure, laboratory results, and medication information. These data have a well-defined structure and can be processed directly by statistical models.
Now imagine that a nurse writes:
“Patient seems unusually confused this morning despite stable vital signs.”
A physician notes that several similar cases have deteriorated unusually quickly. A pharmacy bulletin announces that a standard medication will temporarily need to be replaced, while a new scientific paper suggests that the circulating pathogen responds differently to the treatment represented in the hospital’s historical data.
None of these pieces of information is a conventional sensor measurement, but all of them may indicate that something important is changing. This is where an additional role for large language models becomes possible.
LLMs are usually presented as systems capable of answering questions, summarizing documents, translating text, writing code, and generating language. Within a broader adaptive architecture, however, they may also function as semantic sensors.
A conventional sensor measures a predefined quantity. A language model can analyze reports, clinical notes, regulations, scientific papers, messages, and other unstructured sources in order to identify claims about changes, contradictions, emerging risks, temporal relationships, and possible causal explanations.
Consider, for example, a system responsible for a supply chain. Its numerical indicators show inventory levels, delivery times, transportation costs, and port capacity, and none has yet crossed an alarm threshold. At the same time, however, suppliers begin mentioning unusual delays, local newspapers report labor unrest, and government statements suggest possible export restrictions.
None of these texts, taken alone, proves that a regime change has occurred. Taken together, however, they may provide an important early signal. An LLM could extract these pieces of evidence, identify their convergence, and convert them into a structured hypothesis that another component of the system could test.
This distinction is essential: the language model should not replace statistical inference; it should expand what the system is capable of observing.
A future adaptive architecture could therefore combine numerical sensing, semantic interpretation, state estimation, change detection, model revision, and decision-making within the same cycle:
observe → interpret → estimate → detect change → revise → reprioritize → act → observe again
In such a system, the LLM would not replace the Whittle Index. Its role would be to help determine which state and which version of the world the Whittle calculation should represent.
Naturally, this role requires caution. Language models can hallucinate, misjudge source reliability, confuse correlation with causation, or express excessive confidence. Semantic sensing therefore requires provenance, verification, calibration, representation of uncertainty, and comparison with independent evidence.
The proposal is not to hand the entire decision process to a language model. It is to use language as an additional source of evidence about parts of the world that conventional numerical sensors cannot directly observe.
The Future of Adaptive Intelligence
Traditional machine learning is often described as a relatively linear sequence: collect data, train a model, evaluate it, and deploy it. In this paradigm, deployment appears to be the destination. For adaptive intelligence, however, that is precisely where the harder problem begins.
Once deployed, the model enters an environment that continues evolving. Some of the knowledge acquired remains useful, some gradually loses relevance, and some relationships may become dangerously misleading. New evidence appears, new constraints emerge, and correlations that once seemed stable may disappear.
A genuinely adaptive system should therefore be evaluated not only by its accuracy at a particular moment, but by its behavior across time. It should be capable of recognizing meaningful change, distinguishing structural transformation from temporary fluctuation, adapting without destroying knowledge that remains valid, and reconsidering its priorities before the cost of waiting becomes excessive.
Classical optimization asks:
Given this model and these objectives, what is the best action?
Adaptive intelligence must ask a logically prior question:
Is this still the right model? Are these still the right objectives?
A system can solve the wrong problem with extraordinary precision. In that case, its efficiency does not make it more intelligent; it merely makes it more efficient at producing an answer for a reality that no longer exists.
Conclusion: The Golden Key
This series began by asking how intelligent decisions can be made before certainty is available. Part 2 transformed that question into the tension between exploration and exploitation, while Part 3 showed that, under special conditions, the future value of exploration could be compressed into the elegant logic of the Gittins Index.
Part 4 removes the final convenience: the alternatives do not wait. Patients change, machines deteriorate, markets shift between regimes, users alter their preferences, and scientific opportunities may disappear while our attention is directed elsewhere.
Once waiting has consequences, intelligence no longer means merely selecting the best available option. It becomes the continuous allocation of scarce attention among futures that are unfolding simultaneously.
The Gittins Index asks which opportunity deserves attention when the others wait. The Whittle Index asks which deserves attention when the others continue changing. Reinforcement learning asks how our actions reshape the future; continual learning asks how the agent itself can continue learning throughout those changes; and semantic sensing asks how a system can recognize that even the meaning of the evidence around it may have shifted.
These different lines of reasoning converge on the same conclusion:
the future is not merely uncertain. It is moving while we decide.
The most capable intelligent system will not be the one that finally discovers a perfect and definitive model of the world, because no model remains definitive in an environment that continues changing. It will be the system capable of recognizing when its model begins to lose validity, reallocating attention before the cost of delay becomes too high, preserving what remains useful, abandoning what no longer applies, and continuing to learn without losing the ability to act.
The future, therefore, will not belong simply to systems that know more. It will belong to systems that know when their knowledge has expired.
That is the golden key connecting every article in this series.
And that is adaptive intelligence.
References
Adaptive Intelligence, Part 1 — Why Every Intelligent Decision Begins Before Certainty
Read Part 1 on AI-Talks.org
Adaptive Intelligence, Part 2 — Classical Multi-Armed Bandit: Learning Under Uncertainty
Read Part 2 on AI-Talks.org
Adaptive Intelligence, Part 3 — The Gittins Index: One of the Most Elegant Algorithms Ever Created
Read Part 3 on AI-Talks.org
Gama, João, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. “A Survey on Concept Drift Adaptation.” ACM Computing Surveys 46 (4).
Gittins, John C. 1979. “Bandit Processes and Dynamic Allocation Indices.” Journal of the Royal Statistical Society: Series B 41 (2): 148–177.
Gittins, John C., and David M. Jones. 1974. “A Dynamic Allocation Index for the Sequential Design of Experiments.” In Progress in Statistics, edited by J. Gani. Amsterdam: North-Holland.
Khetarpal, Khimya, Matthew Riemer, Irina Rish, and Doina Precup. 2022. “Towards Continual Reinforcement Learning: A Review and Perspectives.” Journal of Artificial Intelligence Research.
Lesort, Timothée, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez. 2020. “Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges.” Information Fusion 58: 52–68.
Papadimitriou, Christos H., and John N. Tsitsiklis. 1999. “The Complexity of Optimal Queueing Network Control.” Mathematics of Operations Research 24 (2): 293–305.
Weber, Richard R., and Gideon Weiss. 1990. “On an Index Policy for Restless Bandits.” Journal of Applied Probability 27 (3): 637–648.
Whittle, Peter. 1988. “Restless Bandits: Activity Allocation in a Changing World.” Journal of Applied Probability 25A: 287–298.
Editorial transparency note: This article, as with all articles published on this site, was conceived, directed, written, and reviewed by Prof. Maurício Veloso Brant Pinheiro. Artificial intelligence was used as an assistant for editorial refinement, formatting, image generation, SEO metadata, and publication workflow.
Copyright 2026 AI-Talks.org
