In reinforcement learning (RL), training an agent to master a new task typically requires millions of pieces of environment interaction. If an agent learns how to "find a red object" and "pick up a block" independently, traditional RL requires entirely new training cycles to teach it to "find a red block."
To achieve true lifelong learning, an agent should instead be able to treat its existing skills as reusable primitives, combining them on the fly to solve novel, complex tasks without further training. Early approaches to this problem, such as work by Jonathan J. Hunt et al. in Entropic Policy Composition with Generalized Policy Improvement and Divergence Correction, demonstrated that agents can approximate optimal behavior for new objectives by expressing reward functions as convex combinations of base tasks. Similarly, Benjamin van Niekerk et al. in Composing Value Functions in Reinforcement Learning showed that closed-form composition of value functions could instantly solve disjunctive goals.
The frontier of this research direction formalizes these combinations into a strict Boolean Task Algebra, allowing agents to compose skills using the full expressive power of logical operators: AND (conjunction), OR (disjunction), and NOT (negation).
The Intuition: Goals as Sets, Values as Bounds
Why can logical operations on high-level tasks be computed directly from low-level value functions? The core insight is to treat a task's goal as a set of desirable states in the environment. Under this framing, logical operations on goals map directly to standard set operations:
- Conjunction (A AND B) corresponds to the intersection of goal states.
- Disjunction (A OR B) corresponds to the union of goal states.
- Negation (NOT A) corresponds to the complement of goal states (i.e., safety constraints or avoidance).
Value functions (such as $Q$-functions) quantify the expected long-term return or reachability of these goal states. As a result, set operations on goals translate directly to mathematical bounds on their corresponding value functions.
As shown by Geraud Nangue Tasse et al. in Generalisation in Lifelong Reinforcement Learning through Logical Composition, if an agent wants to satisfy both Task A and Task B simultaneously, its expected performance is fundamentally constrained by the harder of the two objectives. Thus, the value function for the joint task can be approximated by taking the pointwise minimum of the individual value functions:
$$Q_{A \wedge B}(s, a) \approx \min\big(Q_A(s, a), Q_B(s, a)\big)$$
Conversely, if the task is to satisfy Task A or Task B, the agent can simply pursue whichever goal is currently more optimal or closer to its current state. This behavior maps directly to the pointwise maximum:
$$Q_{A \vee B}(s, a) \approx \max\big(Q_A(s, a), Q_B(s, a)\big)$$
By framing value functions within lattice structures, the authors proved in Composition and Zero-Shot Transfer with Lattice Structures in Reinforcement Learning that optimal value functions for logically composed tasks can be computed directly from pre-trained World Value Functions (WVFs) using these exact lattice operations ($\min$ and $\max$) and linear transformations.
The Combinatorial Explosion of Capabilities
When task composition is formalized as a true Boolean algebra, the agent's capabilities scale super-exponentially. In A Boolean Task Algebra for Reinforcement Learning., Geraud Nangue Tasse et al. demonstrated that an agent that learns a small set of $K$ base tasks can theoretically solve up to $2^{2^K}$ distinct logical variations zero-shot. For instance, learning just 7 base tasks allows an agent to instantly synthesize optimal policies for 121 distinct goal-reaching tasks without a single additional environment transition.
This algebraic framework is not restricted to simple, discrete environments. Kevin Leahy et al. demonstrated in Run-Time Task Composition with Safety Semantics that Boolean task composition can be successfully deployed in continuous action spaces. By evaluating and comparing the value functions of primitive policies at runtime, their framework allows physical systems to satisfy multi-objective constraints—such as navigating to a target while dynamically avoiding hazardous regions—zero-shot.
Scaling to Temporal Logic and Language
While static Boolean logic can handle simultaneous constraints, real-world tasks often require sequencing over time (e.g., "go to room A, then pick up the block, but never enter room C").
To handle these long-horizon instructions, Geraud Nangue Tasse et al. introduced Skill Machines in Skill Machines: Temporal Logic Composition in Reinforcement Learning. This framework maps high-level Linear Temporal Logic (LTL) formulas directly to finite state machines, where each state transition executes a specific sequence of logically composed value functions. This ensures that the zero-shot controllers are provably satisficing, guaranteeing task completion if the goal is reachable.
To make these systems practical, researchers have integrated Large Language Models (LLMs) to act as compilers that translate natural language into algebraic expressions. In Compositional Instruction Following with Language Models and Reinforcement Learning, Vanya Cohen et al. developed CERLLA, a framework where an LLM parses human instructions into explicit Boolean expressions. These expressions then automatically combine pre-trained base value functions, allowing the agent to follow complex, novel instructions with massive data efficiency.
Beyond Reinforcement Learning: Conceptor Algebra
The power of Boolean composition extends beyond control policies and value functions. In Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineering, Joris Postmus et al. applied these exact algebraic principles to control the internal representations of Large Language Models.
Instead of using simple vector addition to steer LLM behavior, the authors represented steering targets as matrices called "conceptors," which define ellipsoidal regions of activation space. Because these matrices form a Boolean algebra, researchers can apply mathematical AND, OR, and NOT operations to combine steering goals. This allows for highly precise, multi-goal behavioral steering that far outperforms traditional arithmetic averaging, proving that Boolean representation algebra is a foundational tool for zero-shot composition across AI paradigms.
Key Foundational Work
Composing Value Functions in Reinforcement Learning (2019) — contributed 1 concept elements to this direction.
Entropic Policy Composition with Generalized Policy Improvement and Divergence Correction. (2018) — contributed 1 concept elements to this direction.