Nov 2025 – Jan 2026 · Reinforcement learning research

TQ-IoT — Multi-Objective RL for IoT Duty Cycling

Tabular multi-objective reinforcement learning scheduler for IoT devices. Balances sensing quality against battery life via Chebyshev scalarization, learning adaptive duty-cycle policies that outperform fixed baselines.

Python
NumPy
Matplotlib
Google Colab
Q-Learning
Chebyshev Scalarization

Last updated January 31, 2026

Thesis (DiVA): diva-portal.org/smash/record.jsf?pid=diva2:2032522

Context

Bachelor thesis, Stockholm University. A Design Science Research (DSR) project building an adaptive artifact — TQ-IoT — that replaces the fixed sleep/wake schedules used by most battery-powered IoT sensors with a policy that learns when to sleep, sense, or transmit.

The core conflict: IoT networks want high sensing quality and long battery life, but these objectives pull in opposite directions. Static duty cycles can't respond to dynamic environments where event likelihood shifts over time.

The artifact

Reinforcement learning agent–environment interaction loop — The RL loop — agent observes state, picks an action, receives reward, environment transitions.

A tabular multi-objective reinforcement learning (MORL) agent that picks one of three actions per step:

Sleep — low energy, no data
Sense — medium energy, local observation
Sense + Transmit — high energy, remote report

The agent's state encodes binned battery level, a 3-state Markov chain over event likelihood (Low / Medium / High), and recent sensing outcomes.

Algorithm

Framework — value-based MORL over a discrete MDP.
Learning rule — Q-learning with temporal-difference updates; ε-decay schedule to shift from exploration to exploitation as training converges.
Scalarization — Chebyshev over the (quality, energy) objective vector. Chosen over linear scalarization because it can recover Pareto-optimal policies in non-convex regions of the front — the exact regime where IoT trade-offs tend to live.
Reward — negative of the weighted Chebyshev distance to a utopian point; sweeping the preference weight traces out the Pareto front.

Results

Pareto efficiency — intermediate weight settings produced policies with markedly higher sensing-quality-per-joule than any fixed-rule baseline.
Adaptive behavior — the agent learned to bank energy during low-event periods so the battery is available for high-likelihood event bursts, a strategy no static duty cycle can replicate.
Evaluation — measured via device lifetime, event-detection rate, and cumulative training regret across weight sweeps.

Pareto front of device lifetime vs. sensing quality — Pareto front — TQ-IoT (learned) vs. fixed-rule baselines. Intermediate weights dominate the static schedules across the lifetime / quality trade-off.

Action distribution by preference weight — Action mix as preferences shift — the agent morphs from "always sleep" (energy-heavy weight) to "always sense + transmit" (quality-heavy weight). Scalarization made visible.

Code

Chebyshev scalarization — the hero of the algorithm. Two competing objectives collapse to a single score by minimizing the weighted maximum distance to the running utopia point, which is what lets the agent reach non-convex Pareto regions that linear scalarization can't touch.

def chebyshev_scores(self, s: int, w: Tuple[float, float]):
    # Utopia point: max value seen so far in each objective
    zE, zQ = np.max(self.QE[s]), np.max(self.QQ[s])
 
    # Distance from each action's value to the utopia point
    dE, dQ = zE - self.QE[s], zQ - self.QQ[s]
    wE, wQ = w
 
    # Minimize the weighted worst-case gap
    return -np.maximum(wE * dE, wQ * dQ)

Dual Q-updates — two Q-tables, one per objective, updated in lockstep via standard temporal-difference learning. Keeping the objectives separate is what makes Pareto-front sweeping cheap: change weights, re-score, done. No re-training.

def update(self, s: int, a: int, rE: float, rQ: float, s_next: int):
    g, alpha = self.cfg.gamma, self.cfg.alpha
 
    # Energy Q-table
    self.QE[s, a] += alpha * ((rE + g * np.max(self.QE[s_next])) - self.QE[s, a])
 
    # Quality Q-table
    self.QQ[s, a] += alpha * ((rQ + g * np.max(self.QQ[s_next])) - self.QQ[s, a])

Environment step — the MDP made concrete. Battery is a hard constraint, events fire stochastically from the current likelihood level, and the level itself evolves via a Markov transition matrix.

def step(self, a: int):
    cost = {0: self.cfg.energy_cost_sleep,
            1: self.cfg.energy_cost_sense,
            2: self.cfg.energy_cost_sense_tx}[a]
 
    if self.battery < cost:
        return self._obs(), (-float(self.battery), 0.0), True, {"event": False}
 
    self.battery -= cost
 
    p_event = self.cfg.event_probs[self.level]
    event_occurs = (self.rng.random() < p_event)
 
    # ... reward calculation based on detection ...
 
    # Evolve likelihood via Markov transition matrix
    self.level = int(np.argmax(self.np_rng.multinomial(1, self.cfg.trans[self.level])))
 
    return self._obs(), (r_E, r_Q), done, info

Stack

Python, NumPy, Matplotlib. Prototyped and trained in Google Colab. Methodology: Design Science Research.

Takeaways

Multi-objective RL isn't just single-objective RL with a bigger reward — the scalarization choice is the research contribution. Chebyshev's ability to reach non-convex Pareto regions was the difference between a toy result and one that dominated the baseline. Tabular methods kept the search transparent: every policy the agent found could be read directly out of the Q-table and explained.