A Reinforcement Learning Framework for Optimizing AgeofInformation in RFpowered Communication Systems
Abstract
In this paper, we study a realtime monitoring system in which multiple source nodes are responsible for sending update packets to a common destination node in order to maintain the freshness of information at the destination. Since it may not always be feasible to replace or recharge batteries in all source nodes, we consider that the nodes are powered through wireless energy transfer (WET) by the destination. For this system setup, we investigate the optimal online sampling policy (referred to as the ageoptimal policy) that jointly optimizes WET and scheduling of update packet transmissions with the objective of minimizing the longterm average weighted sum of AgeofInformation (AoI) values for different physical processes (observed by the source nodes) at the destination node, referred to as the sumAoI. To solve this optimization problem, we first model this setup as an average cost Markov decision process (MDP). Due to the extreme curse of dimensionality in the state space of the formulated MDP, classical reinforcement learning algorithms are no longer applicable to our problem. Motivated by this, we propose a deep reinforcement learning (DRL) algorithm that can learn the ageoptimal policy in a computationallyefficient manner. We further characterize the structural properties of the ageoptimal policy analytically, and demonstrate that it has a thresholdbased structure with respect to the AoI values for different processes. We extend our analysis to characterize the structural properties of the policy that maximizes average throughput for our system setup, referred to as the throughputoptimal policy. Afterwards, we analytically demonstrate that the structures of the ageoptimal and throughputoptimal policies are different. We also numerically demonstrate these structures as well as the impact of system design parameters on the optimal achievable average weighted sumAoI.
 Publication:

arXiv eprints
 Pub Date:
 August 2019
 arXiv:
 arXiv:1908.06367
 Bibcode:
 2019arXiv190806367A
 Keywords:

 Computer Science  Information Theory;
 Computer Science  Networking and Internet Architecture