Counterfactual reasoning is a powerful idea from reinforcement learning (RL) that allows us to evaluate new candidate policies for a system, without actually deploying those policies. Traditionally, this is done by collecting randomized data from an existing policy and matching this data against the decisions of a candidate policy. In many systems, we observe that a special kind of information exists that can boost the power of counterfactual reasoning. Specifically, system policies that make threshold decisions involving a resource (e.g., time, memory, cores) naturally reveal additional, or implicit feedback about alternative decisions. For example, if a system waits X min for an event to occur, then it automatically learns what would have happened if it waited

Bio: Siddhartha Sen is a Principal Researcher in the Microsoft Research New York City lab. His research trajectory started with distributed systems and data structures, evolved to incorporate machine learning, and is currently most inspired by humans. His current mission is to use AI to design human-oriented and human-inspired systems that advance human skills and empower them to achieve more. Siddhartha received his BS/MEng degrees in computer science and mathematics from MIT, then worked for three years as a developer in Microsoft’s Windows Server team before returning to academia to complete his PhD from Princeton University. Siddhartha’s work on data structures and human/AI gaming has been featured in several textbooks and podcasts.