Zhefan Ye. 2026.
Crowd-in-the-Loop Reinforcement Learning.
PhD thesis, Computer Science & Engineering, University of Michigan, 2026.
This dissertation explores novel methods for leveraging human input to expand the scope and scale of interactive tasks that reinforcement learning (RL) is able to address. While RL provides a general framework for solving sequential decision-making problems, its generality comes at the expense of high training costs due to the fundamental difficulty of disentangling conflated signals from actions taken in an environment. RL requires both a well-defined reward function and a specification of the agent's basic abilities to sense and act within an environment. However, turning these primitives into a model that captures cause and effect, particularly when the reward signal is temporally separated from the action by an unknown delay, requires a vast number of examples that scale dramatically with the complexity of the problem domain. This dissertation aims to leverage the pre-existing knowledge that humans possess across a diverse range of domains, analogous to how people learn from others via social learning, to make RL more practical in a broader spectrum of settings.
To make this guidance maximally scalable, available, and robust, this dissertation seeks to elicit feedback from dynamic groups of people (crowds) with minimal assumptions about their expertise. By employing crowds to provide otherwise missing signals, this work broadens the applicability of RL through two pathways: reward specification and relevance guidance. In Part I, we introduce a method for eliciting and constructing reward functions using patterns, demonstrating that non-experts can specify robust rewards via this approach. In Part II, we present methods for capturing general knowledge via the concept of relevance, utilizing human intuition to refine temporal credit assignment and construct state representations. These methods collectively demonstrate that human insight, when properly structured, can effectively supplement existing RL algorithms.