2026-09-19T22:08:09Zhttps://api.repository.cam.ac.uk/server/oai/request

oai:www.repository.cam.ac.uk:1810/3883322025-08-19T09:47:27Zcom_1810_223848com_1810_256065col_1810_223849

Dynamic Discrete Choice Estimation using Reinforcement Learning with Applications in Online Food Markets https://doi.org/10.17863/CAM.120730 Srivastava, Sonal Prabhu, Jaideep Dynamic Discrete Choice (DDC) models are widely used to analyze sequential decision-making in economics and marketing. However, their estimation remains computationally challenging, especially as state spaces expand, limiting their application to large-scale consumer datasets. This thesis develops Reinforcement Learning (RL)-based estimation methods to improve the speed and scalability of DDC estimation. The second chapter establishes a theoretical foundation for integrating RL with DDC estimation, emphasizing the shared mathematical structure of Markov Decision Processes (MDPs) in both frameworks. It reviews key tabular RL methods including Dynamic Programming (DP), Monte Carlo (MC), Temporal Difference (TD) and Q-learning, and draws conceptual parallels with DDC methods including Nested Fixed-Point Algorithm (NFXP), Conditional Choice Simulation (CCS), and Nested Pseudo-Likelihood (NPL). The chapter discusses how incorporating both tabular RL methods and those using function or policy approximation into the DDC estimation process can reduce computation time and improve scalability in high-dimensional settings. It also discusses additional RL approaches such as Prioritized Sweeping (PS), Inverse Reinforcement Learning (IRL), and Learning-to-Optimize (L2O) as potential tools for further improving computational efficiency. Building on this, the third chapter introduces the Reinforcement Learning Temporal Difference based Condition Choice Simulation (RLTD-CCS) algorithm. The algorithm leverages forward simulations (CCS) and stepwise TD learning to update value functions more efficiently than CCS. Monte Carlo studies on a small state space machine replacement model and a large state space prototype food choice model confirm that RLTD-CCS achieves estimation accuracy comparable to CCS while being up to 14 times faster, making it a viable approach for scalable DDC estimation. Additionally, in the fourth chapter, RLTD-CCS is embedded within the Expectation-Maximization (EM) algorithm to accommodate models with time-invariant persistent unobservables. In the fifth chapter, the proposed RL-based approach is applied to real-world consumer decision-making using weekly purchase data from a UK recipe box provider. The empirical analysis explores how nutritional attributes (protein and carbohydrate content) and habit persistence shape recipe box selection over time. Results indicate that nutritional factors are the primary drivers of choice, with consumers strongly preferring high-protein meals while avoiding carbohydrate-heavy options. A time-segmented static analysis further reveals a post-COVID shift: demand for high-protein meals increased, while carbohydrate aversion intensified among loyal consumers, suggesting lasting changes in dietary preferences. Counterfactual simulations assess the impact of nutritional composition adjustments and habit persistence modifications, offering actionable insights for business strategy and consumer retention. Based on these findings, recommendations are provided for optimizing menu offerings, encouraging repeat purchases, and improving personalization strategies. This thesis advances the integration of machine learning and structural econometrics by demonstrating the potential of RL-based methods in DDC estimation. By highlighting opportunities to further incorporate RL into DDC modeling, it paves the way for more scalable and computationally efficient approaches to estimating complex structural choice models. University of Cambridge 2025-02-18 Thesis Doctoral Doctor of Philosophy (PhD) eng https://www.repository.cam.ac.uk/handle/1810/388332 https://www.repository.cam.ac.uk/bitstreams/b7a44324-428d-47bb-81bd-b3d784e1223d/download d16a4f8aefb01f10c543e4dc95aba817 https://www.repository.cam.ac.uk/bitstreams/cd66ceb7-09e4-437e-9c58-b3ca7d2e5ad9/download 87eda9de84448d1f82354d60eee3eb5f http://purl.org/NET/rdflicense/allrightsreserved Conditional Choice Simulation Dynamic Discrete Choice Food Choice Modeling Reinforcement Learning Two-step Estimation