검색 상세

Sample Efficient Reinforcement Learning using Dynamics Modeling

동역학 모델링을 이용한 샘플 효율적인 강화학습

  • Jaehoon Kim (김재훈, 대학원 산업경영공학과 산업경영공학전공)

초록/요약

심층 강화학습(deep reinforcement learning)에서 에이전트가 효과적인 정책을 학습하기 위해 수백만 번의 상호작용을 필요로 한다는 점에서, 샘플 비효율성(sample inefficiency)은 여전히 핵심적인 난제로 남아있다. 이러한 한계를 극복하기 위해, 본 학위논문은 데이터 수집이 제한된 상황에서 동역학 모델링(dynamics modeling)을 활용하여 샘플 효율성을 향상시키는 두 가지 프레임워크를 제안한다. 첫째, 이중 환경 동역학 모델링과 순차적 시계열 정보를 사용하여 연속된 상호작용 간의 상관관계(trajectory correlations)를 포착함으로써 상태 표현 학습(state representation learning)을 강화한다. 둘째, 동역학 모델링을 통해 정량화된 불확실성(quantified uncertainty)을 기반으로 미니배치(minibatch)를 구성하는 온라인 및 오프라인 데이터의 비율을 동적으로 조절하는 혼합 학습 전략을 제안한다. 본 학위논문은 동역학 모델링을 통하여 표현 학습과 불확실성 기반 학습을 적용함으로써 효율적인 심층 강화학습 연구에 기여하였으며, 데이터 수집 비용이 높거나 제약이 있는 분야에 심층 강화학습을 구현하기 위한 실용적인 방식을 제안한다.

more

초록/요약

Sample inefficiency remains a central challenge in deep reinforcement learning (DRL), where agents typically require millions of interactions to learn effective policies. To overcome this limitation, this dissertation introduces two frameworks that use dynamics modeling to improve sample efficiency when data collection is constrained. First, I enhance state representation learning by incorporating dual environmental dynamics and sequential temporal information to capture trajectory correlations. Second, I propose a hybrid learning strategy that dynamically adjusts the ratio of online and offline data to form the minibatch based on uncertainty quantified through dynamics modeling. In summary, this dissertation contributes to efficient DRL by improving representation learning and uncertainty-aware training using dynamics modeling, providing a practical solution for implementing DRL in domains where data collection is costly or constrained.

more

목차

Abstract i
국문 초록 ii
Contents iii
List of Tables vi
List of Figures vii

1 Introduction 1
1.1 Deep Reinforcement Learning 1
1.2 Dynamics Modeling for sample efficient Reinforcement Learning 4
1.3 Integrating Offline Data for sample efficient Reinforcement Learning 7
1.4 Outline of Dissertation 9

2 DynaSTI: Dynamics Modeling with Sequential Temporal Information for Reinforcement Learning in Atari 11
2.1 Introduction 11
2.2 Related Works 13
2.2.1 Sample-Efficient Reinforcement Learning 13
2.2.2 Dynamics Modeling in Reinforcement Learning 15
2.3 Proposed Method 16
2.3.1 Deep Q-Learning 17
2.3.2 Stacked Dynamics Module 18
2.4 Experiments 22
2.5 Ablation Study and Hyperparameter Selection 27
2.5.1 Effectiveness of DynaSTI Components 28
2.5.2 Study on Sequence Length 32
2.6 Conclusion 36

3 ABaRL: Uncertainty-Aware Adaptive Batching for Sample Efficient Reinforcement Learning with Unlabeled Offline Data 38
3.1 Introduction 38
3.2 Related Works 42
3.2.1 Using Offline Data for Sample Efficient Reinforcement Learning 42
3.2.2 Sample Efficient Reinforcement Learning with Semi- and Self-Supervised Learning .43
3.3 Proposed Method 46
3.3.1 Adaptive Batching Reinforcement Learning 46
3.3.2 Reward Estimator with Uncertainty Quantification 48
3.4 Experimental Results & Analysis 54
3.4.1 Experiment Settings 54
3.4.2 Performance Comparison Experiments 55
3.4.3 Effectiveness of ABaRL Components 61
3.4.4 Effectiveness of Ensemble Numbers 65
3.4.5 Effectiveness of Batch Ratio Strategy 66
3.4.6 Study on Batch Ratio Dynamics 67
3.5 Conclusion 69

4 Summary and Future Studies 71

Reference 75

more
반출 Meta View 목록