We propose a model-free deep reinforcement learning method that leverages a small amount of demonstration data to assist a reinforcement learning agent. 作者在文章中提出了一种基于自由的深度强化学习方法，帮助一个强化学习智能体(训练)的数据较少。

For robotics, RL in combination with powerful function approximators such as neural networks provides a general framework for designing sophisticated controllers that would be hard to handcraft otherwise. 结合强大的函数近似，如神经网络，强化学习可以提供一个通用的计算框架来设计难以手工设计的控制器。

Nevertheless, end-to-end learning of visuomotor controllers for long-horizon and multi-stage manipulation tasks using model-free RL techniques remains a challenging problem. 使用无模型强化学习技术来处理移动视觉控制器的端到端学习，并在长期、多阶段的操作任务中使用，仍然是一个难题。

Three classes of RL algorithms are currently dominant for continuous control problems: guided policy search methods (GPS; Levine and Koltun [22]), value-based methods such as the deterministic policy gradient (DPG; Silver et al. [45], Lillicrap et al. [26], Heess et al. [12]) or the normalized advantage function (NAF; Gu et al. [10]) algorithm, and trust-region based policy gradient algorithms such as trust region policy optimization (TRPO [42]) and proximal policy optimization (PPO [43]). 在连续环境主流的三种强化学习算法类： 1、guided policy search methods——“引导性策略搜索模型”

[22] Sergey Levine and Vladlen Koltun. Guided policy search. In ICML, pages 1–9, 2013.

2、基于价值的方法 the deterministic policy gradient——确定性策略梯度

[45] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.

[26] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.

Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In NIPS, pages 2926–2934, 2015.

3、基于置信区的策略梯度算法 the normalized advantage function algorithm——标准优势函数算法；

[10] Shixiang Gu, Tim Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep Q-learning with model-based acceleration. In ICML, 2016.

trust region policy optimization——置信区策略最优；

[42] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pages 1889–1897, 2015.

proximal policy optimization——近似策略最优；

[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

TRPO [42] and PPO [43] hold appeal due to their robustness to hyperparameter settings as well as their scalability [14] but the lack of sample efficiency makes them unsuitable for training directly on robotics hardware. TRPO和PPO具有较高的吸引力是因为他们的超参数的设定具有鲁棒性以及他们具有可测量性；但是采样有效性比较少（【思考】可能是数据不能充分训练？）使得他们不能直接训练硬件机器人。

The idea of using large-scale data collection for training visuomotor controllers has been the focus of Levine et al. [24] and Pinto and Gupta [33] who train a convolutional network to predict grasp success for diverse sets of objects using a large dataset with 10s or 100s of thousands of grasp attempts collected from multiple robots in a self-supervised setting. 用大规模数据收集来训练移动视觉控制器的想法： 训练一个CNN，用来预测抓取的成功； 采用的是多个机器人在自监督设置下生成的10秒或者100秒的大量抓取试图数据集；

Demonstrations can be used to initialize policies, design cost functions, guide exploration, augment the training data, or a combination of these. 演示数据可以被用来初始化策略、设计损失函数、引导探索和增加训练数据，甚至是这些的结合体

Our method learns end-to-end visuomotor policies without reliance on demonstrator actions. 我们的方法是学习端到端的移动视觉策略，这个移动视觉策略不需要依赖演示者的行为动作。因此不同演示者操作也可以。

The policy takes both an RGB camera observation and a proprioceptive feature vector that describes the joint positions and angular velocities. 一个RGB相机的观测 一个机器人关节信息的向量 采用这样的原因在与实体机器人也这样做，因此不需要进行额外的信息转变

图像经过CNN处理，主观触觉经过MLP处理 提取之后的信息连接好后，传入LSTM模型

启发式学习(Imitation Learning)：

假设：人类的演示是一种动作状态对的数据集。 D = { s i , a i } , i = 1 , 2 , 3 ⋯ N D=\lbrace s_{i},a_{i} \rbrace,i=1,2,3 \cdots N D={ si​,ai​},i=1,2,3⋯N 行为克隆算法是将人类演示的数据集做监督学习问题。 这个算法使用最大似然估计来训练一个参数化的策略 π \pi π。 π θ : S → A \pi_{\theta}:S \rightarrow A πθ​:S→A 那么这样的话，最优参数表示如下： θ ∗ = a r g m a x θ ∑ N l o g θ π ( a i ∣ s i ) \theta^{*}=argmax_{\theta} \sum_{N}log_{\theta} \pi(a_{i}|s_{i}) θ∗=argmaxθ​N∑​logθ​π(ai​∣si​) 行为克隆在数据集充分的情况下好用 但是，机器人数据集的收集耗时耗力，因此需要较少量的数据集实现学习

既使用了演示的数据 有允许智能体学习从环境中交互的经验 GAIL使用两个网络： 一个是策略网络： π θ : S → A \pi_{\theta}:S \rightarrow A πθ​:S→A 一个是鉴别网络： D ψ : S × A → [ 0 , 1 ] D_{\psi}:S\times A \rightarrow [0,1] Dψ​:S×A→[0,1] 使用最大值-最小值目标函数： max ⁡ θ min ⁡ ψ E π E [ l o g D ψ ( s , a ) ] + E π θ [ l o g ( 1 − D ψ ( s , a ) ) ] \operatorname{max}_\theta \operatorname{min}_\psi E_{\pi_{E}}[logD_{\psi}(s,a)] + E_{\pi_{\theta}}[log(1-D_{\psi}(s,a))] maxθ​minψ​EπE​​[logDψ​(s,a)]+Eπθ​​[log(1−Dψ​(s,a))] π E \pi_{E} πE​指的是演示轨迹生成的专家的策略； 目标函数的目标是尽可能让智能体的策略尽可能与专家的策略一致 训练 π θ \pi_{\theta} πθ​的方法：用策略梯度的方法最大化一个奖励函数 r g a i l = − l o g ( 1 − D ψ ( s t , a t ) ) r_{gail}=-log(1-D_{\psi}(s_{t},a_{t})) rgail​=−log(1−Dψ​(st​,at​))，使用限幅函数限制在最大值是10 GAIL通常是和TRPO结合起来使用 3. TRPO和PPO PPO only relies on first-order gradients and can be easily implemented with recurrent networks in a distributed setting [14]. PPO只需要一阶梯度，可以很简单的在循环神经网络中进行运用

PPO implements an approximate trust region that limits the change in the policy per iteration. PPO采用近似置信区来限制每次迭代的策略的改变

This is achieved via a regularization term based on the Kullback-Leibler (KL) divergence, the strength of which is adjusted dynamically depending on actual change in the policy in past iterations. 这个通过一个基于KL散度的正则化项，这个正则化项依靠在过去策略的迭代中实际的改变来动态的调整

Hence, we design the task rewards as sparse piecewise constant functions based on the different stages of the respective tasks. 根据任务的不同阶段，设置分段的常值奖励函数 奖励值的变换，只在任务从一个状态像另一个状态的改变

we provide additional guidance via a hybrid reward function that combines the imitation reward r g a i l r_{gail} rgail​ with the task reward r t a s k r_{task} rtask​. 在任务中增加了混合的奖励函数 结合了启发奖励和任务的奖励 r ( s t , a t ) = λ r g a i l ( s t , a t ) + ( 1 − λ ) r t a s k ( s t , a t ) , λ ∈ [ 0 , 1 ] r(s_{t},a_{t})=\lambda r_{gail}(s_{t},a_{t})+(1-\lambda)r_{task}(s_{t},a_{t}),\lambda \in [0,1] r(st​,at​)=λrgail​(st​,at​)+(1−λ)rtask​(st​,at​),λ∈[0,1] where the imitation reward encourages the policy to generate trajectories closer to demonstration trajectories, and the task reward encourages the policy to achieve high returns on the task. 启发式奖励——鼓励生成的策略轨迹更接近于人的演示轨迹 任务式奖励——鼓励策略获得更多的回报

设置奖励函数的结果——获得的成绩比单强化学习/单启发式学习都要好

The physics simulator we employ for training exposes the full state of the system. 用系统的全状态进行物理仿真

Previous work indicates that shaping the distribution of start states towards states where the optimal policy tends to visit can greatly improve policy learning [18, 35]. 先前的工作指出：对开始状态到最优策略即将达到的状态之间进行塑造可以很大程度提升策略学习 从演示实验中获得开始状态的分布 对于不同的任务，分成不同个curriculums 初始状态选择策略： ϵ \epsilon ϵ：在环境中随机选取状态 1 − ϵ 1-\epsilon 1−ϵ：在环境中选取演示起点的状态

During training, each PPO worker executes the policy for K steps and uses the discounted sum of rewards and the value as an advantage function estimator A t ∧ = ∑ i = 1 K γ i − 1 r t + i + γ K − 1 V ϕ ( s t + K ) − V ϕ ( s t ) \stackrel{\wedge}{A_{t}} = \sum_{i=1}^{K} \gamma^{i-1}r_{t+i} + \gamma^{K-1} V_{\phi}(s_{t+K}) -V_{\phi}(s_{t}) At​∧​=i=1∑K​γi−1rt+i​+γK−1Vϕ​(st+K​)−V