{"id":70380,"date":"2026-02-01T22:05:59","date_gmt":"2026-02-01T14:05:59","guid":{"rendered":"https:\/\/www.wsisp.com\/helps\/70380.html"},"modified":"2026-02-01T22:05:59","modified_gmt":"2026-02-01T14:05:59","slug":"%e8%ae%a9-q-%e5%80%bc%e4%bc%b0%e8%ae%a1%e6%9b%b4%e5%87%86%e7%a1%ae%ef%bc%9a%e4%bb%8e-dqn-%e5%88%b0-double-dqn-%e7%9a%84%e6%94%b9%e8%bf%9b%e6%96%b9%e6%a1%88","status":"publish","type":"post","link":"https:\/\/www.wsisp.com\/helps\/70380.html","title":{"rendered":"\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848"},"content":{"rendered":"

DQN \u7528<\/p>\n

max Q(s',a')<\/p>\n

\u8ba1\u7b97\u76ee\u6807\u503c，\u7b49\u4e8e\u5728\u6311 Q \u503c\u6700\u9ad8\u7684\u52a8\u4f5c，\u4f46\u662f\u8fd9\u4e9b\u52a8\u4f5c\u4e2d\u5305\u62ec\u4e86\u90a3\u4e9b\u56e0\u4e3a\u4f30\u8ba1\u566a\u58f0\u800c\u88ab\u9ad8\u4f30\u7684\u52a8\u4f5c，\u7d20\u4ee5\u5c31\u4f1a\u4ea7\u751f\u8fc7\u4f30\u8ba1\u504f\u5dee，\u76f4\u63a5\u540e\u679c\u662f\u8bad\u7ec3\u4e0d\u7a33\u5b9a\u3001\u7b56\u7565\u6b21\u4f18\u3002<\/p>\n

\u8fd9\u7bc7\u6587\u7ae0\u8981\u89e3\u51b3\u7684\u5c31\u662f\u8fd9\u4e2a\u95ee\u9898，\u5185\u5bb9\u5305\u62ec：DQN \u4e3a\u4ec0\u4e48\u4f1a\u8fc7\u4f30\u8ba1\u3001Double DQN \u600e\u4e48\u628a\u52a8\u4f5c\u9009\u62e9\u548c\u8bc4\u4f30\u62c6\u5f00\u3001Dueling DQN \u600e\u4e48\u5206\u79bb\u72b6\u6001\u503c\u548c\u52a8\u4f5c\u4f18\u52bf\u3001\u4f18\u5148\u7ecf\u9a8c\u56de\u653e\u5982\u4f55\u8ba9\u91c7\u6837\u66f4\u806a\u660e，\u4ee5\u53ca\u7528 PyTorch \u4ece\u5934\u5b9e\u73b0\u8fd9\u4e9b\u6539\u8fdb\u3002\u6700\u540e\u8fd8\u4f1a\u4ecb\u7ecd\u4e00\u4e2a CleanRL \u7684\u4e13\u4e1a\u5b9e\u73b0\u3002 \"\"<\/p>\n

\u8fc7\u4f30\u8ba1\u95ee\u9898<\/h3>\n

DQN \u7684\u76ee\u6807\u503c\u5982\u4e0b：<\/p>\n

y = r + \u03b3\u00b7max\u2090' Q(s', a'; \u03b8\u207b)<\/p>\n

\u95ee\u9898\u5c31\u5728\u4e8e，\u540c\u4e00\u4e2a\u7f51\u7edc\u65e2\u8d1f\u8d23\u9009\u52a8\u4f5c（a* = argmax Q），\u53c8\u8d1f\u8d23\u8bc4\u4f30\u8fd9\u4e2a\u52a8\u4f5c\u7684\u4ef7\u503c\u3002Q \u503c\u672c\u8eab\u662f\u5e26\u566a\u58f0\u7684\u4f30\u8ba1\u6240\u4ee5\u6709\u65f6\u5019\u566a\u58f0\u4f1a\u8ba9\u5dee\u52a8\u4f5c\u7684 Q \u503c\u504f\u9ad8，\u53d6 max \u64cd\u4f5c\u5929\u7136\u504f\u5411\u9009\u90a3\u4e9b\u88ab\u9ad8\u4f30\u7684\u52a8\u4f5c\u3002<\/p>\n

\u6570\u5b66\u4e0a\u6709\u4e2a\u76f4\u89c2\u7684\u89e3\u91ca：<\/p>\n

E[max(X\u2081, X\u2082, …, X\u2099)] \u2265 max(E[X\u2081], E[X\u2082], …, E[X\u2099])<\/p>\n

\u6700\u5927\u503c\u7684\u671f\u671b\u603b\u662f\u5927\u4e8e\u7b49\u4e8e\u671f\u671b\u7684\u6700\u5927\u503c，\u8fd9\u662f\u51f8\u51fd\u6570\u7684 Jensen \u4e0d\u7b49\u5f0f\u3002<\/p>\n

<\/h4>\n

\u8fc7\u4f30\u8ba1\u4f1a\u5bfc\u81f4\u6536\u655b\u53d8\u6162，\u667a\u80fd\u4f53\u628a\u65f6\u95f4\u6d6a\u8d39\u5728\u63a2\u7d22\u90a3\u4e9b\u88ab\u9ad8\u4f30\u7684\u52a8\u4f5c\u4e0a\u3002\u5176\u6b21\u662f\u7b56\u7565\u8d28\u91cf\u6253\u6298\u6263，\u9ad8\u566a\u58f0\u7684\u52a8\u4f5c\u53ef\u80fd\u6bd4\u771f\u6b63\u597d\u7684\u52a8\u4f5c\u66f4\u53d7\u9752\u7750\u3002\u66f4\u7cdf\u7684\u662f\u8fc7\u4f30\u8ba1\u4f1a\u4e0d\u65ad\u7d2f\u79ef，\u5bfc\u81f4\u8bad\u7ec3\u53d1\u6563\u3002\u6cdb\u5316\u80fd\u529b\u4e5f\u4f1a\u53d7\u635f\u2014\u2014\u5728\u72b6\u6001\u7a7a\u95f4\u7684\u566a\u58f0\u533a\u57df，\u667a\u80fd\u4f53\u4f1a\u8868\u73b0\u5f97\u8fc7\u4e8e\u81ea\u4fe1\u3002<\/p>\n

Double DQN：\u628a\u9009\u62e9\u548c\u8bc4\u4f30\u62c6\u5f00<\/h3>\n

\u6807\u51c6 DQN \u4e00\u4e2a\u7f51\u7edc\u5e72\u4e24\u4ef6\u4e8b：<\/p>\n

a* = argmax\u2090' Q(s', a'; \u03b8\u207b) # \u9009\u6700\u4f73\u52a8\u4f5c
\n y = r + \u03b3 \u00b7 Q(s', a*; \u03b8\u207b) # \u8bc4\u4f30\u8fd9\u4e2a\u52a8\u4f5c（\u540c\u4e00\u4e2a\u7f51\u7edc）<\/p>\n

Double DQN \u7528\u4e24\u4e2a\u7f51\u7edc，\u5404\u7ba1\u4e00\u4ef6：<\/p>\n

a* = argmax\u2090' Q(s', a'; \u03b8) # \u7528\u5f53\u524d\u7f51\u7edc\u9009
\n y = r + \u03b3 \u00b7 Q(s', a*; \u03b8\u207b) # \u7528\u76ee\u6807\u7f51\u7edc\u8bc4\u4f30<\/p>\n

\u5f53\u524d\u7f51\u7edc（\u03b8）\u9009\u52a8\u4f5c，\u76ee\u6807\u7f51\u7edc（\u03b8\u207b）\u8bc4\u4f30\u3002\u4e24\u4e2a\u7f51\u7edc\u7684\u8bef\u5dee\u4e0d\u76f8\u5173\u8fd9\u6837\u6700\u5927\u5316\u504f\u5dee\u5c31\u88ab\u6253\u7834\u4e86\u3002<\/p>\n

\u4e3a\u4ec0\u4e48\u6709\u6548\u5462？<\/p>\n

\u5047\u8bbe\u5f53\u524d\u7f51\u7edc\u628a\u52a8\u4f5c a \u7684\u4ef7\u503c\u4f30\u9ad8\u4e86，\u76ee\u6807\u7f51\u7edc（\u53c2\u6570\u4e0d\u540c）\u5927\u6982\u7387\u4e0d\u4f1a\u72af\u540c\u6837\u7684\u9519\u3002\u8bef\u5dee\u76f8\u4e92\u72ec\u7acb，\u503e\u5411\u4e8e\u62b5\u6d88\u800c\u975e\u7d2f\u52a0\u3002<\/p>\n

\u6700\u901a\u4fd7\u7684\u89e3\u91ca\u5c31\u662fDQN \u50cf\u662f\u81ea\u5df1\u7ed9\u83dc\u6253\u5206\u3001\u81ea\u5df1\u6311\u83dc\u5403，\u8fd9\u6837\u70c2\u83dc\u53ef\u80fd\u5c31\u6df7\u8fdb\u6765\u4e86，\u800cDouble DQN \u8ba9\u670b\u53cb\u6253\u5206\u3001\u4f60\u6765\u6311，\u4e24\u8fb9\u7684\u8bef\u5dee\u5bf9\u51b2\u6389\u4e86\u3002<\/p>\n

Standard DQN: E[Q(s, argmax\u2090 Q(s,a))] \u2265 max\u2090 E[Q(s,a)] （\u6709\u504f）
\n Double DQN: E[Q\u2082(s, argmax\u2090 Q\u2081(s,a))] \u2248 max\u2090 E[Q(s,a)] （\u65e0\u504f）<\/p>\n

\u4ece DQN \u5230 Double DQN，\u53ea\u9700\u8981\u6539\u4e00\u884c：<\/p>\n

# DQN \u76ee\u6807
\nnext_q_values=target_network(next_states).max(1)[0]
\ntarget=rewards+gamma*next_q_values* (1-dones) <\/p>\n

# Double DQN \u76ee\u6807
\nnext_actions=current_network(next_states).argmax(1) # <- \u7528\u5f53\u524d\u7f51\u7edc\u9009
\nnext_q_values=target_network(next_states).gather(1, next_actions.unsqueeze(1)) # <- \u7528\u76ee\u6807\u7f51\u7edc\u8bc4\u4f30
\n target=rewards+gamma*next_q_values.squeeze() * (1-dones)<\/p>\n

\u5c31\u8fd9\u4e00\u884c\u6539\u52a8\u6781\u5c0f，\u6548\u679c\u5374\u5f88\u660e\u663e\u3002<\/p>\n

\u5b9e\u73b0：Double DQN<\/h3>\n

\u6269\u5c55 DQN Agent<\/p>\n

classDoubleDQNAgent(DQNAgent):
\n """
\n Double DQN: \u901a\u8fc7\u89e3\u8026\u52a8\u4f5c\u9009\u62e9\u548c\u8bc4\u4f30\u6765\u51cf\u5c11\u8fc7\u4f30\u8ba1\u504f\u5dee\u3002
\n """ <\/p>\n

def__init__(self, *args, **kwargs):
\n """
\n \u521d\u59cb\u5316 Double DQN agent\u3002
\n \u4ece DQN \u7ee7\u627f\u6240\u6709\u5185\u5bb9，\u53ea\u6539\u53d8\u76ee\u6807\u8ba1\u7b97\u3002
\n """
\n super().__init__(*args, **kwargs) <\/p>\n

defupdate(self) ->Dict[str, float]:
\n """
\n \u6267\u884c Double DQN \u66f4\u65b0\u3002 <\/p>\n

Returns:
\n metrics: \u8bad\u7ec3\u6307\u6807
\n """
\n iflen(self.replay_buffer) <self.batch_size:
\n return {} <\/p>\n

# \u91c7\u6837\u6279\u6b21
\n states, actions, rewards, next_states, dones=self.replay_buffer.sample(
\n self.batch_size
\n ) <\/p>\n

states=states.to(self.device)
\n actions=actions.to(self.device)
\n rewards=rewards.to(self.device)
\n next_states=next_states.to(self.device)
\n dones=dones.to(self.device) <\/p>\n

# \u5f53\u524d Q \u503c Q(s,a;\u03b8)
\n current_q_values=self.q_network(states).gather(1, actions.unsqueeze(1)) <\/p>\n

# Double DQN \u76ee\u6807\u8ba1\u7b97
\n withtorch.no_grad():
\n # \u4f7f\u7528\u5f53\u524d\u7f51\u7edc\u9009\u62e9\u52a8\u4f5c
\n next_actions=self.q_network(next_states).argmax(1) <\/p>\n

# \u4f7f\u7528\u76ee\u6807\u7f51\u7edc\u8bc4\u4f30\u52a8\u4f5c
\n next_q_values=self.target_network(next_states).gather(
\n 1, next_actions.unsqueeze(1)
\n ).squeeze() <\/p>\n

# \u8ba1\u7b97\u76ee\u6807
\n target_q_values=rewards+ (1-dones) *self.gamma*next_q_values <\/p>\n

# \u8ba1\u7b97\u635f\u5931
\n loss=F.mse_loss(current_q_values.squeeze(), target_q_values) <\/p>\n

# \u68af\u5ea6\u4e0b\u964d
\n self.optimizer.zero_grad()
\n loss.backward()
\n torch.nn.utils.clip_grad_norm_(self.q_network.parameters(), max_norm=10.0)
\n self.optimizer.step() <\/p>\n

self.training_step+=1 <\/p>\n

return {
\n 'loss': loss.item(),
\n 'q_mean': current_q_values.mean().item(),
\n 'q_std': current_q_values.std().item(),
\n 'target_q_mean': target_q_values.mean().item()
\n }<\/p>\n

\u8bad\u7ec3\u51fd\u6570：<\/p>\n

deftrain_double_dqn(
\n env_name: str,
\n n_episodes: int=1000,
\n max_steps: int=500,
\n train_freq: int=1,
\n eval_frequency: int=50,
\n eval_episodes: int=10,
\n verbose: bool=True,
\n **kwargs
\n) ->Tuple:
\n """
\n \u8bad\u7ec3 Double DQN agent（\u4f7f\u7528 DoubleDQNAgent \u800c\u4e0d\u662f DQNAgent）\u3002
\n """
\n # \u4e0e train_dqn \u76f8\u540c\u4f46\u4f7f\u7528 DoubleDQNAgent
\n env=gym.make(env_name)
\n eval_env=gym.make(env_name) <\/p>\n

state_dim=env.observation_space.shape[0]
\n action_dim=env.action_space.n <\/p>\n

# \u4f7f\u7528 DoubleDQNAgent
\n agent=DoubleDQNAgent(
\n state_dim=state_dim,
\n action_dim=action_dim,
\n **kwargs
\n ) <\/p>\n

# \u8bad\u7ec3\u5faa\u73af（\u4e0e DQN \u76f8\u540c）
\n stats= {
\n 'episode_rewards': [],
\n 'episode_lengths': [],
\n 'losses': [],
\n 'q_values': [],
\n 'target_q_values': [],
\n 'eval_rewards': [],
\n 'eval_episodes': [],
\n 'epsilons': []
\n } <\/p>\n

print(f"Training Double DQN on {env_name}")
\n print(f"State dim: {state_dim}, Action dim: {action_dim}")
\n print("="*70) <\/p>\n

forepisodeinrange(n_episodes):
\n state, _=env.reset()
\n episode_reward=0
\n episode_length=0
\n episode_metrics= [] <\/p>\n

forstepinrange(max_steps):
\n action=agent.select_action(state, training=True)
\n next_state, reward, terminated, truncated, _=env.step(action)
\n done=terminatedortruncated <\/p>\n

agent.store_transition(state, action, reward, next_state, done) <\/p>\n

ifstep%train_freq==0:
\n metrics=agent.update()
\n ifmetrics:
\n episode_metrics.append(metrics) <\/p>\n

episode_reward+=reward
\n episode_length+=1
\n state=next_state <\/p>\n

ifdone:
\n break <\/p>\n

# \u66f4\u65b0\u76ee\u6807\u7f51\u7edc
\n if (episode+1) %kwargs.get('target_update_freq', 10) ==0:
\n agent.update_target_network() <\/p>\n

agent.decay_epsilon() <\/p>\n

# \u5b58\u50a8\u7edf\u8ba1\u4fe1\u606f
\n stats['episode_rewards'].append(episode_reward)
\n stats['episode_lengths'].append(episode_length)
\n stats['epsilons'].append(agent.epsilon) <\/p>\n

ifepisode_metrics:
\n stats['losses'].append(np.mean([m['loss'] forminepisode_metrics]))
\n stats['q_values'].append(np.mean([m['q_mean'] forminepisode_metrics]))
\n stats['target_q_values'].append(np.mean([m['target_q_mean'] forminepisode_metrics])) <\/p>\n

# \u8bc4\u4f30
\n if (episode+1) %eval_frequency==0:
\n eval_reward=evaluate_dqn(eval_env, agent, eval_episodes)
\n stats['eval_rewards'].append(eval_reward)
\n stats['eval_episodes'].append(episode+1) <\/p>\n

ifverbose:
\n avg_reward=np.mean(stats['episode_rewards'][-50:])
\n avg_loss=np.mean(stats['losses'][-50:]) ifstats['losses'] else0
\n avg_q=np.mean(stats['q_values'][-50:]) ifstats['q_values'] else0 <\/p>\n

print(f"Episode {episode+1:4d} | "
\n f"Reward: {avg_reward:7.2f} | "
\n f"Eval: {eval_reward:7.2f} | "
\n f"Loss: {avg_loss:7.4f} | "
\n f"Q: {avg_q:6.2f} | "
\n f"\u03b5: {agent.epsilon:.3f}") <\/p>\n

env.close()
\n eval_env.close() <\/p>\n

print("="*70)
\n print("Training complete!") <\/p>\n

returnagent, stats<\/p>\n

LunarLander-v3<\/p>\n

# \u8bad\u7ec3 Double DQN
\nif__name__=="__main__":
\n device='cuda'iftorch.cuda.is_available() else'cpu' <\/p>\n

agent_ddqn, stats_ddqn=train_double_dqn(
\n env_name='LunarLander-v3',
\n n_episodes=4000,
\n max_steps=1000,
\n learning_rate=5e-4,
\n gamma=0.99,
\n epsilon_start=1.0,
\n epsilon_end=0.01,
\n epsilon_decay=0.9995,
\n buffer_capacity=100000,
\n batch_size=128,
\n target_update_freq=20,
\n train_freq=4,
\n eval_frequency=100,
\n eval_episodes=10,
\n hidden_dims=[256, 256],
\n device=device,
\n verbose=True
\n ) <\/p>\n

# \u4fdd\u5b58\u6a21\u578b
\n agent_ddqn.save('doubledqn_lunar_lander.pth')<\/p>\n

\u8f93\u51fa：<\/p>\n

Training Double DQN on LunarLander-v3
\nState dim: 8, Action dim: 4
\n======================================================================
\nEpisode 100 | Reward: -155.24 | Eval: -885.72 | Loss: 52.9057 | Q: 0.20 | \u03b5: 0.951
\nEpisode 200 | Reward: -148.85 | Eval: -85.94 | Loss: 37.2449 | Q: 2.14 | \u03b5: 0.905
\nEpisode 300 | Reward: -111.61 | Eval: -172.48 | Loss: 37.4279 | Q: 3.52 | \u03b5: 0.861
\nEpisode 400 | Reward: -99.21 | Eval: -198.43 | Loss: 41.5296 | Q: 8.15 | \u03b5: 0.819
\nEpisode 500 | Reward: -80.75 | Eval: -103.26 | Loss: 56.2701 | Q: 11.70 | \u03b5: 0.779
\n…
\nEpisode 3200 | Reward: 102.04 | Eval: 159.71 | Loss: 16.5263 | Q: 27.94 | \u03b5: 0.202
\nEpisode 3300 | Reward: 140.37 | Eval: 191.79 | Loss: 22.5564 | Q: 29.81 | \u03b5: 0.192
\nEpisode 3400 | Reward: 114.08 | Eval: 269.40 | Loss: 23.2846 | Q: 32.40 | \u03b5: 0.183
\nEpisode 3500 | Reward: 166.33 | Eval: 244.32 | Loss: 21.8558 | Q: 32.51 | \u03b5: 0.174
\nEpisode 3600 | Reward: 150.80 | Eval: 265.42 | Loss: 21.6430 | Q: 33.18 | \u03b5: 0.165
\nEpisode 3700 | Reward: 148.59 | Eval: 239.56 | Loss: 23.8328 | Q: 34.65 | \u03b5: 0.157
\nEpisode 3800 | Reward: 162.82 | Eval: 233.36 | Loss: 28.3445 | Q: 37.46 | \u03b5: 0.149
\nEpisode 3900 | Reward: 177.70 | Eval: 259.99 | Loss: 36.2971 | Q: 40.22 | \u03b5: 0.142
\nEpisode 4000 | Reward: 156.60 | Eval: 251.17 | Loss: 46.7266 | Q: 42.15 | \u03b5: 0.135
\n======================================================================
\n Training complete!<\/p>\n

Dueling DQN：\u5206\u79bb\u503c\u548c\u4f18\u52bf<\/h3>\n

\u5f88\u591a\u72b6\u6001\u4e0b，\u9009\u54ea\u4e2a\u52a8\u4f5c\u5176\u5b9e\u5dee\u522b\u4e0d\u5927\u3002CartPole \u91cc\u6746\u5b50\u521a\u597d\u5e73\u8861\u65f6，\u5411\u5de6\u5411\u53f3\u90fd\u884c；\u5f00\u8f66\u8d70\u76f4\u7ebf\u65b9\u5411\u76d8\u5fae\u8c03\u7684\u7ed3\u679c\u5dee\u4e0d\u591a；LunarLander \u79bb\u5730\u9762\u8fd8\u8fdc\u7684\u65f6\u5019，\u5f15\u64ce\u600e\u4e48\u55b7\u5f71\u54cd\u4e5f\u6709\u9650\u3002<\/p>\n

\u6807\u51c6 DQN \u5bf9\u6bcf\u4e2a\u52a8\u4f5c\u5355\u72ec\u5b66 Q(s,a)，\u628a\u7f51\u7edc\u5bb9\u91cf\u6d6a\u8d39\u5728\u5197\u4f59\u4fe1\u606f\u4e0a\u3002Dueling DQN \u7684\u601d\u8def\u662f\u628a Q \u62c6\u6210\u4e24\u90e8\u5206：V(s) \u8868\u793a"\u8fd9\u4e2a\u72b6\u6001\u672c\u8eab\u503c\u591a\u5c11"，A(s,a) \u8868\u793a"\u8fd9\u4e2a\u52a8\u4f5c\u6bd4\u5e73\u5747\u6c34\u5e73\u597d\u591a\u5c11"\u3002<\/p>\n

\u67b6\u6784\u5982\u4e0b<\/p>\n

\u6807\u51c6 DQN:
\n Input -> Hidden Layers -> Q(s,a\u2081), Q(s,a\u2082), …, Q(s,a\u2099) <\/p>\n

Dueling DQN:
\n |-> Value Stream -> V(s)
\nInput -> Shared Layers |
\n |-> Advantage Stream -> A(s,a\u2081), A(s,a\u2082), …, A(s,a\u2099) <\/p>\n

Q(s,a) = V(s) + (A(s,a) – mean(A(s,\u00b7)))<\/p>\n

\u4e3a\u4ec0\u4e48\u8981\u51cf\u53bb\u5747\u503c？\u4e0d\u51cf\u7684\u8bdd，\u4efb\u4f55\u5e38\u6570\u52a0\u5230 V \u518d\u4ece A \u51cf\u6389，\u5f97\u5230\u7684 Q \u5b8c\u5168\u4e00\u6837，\u7f51\u7edc\u5b66\u4e0d\u51fa\u552f\u4e00\u89e3\u3002<\/p>\n

\u6570\u5b66\u8868\u8fbe\u5982\u4e0b：<\/p>\n

Q(s,a) = V(s) + A(s,a) – (1\/|A|)\u00b7\u03a3\u2090' A(s,a')<\/p>\n

\u4e5f\u53ef\u4ee5\u7528 max \u4ee3\u66ff mean：<\/p>\n

Q(s,a) = V(s) + A(s,a) – max\u2090' A(s,a')<\/p>\n

\u5b9e\u8df5\u4e2d max \u7248\u672c\u6709\u65f6\u6548\u679c\u66f4\u597d\u3002<\/p>\n

\u4e3e\u4e2a\u4f8b\u5b50：V(s) = 10，\u597d\u52a8\u4f5c\u7684 A \u662f +5，\u5dee\u52a8\u4f5c\u7684 A \u662f -3，\u5e73\u5747\u4f18\u52bf = (+5-3)\/2 = +1\u3002\u90a3\u4e48 Q(s, \u597d\u52a8\u4f5c) = 10 + 5 – 1 = 14，Q(s, \u5dee\u52a8\u4f5c) = 10 – 3 – 1 = 6\u3002<\/p>\n

\u5b9e\u73b0<\/p>\n

classDuelingQNetwork(nn.Module):
\n """
\n Dueling DQN \u67b6\u6784，\u5206\u79bb\u503c\u548c\u4f18\u52bf\u3002 <\/p>\n

\u7406\u8bba: Q(s,a) = V(s) + A(s,a) – mean(A(s,\u00b7))
\n """ <\/p>\n

def__init__(
\n self,
\n state_dim: int,
\n action_dim: int,
\n hidden_dims: List[int] = [128, 128]
\n ):
\n """
\n \u521d\u59cb\u5316 Dueling Q \u7f51\u7edc\u3002 <\/p>\n

Args:
\n state_dim: \u72b6\u6001\u7a7a\u95f4\u7ef4\u5ea6
\n action_dim: \u52a8\u4f5c\u6570\u91cf
\n hidden_dims: \u5171\u4eab\u5c42\u5927\u5c0f
\n """
\n super(DuelingQNetwork, self).__init__() <\/p>\n

self.state_dim=state_dim
\n self.action_dim=action_dim <\/p>\n

# \u5171\u4eab\u7279\u5f81\u63d0\u53d6\u5668
\n shared_layers= []
\n input_dim=state_dim <\/p>\n

forhidden_diminhidden_dims:
\n shared_layers.append(nn.Linear(input_dim, hidden_dim))
\n shared_layers.append(nn.ReLU())
\n input_dim=hidden_dim <\/p>\n

self.shared_network=nn.Sequential(*shared_layers) <\/p>\n

# \u503c\u6d41: V(s) = \u72b6\u6001\u7684\u6807\u91cf\u503c
\n self.value_stream=nn.Sequential(
\n nn.Linear(hidden_dims[-1], 128),
\n nn.ReLU(),
\n nn.Linear(128, 1)
\n ) <\/p>\n

# \u4f18\u52bf\u6d41: A(s,a) = \u6bcf\u4e2a\u52a8\u4f5c\u7684\u4f18\u52bf
\n self.advantage_stream=nn.Sequential(
\n nn.Linear(hidden_dims[-1], 128),
\n nn.ReLU(),
\n nn.Linear(128, action_dim)
\n ) <\/p>\n

# \u521d\u59cb\u5316\u6743\u91cd
\n self.apply(self._init_weights) <\/p>\n

def_init_weights(self, module):
\n """\u521d\u59cb\u5316\u7f51\u7edc\u6743\u91cd\u3002"""
\n ifisinstance(module, nn.Linear):
\n nn.init.kaiming_normal_(module.weight, nonlinearity='relu')
\n nn.init.constant_(module.bias, 0.0) <\/p>\n

defforward(self, state: torch.Tensor) ->torch.Tensor:
\n """
\n \u901a\u8fc7 dueling \u67b6\u6784\u7684\u524d\u5411\u4f20\u64ad\u3002 <\/p>\n

Args:
\n state: \u72b6\u6001\u6279\u6b21, \u5f62\u72b6 (batch_size, state_dim) <\/p>\n

Returns:
\n q_values: \u6240\u6709\u52a8\u4f5c\u7684 Q(s,a), \u5f62\u72b6 (batch_size, action_dim)
\n """
\n # \u5171\u4eab\u7279\u5f81
\n features=self.shared_network(state) <\/p>\n

# \u503c: V(s) -> \u5f62\u72b6 (batch_size, 1)
\n value=self.value_stream(features) <\/p>\n

# \u4f18\u52bf: A(s,a) -> \u5f62\u72b6 (batch_size, action_dim)
\n advantages=self.advantage_stream(features) <\/p>\n

# \u7ec4\u5408: Q(s,a) = V(s) + A(s,a) – mean(A(s,\u00b7))
\n q_values=value+advantages-advantages.mean(dim=1, keepdim=True) <\/p>\n

returnq_values <\/p>\n

defget_action(self, state: np.ndarray, epsilon: float=0.0) ->int:
\n """
\n \u4f7f\u7528 \u03b5-greedy \u7b56\u7565\u9009\u62e9\u52a8\u4f5c\u3002
\n """
\n ifrandom.random() <epsilon:
\n returnrandom.randint(0, self.action_dim-1)
\n else:
\n withtorch.no_grad():
\n state_tensor=torch.FloatTensor(state).unsqueeze(0).to(
\n next(self.parameters()).device
\n )
\n q_values=self.forward(state_tensor)
\n returnq_values.argmax(dim=1).item()<\/p>\n

Dueling \u67b6\u6784\u7684\u597d\u5904：\u5728\u52a8\u4f5c\u5f71\u54cd\u4e0d\u5927\u7684\u72b6\u6001\u4e0b\u5b66\u5f97\u66f4\u597d，\u68af\u5ea6\u6d41\u52a8\u66f4\u901a\u7545\u6240\u4ee5\u6536\u655b\u66f4\u5feb，\u503c\u4f30\u8ba1\u4e5f\u66f4\u7a33\u5065\u3002<\/p>\n

\u8fd8\u53ef\u4ee5\u628a\u4e24\u79cd\u6539\u8fdb\u53e0\u5728\u4e00\u8d77，\u505a\u6210Double Dueling DQN<\/p>\n

classDoubleDuelingDQNAgent(DoubleDQNAgent):
\n """
\n \u7ed3\u5408 Double DQN \u548c Dueling DQN \u7684\u667a\u80fd\u4f53\u3002
\n """ <\/p>\n

def__init__(
\n self,
\n state_dim: int,
\n action_dim: int,
\n hidden_dims: List[int] = [128, 128],
\n **kwargs
\n ):
\n """
\n \u521d\u59cb\u5316 Double Dueling DQN \u667a\u80fd\u4f53\u3002
\n \u4f7f\u7528 DuelingQNetwork \u800c\u4e0d\u662f\u6807\u51c6 QNetwork\u3002
\n """
\n # \u6682\u4e0d\u8c03\u7528 super().__init__()
\n # \u6211\u4eec\u9700\u8981\u4ee5\u4e0d\u540c\u65b9\u5f0f\u8bbe\u7f6e\u7f51\u7edc <\/p>\n

self.state_dim=state_dim
\n self.action_dim=action_dim
\n self.gamma=kwargs.get('gamma', 0.99)
\n self.batch_size=kwargs.get('batch_size', 64)
\n self.target_update_freq=kwargs.get('target_update_freq', 10)
\n self.device=torch.device(kwargs.get('device', 'cpu')) <\/p>\n

# \u63a2\u7d22
\n self.epsilon=kwargs.get('epsilon_start', 1.0)
\n self.epsilon_end=kwargs.get('epsilon_end', 0.01)
\n self.epsilon_decay=kwargs.get('epsilon_decay', 0.995) <\/p>\n

# \u4f7f\u7528 Dueling \u67b6\u6784
\n self.q_network=DuelingQNetwork(
\n state_dim, action_dim, hidden_dims
\n ).to(self.device) <\/p>\n

self.target_network=DuelingQNetwork(
\n state_dim, action_dim, hidden_dims
\n ).to(self.device) <\/p>\n

self.target_network.load_state_dict(self.q_network.state_dict())
\n self.target_network.eval() <\/p>\n

# \u4f18\u5316\u5668
\n learning_rate=kwargs.get('learning_rate', 1e-3)
\n self.optimizer=torch.optim.Adam(self.q_network.parameters(), lr=learning_rate) <\/p>\n

# \u56de\u653e\u7f13\u51b2\u533a
\n buffer_capacity=kwargs.get('buffer_capacity', 100000)
\n self.replay_buffer=ReplayBuffer(buffer_capacity) <\/p>\n

# \u7edf\u8ba1
\n self.episode_count=0
\n self.training_step=0 <\/p>\n

# update() \u65b9\u6cd5\u7ee7\u627f\u81ea DoubleDQNAgent<\/p>\n

\u4f18\u5148\u7ecf\u9a8c\u56de\u653e<\/h3>\n

\u4e0d\u662f\u6240\u6709\u7ecf\u9a8c\u90fd\u540c\u7b49\u6709\u4ef7\u503c\u3002TD \u8bef\u5dee\u5927\u7684\u8f6c\u6362\u8bf4\u660e\u9884\u6d4b\u504f\u79bb\u73b0\u5b9e，\u80fd\u5b66\u5230\u4e1c\u897f；TD \u8bef\u5dee\u5c0f\u7684\u8f6c\u6362\u8bf4\u660e\u5df2\u7ecf\u5b66\u5f97\u5dee\u4e0d\u591a\u4e86\u518d\u91c7\u5230\u4e5f\u6ca1\u591a\u5927\u7528\u3002<\/p>\n

\u5747\u5300\u91c7\u6837\u628a\u6240\u6709\u8f6c\u6362\u4e00\u89c6\u540c\u4ec1，\u6d6a\u8d39\u4e86\u5b66\u4e60\u673a\u4f1a\u3002\u4f18\u5148\u7ecf\u9a8c\u56de\u653e\u7684\u601d\u8def\u662f：\u8ba9\u91cd\u8981\u7684\u8f6c\u6362\u88ab\u91c7\u5230\u7684\u6982\u7387\u66f4\u9ad8\u3002<\/p>\n

\u4f18\u5148\u7ea7\u600e\u4e48\u7b97<\/p>\n

p\u1d62 = |\u03b4\u1d62| + \u03b5 <\/p>\n

\u5176\u4e2d:
\n \u03b4\u1d62 = r + \u03b3\u00b7max Q(s',a') – Q(s,a) （TD \u8bef\u5dee）
\n \u03b5 = \u5c0f\u5e38\u6570，\u4fdd\u8bc1\u6240\u6709\u8f6c\u6362\u90fd\u6709\u88ab\u91c7\u5230\u7684\u53ef\u80fd<\/p>\n

\u91c7\u6837\u6982\u7387：<\/p>\n

P(i) = p\u1d62^\u03b1 \/ \u03a3\u2c7c p\u2c7c^\u03b1 <\/p>\n

\u03b1 \u63a7\u5236\u4f18\u5148\u5316\u7a0b\u5ea6:
\n \u03b1 = 0 -> \u9000\u5316\u6210\u5747\u5300\u91c7\u6837
\n \u03b1 = 1 -> \u5b8c\u5168\u6309\u4f18\u5148\u7ea7\u6bd4\u4f8b\u91c7\u6837<\/p>\n

\u4f18\u5148\u91c7\u6837\u6539\u4e86\u6570\u636e\u5206\u5e03，\u4f1a\u5f15\u5165\u504f\u5dee\u3002\u6240\u4ee5\u89e3\u51b3\u529e\u6cd5\u662f\u7528\u91cd\u8981\u6027\u91c7\u6837\u6bd4\u7387\u6765\u52a0\u6743\u66f4\u65b0：<\/p>\n

w\u1d62 = (N \u00b7 P(i))^(-\u03b2) <\/p>\n

\u03b2 \u63a7\u5236\u6821\u6b63\u529b\u5ea6:
\n \u03b2 = 0 -> \u4e0d\u6821\u6b63
\n \u03b2 = 1 -> \u5b8c\u5168\u6821\u6b63<\/p>\n

\u901a\u5e38 \u03b2 \u4ece 0.4 \u5f00\u59cb，\u968f\u8bad\u7ec3\u9010\u6e10\u589e\u5927\u5230 1.0\u3002<\/p>\n

\u5b9e\u73b0<\/p>\n

classPrioritizedReplayBuffer:
\n """
\n \u4f18\u5148\u7ecf\u9a8c\u56de\u653e\u7f13\u51b2\u533a\u3002 <\/p>\n

\u7406\u8bba: \u6309 TD \u8bef\u5dee\u6bd4\u4f8b\u91c7\u6837\u8f6c\u6362\u3002
\n \u6211\u4eec\u53ef\u4ee5\u4ece\u4e2d\u5b66\u5230\u66f4\u591a\u7684\u8f6c\u6362\u4f1a\u88ab\u66f4\u9891\u7e41\u5730\u91c7\u6837\u3002
\n """ <\/p>\n

def__init__(self, capacity: int, alpha: float=0.6, beta: float=0.4):
\n """
\n Args:
\n capacity: \u7f13\u51b2\u533a\u6700\u5927\u5bb9\u91cf
\n alpha: \u4f18\u5148\u5316\u6307\u6570（0=\u5747\u5300, 1=\u6bd4\u4f8b）
\n beta: \u91cd\u8981\u6027\u91c7\u6837\u6307\u6570（\u9000\u706b\u5230 1.0）
\n """
\n self.capacity=capacity
\n self.alpha=alpha
\n self.beta=beta
\n self.beta_increment=0.001 # \u968f\u65f6\u95f4\u9000\u706b beta <\/p>\n

self.buffer= []
\n self.priorities=np.zeros(capacity, dtype=np.float32)
\n self.position=0 <\/p>\n

defpush(self, state, action, reward, next_state, done):
\n """
\n \u4ee5\u6700\u5927\u4f18\u5148\u7ea7\u6dfb\u52a0\u8f6c\u6362\u3002 <\/p>\n

\u7406\u8bba: \u65b0\u8f6c\u6362\u83b7\u5f97\u6700\u5927\u4f18\u5148\u7ea7（\u4f1a\u5f88\u5feb\u88ab\u91c7\u6837）\u3002
\n \u5b83\u4eec\u7684\u5b9e\u9645\u4f18\u5148\u7ea7\u5728\u9996\u6b21 TD \u8bef\u5dee\u8ba1\u7b97\u540e\u66f4\u65b0\u3002
\n """
\n max_priority=self.priorities.max() ifself.bufferelse1.0 <\/p>\n

iflen(self.buffer) <self.capacity:
\n self.buffer.append((state, action, reward, next_state, done))
\n else:
\n self.buffer[self.position] = (state, action, reward, next_state, done) <\/p>\n

self.priorities[self.position] =max_priority
\n self.position= (self.position+1) %self.capacity <\/p>\n

defsample(self, batch_size: int):
\n """
\n \u6309\u4f18\u5148\u7ea7\u6bd4\u4f8b\u91c7\u6837\u6279\u6b21\u3002 <\/p>\n

Returns:
\n batch: \u91c7\u6837\u7684\u8f6c\u6362
\n indices: \u91c7\u6837\u8f6c\u6362\u7684\u7d22\u5f15（\u7528\u4e8e\u4f18\u5148\u7ea7\u66f4\u65b0）
\n weights: \u91cd\u8981\u6027\u91c7\u6837\u6743\u91cd
\n """
\n iflen(self.buffer) ==self.capacity:
\n priorities=self.priorities
\n else:
\n priorities=self.priorities[:len(self.buffer)] <\/p>\n

# \u8ba1\u7b97\u91c7\u6837\u6982\u7387
\n probs=priorities**self.alpha
\n probs\/=probs.sum() <\/p>\n

# \u91c7\u6837\u7d22\u5f15
\n indices=np.random.choice(len(self.buffer), batch_size, p=probs, replace=False) <\/p>\n

# \u83b7\u53d6\u8f6c\u6362
\n batch= [self.buffer[idx] foridxinindices] <\/p>\n

# \u8ba1\u7b97\u91cd\u8981\u6027\u91c7\u6837\u6743\u91cd
\n total=len(self.buffer)
\n weights= (total*probs[indices]) ** (-self.beta)
\n weights\/=weights.max() # \u5f52\u4e00\u5316\u4ee5\u4fdd\u6301\u7a33\u5b9a\u6027 <\/p>\n

# \u9000\u706b beta
\n self.beta=min(1.0, self.beta+self.beta_increment) <\/p>\n

# \u8f6c\u6362\u4e3a tensor
\n states, actions, rewards, next_states, dones=zip(*batch) <\/p>\n

states=torch.FloatTensor(np.array(states))
\n actions=torch.LongTensor(actions)
\n rewards=torch.FloatTensor(rewards)
\n next_states=torch.FloatTensor(np.array(next_states))
\n dones=torch.FloatTensor(dones)
\n weights=torch.FloatTensor(weights) <\/p>\n

return (states, actions, rewards, next_states, dones), indices, weights <\/p>\n

defupdate_priorities(self, indices, td_errors):
\n """
\n \u6839\u636e TD \u8bef\u5dee\u66f4\u65b0\u4f18\u5148\u7ea7\u3002 <\/p>\n

Args:
\n indices: \u91c7\u6837\u8f6c\u6362\u7684\u7d22\u5f15
\n td_errors: \u90a3\u4e9b\u8f6c\u6362\u7684 TD \u8bef\u5dee
\n """
\n foridx, td_errorinzip(indices, td_errors):
\n self.priorities[idx] =abs(td_error) +1e-6 <\/p>\n

def__len__(self):
\n returnlen(self.buffer)<\/p>\n

\u751f\u4ea7\u73af\u5883\u4f1a\u7528 sum-tree \u6570\u636e\u7ed3\u6784，\u91c7\u6837\u590d\u6742\u5ea6\u662f O(log N) \u800c\u4e0d\u662f\u8fd9\u91cc\u7684 O(N)\u3002\u8fd9\u4e2a\u7b80\u5316\u7248\u672c\u4ee5\u53ef\u8bfb\u6027\u4e3a\u4f18\u5148\u3002<\/p>\n

DQN \u53d8\u4f53\u5bf9\u6bd4<\/h3>\n

\u51e0\u4e2a\u53d8\u4f53\u5404\u81ea\u89e3\u51b3\u4ec0\u4e48\u95ee\u9898\u5462？<\/p>\n

DQN \u662f\u57fa\u7ebf，\u7528\u5355\u4e00\u7f51\u7edc\u9009\u52a8\u4f5c\u3001\u8bc4\u4f30\u52a8\u4f5c\u3002\u5b83\u5f15\u5165\u4e86\u76ee\u6807\u7f51\u7edc\u6765\u7a33\u5b9a"\u79fb\u52a8\u76ee\u6807"\u95ee\u9898，\u4f46\u5bb9\u6613\u8fc7\u4f30\u8ba1 Q \u503c，\u566a\u58f0\u8ba9\u667a\u80fd\u4f53\u53bb\u8ffd\u9010\u6839\u672c\u4e0d\u5b58\u5728\u7684"\u5e7d\u7075\u5956\u52b1"\u3002<\/p>\n

Double DQN \u628a\u9009\u548c\u8bc4\u62c6\u5f00\u3002\u5728\u7ebf\u7f51\u7edc\u9009\u52a8\u4f5c，\u76ee\u6807\u7f51\u7edc\u8bc4\u4f30\u4ef7\u503c\u3002\u5b9e\u6d4b\u4e0b\u6765\u80fd\u6709\u6548\u538b\u4f4e\u4e0d\u5207\u5b9e\u9645\u7684 Q \u503c，\u5b66\u4e60\u66f2\u7ebf\u660e\u663e\u66f4\u5e73\u6ed1\u3002<\/p>\n

Dueling DQN \u6362\u4e86\u7f51\u7edc\u67b6\u6784，\u5355\u72ec\u5b66 V(s) \u548c A(s,a)\u3002\u5b83\u7684\u6838\u5fc3\u8ba4\u77e5\u662f：\u5f88\u591a\u72b6\u6001\u4e0b\u5177\u4f53\u52a8\u4f5c\u7684\u5f71\u54cd\u4e0d\u5927\u3002\u5728 LunarLander \u8fd9\u79cd\u5b58\u5728\u5927\u91cf"\u5197\u4f59\u52a8\u4f5c"\u7684\u73af\u5883\u91cc，\u6837\u672c\u6548\u7387\u63d0\u5347\u660e\u663e\u2014\u2014\u4e0d\u7528\u4e3a\u6bcf\u6b21\u5f15\u64ce\u8109\u51b2\u90fd\u91cd\u65b0\u5b66\u72b6\u6001\u503c\u3002<\/p>\n

Double Dueling DQN \u628a\u4e24\u8fb9\u7684\u597d\u5904\u7ed3\u5408\u8d77\u6765，\u65e2\u51cf\u5c11\u4f30\u8ba1\u566a\u58f0，\u53c8\u63d0\u9ad8\u8868\u793a\u6548\u7387\u3002\u5b9e\u6d4b\u4e2d\u8fd9\u4e2a\u7ec4\u5408\u6700\u7a33\u5065，\u8fbe\u5230\u5cf0\u503c\u6027\u80fd\u7684\u901f\u5ea6\u548c\u53ef\u9760\u6027\u90fd\u4f18\u4e8e\u5355\u4e00\u6539\u8fdb\u3002<\/p>\n

\u5b9e\u8df5\u5efa\u8bae<\/h3>\n

\u53d8\u4f53\u9009\u62e9\u5bf9\u6bd4 \"\" Double DQN \u8dd1\u5f97\u6bd4 DQN \u8fd8\u5dee？\u53ef\u80fd\u662f\u8bad\u7ec3\u4e0d\u591f\u957f（Double DQN \u8d77\u6b65\u5076\u5c14\u6162\u4e00\u70b9），\u6216\u8005\u76ee\u6807\u7f51\u7edc\u66f4\u65b0\u592a\u9891\u7e41，\u6216\u8005\u5b66\u4e60\u7387\u504f\u9ad8\u3002\u8fd9\u65f6\u53ef\u4ee5\u5c06\u8bad\u7ec3\u65f6\u95f4\u7ffb\u500d，target_update_freq \u8c03\u5927，\u5b66\u4e60\u7387\u780d 2-5 \u500d\u3002<\/p>\n

Dueling \u67b6\u6784\u6ca1\u5e26\u6765\u6539\u5584？\u53ef\u80fd\u662f\u73af\u5883\u672c\u8eab\u4e0d\u9002\u5408（\u6240\u6709\u72b6\u6001\u90fd\u5f88\u5173\u952e），\u6216\u8005\u7f51\u7edc\u592a\u5c0f，\u6216\u8005\u503c\u6d41\/\u4f18\u52bf\u6d41\u592a\u6d45\u3002\u9700\u8981\u5bf9\u7f51\u7edc\u52a0\u5bbd\u52a0\u6df1，\u786e\u8ba4\u73af\u5883\u91cc\u786e\u5b9e\u6709"\u4e2d\u6027"\u72b6\u6001\u3002<\/p>\n

PER \u5bfc\u81f4\u4e0d\u7a33\u5b9a？\u53ef\u80fd\u662f \u03b2 \u9000\u706b\u592a\u5feb\u3001\u03b1 \u8bbe\u592a\u9ad8\u3001\u91cd\u8981\u6027\u91c7\u6837\u6743\u91cd\u6ca1\u5f52\u4e00\u5316\u3002\u53ef\u4ee5\u51cf\u6162 \u03b2 \u589e\u91cf\u3001\u03b1 \u964d\u5230 0.4-0.6\u3001\u786e\u8ba4\u6743\u91cd\u505a\u4e86\u5f52\u4e00\u5316\u3002<\/p>\n

\u9996\u9009 Double DQN \u8d77\u6b65，\u4ee3\u7801\u6539\u52a8\u6781\u5c0f，\u6536\u76ca\u660e\u786e，\u6ca1\u6709\u989d\u5916\u590d\u6742\u5ea6\u3002<\/p>\n

\u4ec0\u4e48\u65f6\u5019\u52a0 Dueling：\u72b6\u6001\u503c\u6bd4\u52a8\u4f5c\u4f18\u52bf\u66f4\u91cd\u8981\u7684\u73af\u5883，\u5927\u91cf\u72b6\u6001\u4e0b\u52a8\u4f5c\u503c\u5dee\u4e0d\u591a，\u9700\u8981\u66f4\u5feb\u6536\u655b\u3002<\/p>\n

\u4ec0\u4e48\u65f6\u5019\u52a0 PER：\u6837\u672c\u6548\u7387\u81f3\u5173\u91cd\u8981，\u6709\u7b97\u529b\u9884\u7b97（PER \u6bd4\u5747\u5300\u91c7\u6837\u6162），\u5956\u52b1\u7a00\u758f（\u5e2e\u52a9\u5173\u6ce8\u5c11\u89c1\u7684\u6210\u529f\u7ecf\u9a8c）\u3002<\/p>\n

\u6700\u540eRainbow \u628a\u516d\u9879\u6539\u8fdb\u53e0\u5728\u4e00\u8d77：Double DQN\u3001Dueling DQN\u3001\u4f18\u5148\u7ecf\u9a8c\u56de\u653e\u3001\u591a\u6b65\u5b66\u4e60（n-step returns）\u3001\u5206\u5e03\u5f0f RL（C51）\u3001\u566a\u58f0\u7f51\u7edc（\u53c2\u6570\u7a7a\u95f4\u63a2\u7d22）\u3002<\/p>\n

\u591a\u6b65\u5b66\u4e60\u628a 1-step TD \u6362\u6210 n-step \u56de\u62a5：<\/p>\n

# 1-step TD:
\n y = r\u209c + \u03b3\u00b7max Q(s\u209c\u208a\u2081, a) <\/p>\n

# n-step:
\n y = r\u209c + \u03b3\u00b7r\u209c\u208a\u2081 + \u03b3\u00b2\u00b7r\u209c\u208a\u2082 + … + \u03b3\u207f\u00b7max Q(s\u209c\u208a\u2099, a)<\/p>\n

\u597d\u5904\u662f\u4fe1\u7528\u5206\u914d\u66f4\u6e05\u6670，\u5b66\u4e60\u66f4\u5feb\u3002<\/p>\n

\u5c0f\u7ed3<\/h3>\n

\u8fd9\u7bc7\u6587\u7ae0\u4ece DQN \u7684\u8fc7\u4f30\u8ba1\u95ee\u9898\u8bb2\u8d77，\u6cbf\u7740 Double DQN\u3001Dueling \u67b6\u6784\u3001\u4f18\u5148\u7ecf\u9a8c\u56de\u653e\u7b49\u7b49\u4ecb\u7ecd\u4e0b\u6765，\u6bcf\u79cd\u6539\u8fdb\u5bf9\u5e94\u4e00\u4e2a\u5177\u4f53\u7684\u5931\u8d25\u6a21\u5f0f：max \u7b97\u5b50\u7684\u504f\u5dee\u3001\u4f4e\u6548\u7684\u72b6\u6001-\u52a8\u4f5c\u8868\u793a\u3001\u6d6a\u8d39\u7684\u5747\u5300\u91c7\u6837\u3002<\/p>\n

\u4ece\u5934\u5b9e\u73b0\u8fd9\u4e9b\u65b9\u6cd5，\u80fd\u641e\u6e05\u695a\u5b83\u4eec\u4e3a\u4ec0\u4e48\u6709\u6548；\u5f88\u591a"\u9ad8\u7ea7" RL \u7b97\u6cd5\u4e0d\u8fc7\u662f\u7b80\u5355\u60f3\u6cd5\u7684\u7ec4\u5408，\u7406\u89e3\u8fd9\u4e9b\u60f3\u6cd5\u672c\u8eab\u624d\u662f\u771f\u6b63\u53ef\u6269\u5c55\u7684\u4e1c\u897f\u3002<\/p>\n

https:\/\/avoid.overfit.cn\/post\/4c5835f419d840b0acb0a1eb72f92b6f<\/p>\n

\u4f5c\u8005： Jugal Gajjar<\/p>\n","protected":false},"excerpt":{"rendered":"

DQN \u7528
\nmax Q(s,a)\u8ba1\u7b97\u76ee\u6807\u503c，\u7b49\u4e8e\u5728\u6311 Q \u503c\u6700\u9ad8\u7684\u52a8\u4f5c，\u4f46\u662f\u8fd9\u4e9b\u52a8\u4f5c\u4e2d\u5305\u62ec\u4e86\u90a3\u4e9b\u56e0\u4e3a\u4f30\u8ba1\u566a\u58f0\u800c\u88ab\u9ad8\u4f30\u7684\u52a8\u4f5c，\u7d20\u4ee5\u5c31\u4f1a\u4ea7\u751f\u8fc7\u4f30\u8ba1\u504f\u5dee，\u76f4\u63a5\u540e\u679c\u662f\u8bad\u7ec3\u4e0d\u7a33\u5b9a\u3001\u7b56\u7565\u6b21\u4f18\u3002
\n\u8fd9\u7bc7\u6587\u7ae0\u8981\u89e3\u51b3\u7684\u5c31\u662f\u8fd9\u4e2a\u95ee\u9898，\u5185\u5bb9\u5305\u62ec：DQN \u4e3a\u4ec0\u4e48\u4f1a\u8fc7\u4f30\u8ba1\u3001Double DQN \u600e\u4e48\u628a\u52a8\u4f5c\u9009\u62e9\u548c\u8bc4\u4f30\u62c6\u5f00\u3001Dueling DQN \u600e\u4e48\u5206\u79bb\u72b6\u6001\u503c\u548c\u52a8\u4f5c\u4f18\u52bf\u3001\u4f18\u5148\u7ecf\u9a8c\u56de\u653e\u5982\u4f55\u8ba9\u91c7\u6837\u66f4\u806a\u660e，\u4ee5\u53ca<\/p>\n","protected":false},"author":2,"featured_media":70378,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[152,50,752,86],"topic":[],"class_list":["post-70380","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-server","tag-pytorch","tag-50","tag-752","tag-86"],"yoast_head":"\n\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.wsisp.com\/helps\/70380.html\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"og:description\" content=\"DQN \u7528 max Q(s,a)\u8ba1\u7b97\u76ee\u6807\u503c，\u7b49\u4e8e\u5728\u6311 Q \u503c\u6700\u9ad8\u7684\u52a8\u4f5c，\u4f46\u662f\u8fd9\u4e9b\u52a8\u4f5c\u4e2d\u5305\u62ec\u4e86\u90a3\u4e9b\u56e0\u4e3a\u4f30\u8ba1\u566a\u58f0\u800c\u88ab\u9ad8\u4f30\u7684\u52a8\u4f5c，\u7d20\u4ee5\u5c31\u4f1a\u4ea7\u751f\u8fc7\u4f30\u8ba1\u504f\u5dee，\u76f4\u63a5\u540e\u679c\u662f\u8bad\u7ec3\u4e0d\u7a33\u5b9a\u3001\u7b56\u7565\u6b21\u4f18\u3002 \u8fd9\u7bc7\u6587\u7ae0\u8981\u89e3\u51b3\u7684\u5c31\u662f\u8fd9\u4e2a\u95ee\u9898，\u5185\u5bb9\u5305\u62ec：DQN \u4e3a\u4ec0\u4e48\u4f1a\u8fc7\u4f30\u8ba1\u3001Double DQN \u600e\u4e48\u628a\u52a8\u4f5c\u9009\u62e9\u548c\u8bc4\u4f30\u62c6\u5f00\u3001Dueling DQN \u600e\u4e48\u5206\u79bb\u72b6\u6001\u503c\u548c\u52a8\u4f5c\u4f18\u52bf\u3001\u4f18\u5148\u7ecf\u9a8c\u56de\u653e\u5982\u4f55\u8ba9\u91c7\u6837\u66f4\u806a\u660e，\u4ee5\u53ca\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.wsisp.com\/helps\/70380.html\" \/>\n<meta property=\"og:site_name\" content=\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\" \/>\n<meta property=\"article:published_time\" content=\"2026-02-01T14:05:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/02\/20260201140558-697f5dc698df7.png\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u4f5c\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data2\" content=\"10 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/70380.html\",\"url\":\"https:\/\/www.wsisp.com\/helps\/70380.html\",\"name\":\"\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"isPartOf\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\"},\"datePublished\":\"2026-02-01T14:05:59+00:00\",\"dateModified\":\"2026-02-01T14:05:59+00:00\",\"author\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.wsisp.com\/helps\/70380.html#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.wsisp.com\/helps\/70380.html\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/70380.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u9996\u9875\",\"item\":\"https:\/\/www.wsisp.com\/helps\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#website\",\"url\":\"https:\/\/www.wsisp.com\/helps\/\",\"name\":\"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3\",\"description\":\"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"contentUrl\":\"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery\",\"caption\":\"admin\"},\"sameAs\":[\"http:\/\/wp.wsisp.com\"],\"url\":\"https:\/\/www.wsisp.com\/helps\/author\/admin\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.wsisp.com\/helps\/70380.html","og_locale":"zh_CN","og_type":"article","og_title":"\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","og_description":"DQN \u7528 max Q(s,a)\u8ba1\u7b97\u76ee\u6807\u503c，\u7b49\u4e8e\u5728\u6311 Q \u503c\u6700\u9ad8\u7684\u52a8\u4f5c，\u4f46\u662f\u8fd9\u4e9b\u52a8\u4f5c\u4e2d\u5305\u62ec\u4e86\u90a3\u4e9b\u56e0\u4e3a\u4f30\u8ba1\u566a\u58f0\u800c\u88ab\u9ad8\u4f30\u7684\u52a8\u4f5c，\u7d20\u4ee5\u5c31\u4f1a\u4ea7\u751f\u8fc7\u4f30\u8ba1\u504f\u5dee，\u76f4\u63a5\u540e\u679c\u662f\u8bad\u7ec3\u4e0d\u7a33\u5b9a\u3001\u7b56\u7565\u6b21\u4f18\u3002 \u8fd9\u7bc7\u6587\u7ae0\u8981\u89e3\u51b3\u7684\u5c31\u662f\u8fd9\u4e2a\u95ee\u9898，\u5185\u5bb9\u5305\u62ec：DQN \u4e3a\u4ec0\u4e48\u4f1a\u8fc7\u4f30\u8ba1\u3001Double DQN \u600e\u4e48\u628a\u52a8\u4f5c\u9009\u62e9\u548c\u8bc4\u4f30\u62c6\u5f00\u3001Dueling DQN \u600e\u4e48\u5206\u79bb\u72b6\u6001\u503c\u548c\u52a8\u4f5c\u4f18\u52bf\u3001\u4f18\u5148\u7ecf\u9a8c\u56de\u653e\u5982\u4f55\u8ba9\u91c7\u6837\u66f4\u806a\u660e，\u4ee5\u53ca","og_url":"https:\/\/www.wsisp.com\/helps\/70380.html","og_site_name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","article_published_time":"2026-02-01T14:05:59+00:00","og_image":[{"url":"https:\/\/www.wsisp.com\/helps\/wp-content\/uploads\/2026\/02\/20260201140558-697f5dc698df7.png"}],"author":"admin","twitter_card":"summary_large_image","twitter_misc":{"\u4f5c\u8005":"admin","\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"10 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.wsisp.com\/helps\/70380.html","url":"https:\/\/www.wsisp.com\/helps\/70380.html","name":"\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848 - \u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","isPartOf":{"@id":"https:\/\/www.wsisp.com\/helps\/#website"},"datePublished":"2026-02-01T14:05:59+00:00","dateModified":"2026-02-01T14:05:59+00:00","author":{"@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41"},"breadcrumb":{"@id":"https:\/\/www.wsisp.com\/helps\/70380.html#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.wsisp.com\/helps\/70380.html"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.wsisp.com\/helps\/70380.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u9996\u9875","item":"https:\/\/www.wsisp.com\/helps"},{"@type":"ListItem","position":2,"name":"\u8ba9 Q \u503c\u4f30\u8ba1\u66f4\u51c6\u786e\uff1a\u4ece DQN \u5230 Double DQN \u7684\u6539\u8fdb\u65b9\u6848"}]},{"@type":"WebSite","@id":"https:\/\/www.wsisp.com\/helps\/#website","url":"https:\/\/www.wsisp.com\/helps\/","name":"\u7f51\u7855\u4e92\u8054\u5e2e\u52a9\u4e2d\u5fc3","description":"\u9999\u6e2f\u670d\u52a1\u5668_\u9999\u6e2f\u4e91\u670d\u52a1\u5668\u8d44\u8baf_\u670d\u52a1\u5668\u5e2e\u52a9\u6587\u6863_\u670d\u52a1\u5668\u6559\u7a0b","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.wsisp.com\/helps\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Person","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/358e386c577a3ab51c4493330a20ad41","name":"admin","image":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.wsisp.com\/helps\/#\/schema\/person\/image\/","url":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","contentUrl":"https:\/\/gravatar.wp-china-yes.net\/avatar\/?s=96&d=mystery","caption":"admin"},"sameAs":["http:\/\/wp.wsisp.com"],"url":"https:\/\/www.wsisp.com\/helps\/author\/admin"}]}},"_links":{"self":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/70380","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/comments?post=70380"}],"version-history":[{"count":0,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/posts\/70380\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media\/70378"}],"wp:attachment":[{"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/media?parent=70380"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/categories?post=70380"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/tags?post=70380"},{"taxonomy":"topic","embeddable":true,"href":"https:\/\/www.wsisp.com\/helps\/wp-json\/wp\/v2\/topic?post=70380"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}