面向交易的强化学习

在尝试中学习

在监督学习里，每个样本都带有正确答案。交易不是这样：没人告诉你最佳仓位，你行动，市场用盈利或亏损来回答。强化学习把它建模为一个智能体：观察状态，采取动作，获得奖励，进入新状态，目标是最大化随时间累积的总奖励。对交易而言，状态可以是近期收益或当前仓位，动作是买入、持有或卖出，奖励是这一步的盈利减去成本。难点在于动作的后果要过一段时间才显现，智能体必须通过尝试去发现什么有回报。

# Reward of one step: profit from the position held, minus the cost of changing position
def step_reward(position, price_change, previous_position, cost):
    return position * price_change - cost * abs(position - previous_position)

print(round(step_reward(1, 0.5, 0, 0.1), 4))         # 0.4    went long, the price rose 0.5, paid 0.1 to trade
print(round(step_reward(-1, 0.5, 1, 0.1), 4))        # -0.7   flipped from long to short just before a rise: lost 0.5 and paid 0.2
print(step_reward(1, 0.5, 1, 0.1))                   # 0.5    holding the same position costs nothing

Q 学习：某状态下动作的价值

Q 学习维护一张表 Q[状态, 动作]：在某状态采取某动作、此后表现良好时预期的总奖励。每一步之后，它把对应条目推向更好的估计，即目标：刚得到的奖励加上下一状态最佳动作的折现价值，reward + gamma * max(Q[下一状态])。更新为 Q[s, a] += alpha * (目标 - Q[s, a])，其中 alpha 是学习率。折现因子 gamma（0 到 1）说明未来奖励有多重要：gamma = 0 是目光短浅的智能体，接近 1 则是有耐心的，一个序列的回报为 r0 + gamma r1 + gamma^2 r2 + ...。在终止步没有下一状态，目标就是奖励本身。

# One Q-learning update on a table of 2 states x 3 actions (short, flat, long)
import numpy as np

def q_update(q, state, action, reward, next_state, alpha, gamma):
    updated = q.copy()
    target = reward if next_state is None else reward + gamma * updated[next_state].max()
    updated[state, action] += alpha * (target - updated[state, action])
    return updated

q = np.zeros((2, 3))
q = q_update(q, 0, 2, 1.0, 1, 0.5, 0.9)               # in state 0 we went long and earned 1.0
print(q[0])                                           # [0.  0.  0.5]   moved halfway (alpha 0.5) toward the target of 1.0
q = q_update(q, 0, 2, 1.0, 1, 0.5, 0.9)
print(q[0])                                           # [0.   0.   0.75]   a second visit moves it further

探索，以及一个完整的训练循环示例

总是采取已知最佳动作的智能体永远不会发现别的动作是否更好。epsilon-贪心解决了这个问题：以概率 epsilon 探索一个随机动作，否则利用 Q 最大的动作，并列时取第一个。epsilon 通常从高值开始并逐渐衰减。循环就是：观察状态，选动作，获得奖励和下一状态，更新 Q，重复。在一个上涨日之后往往还是上涨日的玩具市场里，看到“昨天上涨”的智能体会学到持多头有回报，两个状态三个动作的表就足以展示这一点。

# A toy market where an up day tends to be followed by another up day; the agent learns to follow the trend
import random
import numpy as np

rng = random.Random(1)
positions = [-1, 0, 1]                                # action 0 = short, 1 = flat, 2 = long
q = np.zeros((2, 3))                                  # state 0 = yesterday down, state 1 = yesterday up
alpha, gamma, cost = 0.1, 0.9, 0.0005
state, previous = 1, 0
for step in range(20000):
    epsilon = max(0.05, 1 - step / 10000)
    u, explore = rng.random(), rng.randrange(3)
    action = explore if u < epsilon else int(np.argmax(q[state]))
    up = rng.random() < (0.65 if state == 1 else 0.35)           # the trend continues 65% of the time
    change = 0.01 if up else -0.01
    reward = positions[action] * change - cost * abs(positions[action] - previous)
    next_state = 1 if up else 0
    target = reward + gamma * q[next_state].max()
    q[state, action] += alpha * (target - q[state, action])
    state, previous = next_state, positions[action]

print(int(np.argmax(q[0])), int(np.argmax(q[1])))     # 0 2   after a down day go short, after an up day go long

# The discounted return: with gamma below 1 a reward now is worth more than the same reward later
def discounted_return(rewards, gamma):
    total = 0.0
    for reward in reversed(rewards):
        total = reward + gamma * total
    return total

rewards = [1.0, 1.0, 1.0]
print(discounted_return(rewards, 0.5))               # 1.75   1 + 0.5 + 0.25
print(discounted_return(rewards, 1.0))               # 3.0    a patient agent: a plain sum
print(discounted_return(rewards, 0.0))               # 1.0    a short-sighted agent: only the first reward counts

• 智能体观察状态、行动、获得奖励（盈利减成本），并学习哪些动作有回报；回报是折现和 r0 + gamma r1 + gamma^2 r2 + ...。
• Q 学习更新 Q[s, a] += alpha * (reward + gamma * max Q[s'] - Q[s, a])（终止步只用 reward），这里不会原地修改旧表。
• epsilon-贪心以概率 epsilon 探索，否则利用最大的 Q；在不同时期上训练和测试，计入真实成本，并与简单基线比较。

下一节：毕业项目，把一切整合在一起的端到端定价或策略库。

实验：评分内容

• 用 choose_action 选择探索或利用25 分
• 用 q_update 从一步中学习25 分
• 在 step_reward 中为交易付费25 分
• 用 discounted_return 累加未来25 分