Reinforcement learning for trading

Learning by trying

In supervised learning every example comes with the right answer. Trading is not like that: nobody tells you the best position, you act and the market answers with a profit or a loss. Reinforcement learning models this as an agent that observes a state, takes an action, receives a reward and moves to a new state, with the goal of maximising the total reward over time. For trading the state can be recent returns or the current position, the actions buy, hold or sell, and the reward the profit of the step minus costs. The difficulty is that actions have consequences later and the agent must discover what pays by trying.

# Reward of one step: profit from the position held, minus the cost of changing position
def step_reward(position, price_change, previous_position, cost):
    return position * price_change - cost * abs(position - previous_position)

print(round(step_reward(1, 0.5, 0, 0.1), 4))         # 0.4    went long, the price rose 0.5, paid 0.1 to trade
print(round(step_reward(-1, 0.5, 1, 0.1), 4))        # -0.7   flipped from long to short just before a rise: lost 0.5 and paid 0.2
print(step_reward(1, 0.5, 1, 0.1))                   # 0.5    holding the same position costs nothing

Q-learning: value of an action in a state

Q-learning keeps a table Q[state, action]: the total reward expected if you take action in state and act well afterwards. After each step it nudges the entry toward a better estimate, the target: the reward just received plus the discounted value of the best action in the next state, reward + gamma * max(Q[next_state]). The update is Q[s, a] += alpha * (target - Q[s, a]), with a learning rate alpha. The discount factor gamma (0 to 1) says how much future rewards matter: gamma = 0 is a short-sighted agent, near 1 a patient one, and the return of a sequence is r0 + gamma r1 + gamma^2 r2 + .... At a terminal step there is no next state and the target is just the reward.

# One Q-learning update on a table of 2 states x 3 actions (short, flat, long)
import numpy as np

def q_update(q, state, action, reward, next_state, alpha, gamma):
    updated = q.copy()
    target = reward if next_state is None else reward + gamma * updated[next_state].max()
    updated[state, action] += alpha * (target - updated[state, action])
    return updated

q = np.zeros((2, 3))
q = q_update(q, 0, 2, 1.0, 1, 0.5, 0.9)               # in state 0 we went long and earned 1.0
print(q[0])                                           # [0.  0.  0.5]   moved halfway (alpha 0.5) toward the target of 1.0
q = q_update(q, 0, 2, 1.0, 1, 0.5, 0.9)
print(q[0])                                           # [0.   0.   0.75]   a second visit moves it further

Exploring, and a worked training loop

An agent that always takes the best action it knows never finds out whether another is better. Epsilon-greedy fixes that: with probability epsilon it explores a random action, otherwise it exploits the action with the largest Q, with ties going to the first. Epsilon usually starts high and decays. The loop is then: observe the state, choose an action, receive the reward and the next state, update Q, repeat. In a toy market where an up day tends to be followed by another up day, the agent that sees "yesterday was up" learns that holding long pays, and a table of two states and three actions is enough to show it.

# A toy market where an up day tends to be followed by another up day; the agent learns to follow the trend
import random
import numpy as np

rng = random.Random(1)
positions = [-1, 0, 1]                                # action 0 = short, 1 = flat, 2 = long
q = np.zeros((2, 3))                                  # state 0 = yesterday down, state 1 = yesterday up
alpha, gamma, cost = 0.1, 0.9, 0.0005
state, previous = 1, 0
for step in range(20000):
    epsilon = max(0.05, 1 - step / 10000)
    u, explore = rng.random(), rng.randrange(3)
    action = explore if u < epsilon else int(np.argmax(q[state]))
    up = rng.random() < (0.65 if state == 1 else 0.35)           # the trend continues 65% of the time
    change = 0.01 if up else -0.01
    reward = positions[action] * change - cost * abs(positions[action] - previous)
    next_state = 1 if up else 0
    target = reward + gamma * q[next_state].max()
    q[state, action] += alpha * (target - q[state, action])
    state, previous = next_state, positions[action]

print(int(np.argmax(q[0])), int(np.argmax(q[1])))     # 0 2   after a down day go short, after an up day go long

# The discounted return: with gamma below 1 a reward now is worth more than the same reward later
def discounted_return(rewards, gamma):
    total = 0.0
    for reward in reversed(rewards):
        total = reward + gamma * total
    return total

rewards = [1.0, 1.0, 1.0]
print(discounted_return(rewards, 0.5))               # 1.75   1 + 0.5 + 0.25
print(discounted_return(rewards, 1.0))               # 3.0    a patient agent: a plain sum
print(discounted_return(rewards, 0.0))               # 1.0    a short-sighted agent: only the first reward counts

• An agent observes a state, acts, receives a reward (profit minus costs) and learns which actions pay; the return is the discounted sum r0 + gamma r1 + gamma^2 r2 + ....
• Q-learning updates Q[s, a] += alpha * (reward + gamma * max Q[s'] - Q[s, a]) (just reward at a terminal step) and never edits the old table in place here.
• Epsilon-greedy explores with probability epsilon and otherwise exploits the largest Q; train and test on separate periods, charge realistic costs and compare with simple baselines.

Next up: the capstone, an end-to-end pricing or strategy library that puts everything together.

Lab · what's graded

• Explore or exploit with choose_action25 pts
• Learn from one step with q_update25 pts
• Pay for trading in step_reward25 pts
• Add up the future with discounted_return25 pts