Sets

A collection with no repeats

A set holds each value at most once and has no position: {"AAPL", "MSFT", "GOOG"}. Adding a value that is already there changes nothing. set(items) turns any iterable into a set, which is the quickest way to remove duplicates, and len then counts the distinct values. Because a set has no order, you cannot index it, and the order you see when printing it is not something to rely on. An empty set is set(): {} is an empty dictionary.

symbols = ["AAPL", "MSFT", "AAPL", "GOOG", "MSFT"]

# set() removes repeats
unique = set(symbols)
print(len(symbols), len(unique))      # 5 3

# Adding something already there changes nothing
unique.add("AAPL")
unique.add("TSLA")
print(len(unique))                    # 4

# {} is a dict; an empty set is set()
print(type({}), type(set()))          # <class 'dict'> <class 'set'>

Fast membership tests

x in some_set answers almost instantly, even with millions of values, because a set finds a value by its hash instead of scanning. The same test on a list checks the items one by one, which gets slower as the list grows. So if you ask in many times against the same collection, such as checking each incoming order against a list of banned symbols, turn that collection into a set once. Set members must be hashable, like dictionary keys.

# Membership: a set is the fast structure for "is it in there?"
banned = {"GME", "AMC", "BBBY"}
orders = ["AAPL", "GME", "MSFT", "AMC"]

allowed = [symbol for symbol in orders if symbol not in banned]
print(allowed)                       # ['AAPL', 'MSFT']

# Sets only keep hashable values
try:
    {["AAPL"]}
except TypeError as error:
    print("not hashable:", error)

Set arithmetic

Sets combine like in school maths. a | b is the union (in either), a & b the intersection (in both), a - b the difference (in a but not in b) and a ^ b the symmetric difference (in exactly one). Each also exists as a method, such as a.union(b), which accepts any iterable. All of them return a new set and leave a and b alone. This is exactly what you want to compare two universes of symbols or two lists of holdings.

# Comparing yesterday's and today's universe
yesterday = {"AAPL", "MSFT", "GOOG", "IBM"}
today = {"AAPL", "MSFT", "TSLA"}

print(sorted(yesterday | today))   # ['AAPL', 'GOOG', 'IBM', 'MSFT', 'TSLA']
print(sorted(yesterday & today))   # ['AAPL', 'MSFT']
print(sorted(today - yesterday))   # ['TSLA']   (new listings)
print(sorted(yesterday - today))   # ['GOOG', 'IBM']   (dropped)
print(sorted(yesterday ^ today))   # ['GOOG', 'IBM', 'TSLA']

Changing a set, and keeping the order

add(x) inserts a value, update(items) inserts several, discard(x) removes a value if present (silently), and remove(x) removes it but raises KeyError if it is absent. A set forgets order, so when order matters you need a list: sorted(some_set) gives a sorted list, and to remove duplicates while keeping the order in which values first appeared, loop and remember what you have seen in a set. The seen check is fast because it is a set.

# Remove repeats and keep the first-seen order
symbols = ["MSFT", "AAPL", "MSFT", "GOOG", "AAPL"]
seen = set()
ordered = []
for symbol in symbols:
    if symbol not in seen:
        seen.add(symbol)
        ordered.append(symbol)
print(ordered)             # ['MSFT', 'AAPL', 'GOOG']

# discard is quiet, remove is strict
seen.discard("TSLA")
try:
    seen.remove("TSLA")
except KeyError:
    print("TSLA was not in the set")

• set(items): keep each value once; len(set(items)) counts distinct values.
• x in s is fast; a | b, a & b, a - b, a ^ b build new sets.
• Sets have no order: use sorted(s) when you need a list.

Next up: reading and writing text files, so your programs can keep data between runs.

Lab · what's graded

• Count different symbols with distinct_count25 pts
• Spot symbols that appeared with new_listings25 pts
• Find the overlap of two portfolios with common_symbols25 pts
• Remove repeats but keep the order with dedupe_keep_order25 pts