Backtest: Preprocessing

Filter

• We might want to drop stocks with prices below some threshold (“drop penny stocks”).
• We might want to drop stocks of certain sizes.
  • Maybe fewer opportunities in large caps?
  • Maybe drop microcaps because they’re harder to trade?

Deal with missing data

• We need to have valid data for all featurews in each observation.
• Can drop NaNs.
• Or can fill NaNs.
  • Maybe use median or mean for the feature in that time period.
  • Or median or mean for the feature for that industry in that time period.

Transform cross-sections

• Rather than pooling across dates and scaling/transforming, it is probably better in our application to scale/transform each cross-section independently.
• We can group by date and apply transformations to the groupby object.

• Let’s look at an example: QuantileTransformer.
• Quantile transformer maps quantiles of the sample distribution to quantiles of a target distribution (uniform or normal).
• So, the transformed data has the target distribution.

Industries

• Industry membership can be used in various ways.
• One example is adding industry dummy variables as features.
• There are various ways to define industries, most of which are based on SIC codes.
• We’ll look at the example of the Fama-French 12 industry classification.

• The SIC code ranges for each industry are in siccodes12.csv, which was obtained from French’s data library.

inds = pd.read_csv(
  "files/siccodes12.csv", 
  index_col="industry"
)
inds

• Find the range in which an SIC code lies to find its industry.
• If it is not in any of the ranges, then the industry is “Other.”

def industry(sic):
  try:
    return inds[(inds.start<=sic)&(sic<=inds.end)].index[0]
  except:
    return "Other"

• We could loop over all observations and define the industry for each observation.
• But it’s faster to pull the unique SIC codes, define the industry for each SIC code, and then do a one-to-many merge into the dataframe of all observations.

codes = pd.Series({code: industry(code) for code in df.siccd.unique()})
codes.name = "industry" 
codes.index.name = "siccd"
df = df.merge(codes, on="siccd")

Polynomial features

• Polynomial features with degree=2 adds products and squares of features.
• Degree=3 adds a*b*c, a**3, etc.
• Adding products facilitates including interactions between variables.
• We can use polynomial features in a pipeline.

Pipeline

• The advantage of putting transformations in a pipeline is that the exact same transformations will be automatically applied to new data when predictions are made.
• sklearn code is

from sklearn.pipeline import make_pipeline
pipe = make_pipeline(transform1, transform2, ..., model)
pipe.fit(...)
pipe.predict(...)

• We will illustrate with OneHotEncoder (to create dummies) and PolynomialFeatures
• OneHotEncoder is only applied to the “industry” column, so we use make_column_transformer to apply different transformations to different columns.