Stock markets are hard to predict because they depend on a tremendous amount of factors including historical data and economic news. Let’s try to improve the explanation of stock markets by including some related search terms from the Google search engine. This work aims at evaluating the use of those search data in nowcasting the stock market. We want to answer the following research question:
- What are the best models to predict the near term price movement of stock markets, more specifically the S&P 500 index?
- Can we improve our predictions using Google Trends? If yes, to what extent?
- Which Google Trends keywords are most relevant in explaining stock market price variations and why?
This work is based on the paper Predicting the Present with Google Trends 1 from HYUNYOUNG CHOI and HAL VARIAN. The original paper explains how Google Trends help improve forecasts with a baseline model. In contrast, we will explore different prediction models to choose the optimal one to forecast the price movement of one of the most popular US Stock index, the S&P 500. Then, Google Trends data will be introduced to see if we can improve these predictions. A Google Trends keywords selection will be performed beforehand to keep the most relevant ones. In the end, we’ll have the possibility to test our models in predicting the stock prices of recession periods from the S&P 500.
Predicting the Present with Google Trends
Let’s get started with a bit of context. We will now give a short and general summary of the work done in the original paper. The authors propose to use google trends data to improve the predicitions of several economic indicators. The table below shows the 4 economic series considered and the Google Trends term associated:
| Economic indicator | Google Trends |
|---|---|
| Automobile sales | Trucks & SUVs, Automotive Insurance |
| Unemployment claims | Jobs, Welfare & Unemployment |
| Hong Kong arrivals | Hong Kong (by countries of origin) |
| Consumer confidence | Crime & Justice, Trucks & SUVs, Hybrid & Alternative Vehicles |
The Google Trends data are time series measuring of the number of Google research for the subject considered.
For each of those time series, the authors compare the usage of expanding window Auto-Regressive models with and without including the related search terms. The models have either a lag of -1 or a lag of -1 and -12. The series of economic indicators can be pre-processed to be log-transformed and are sometimes seasonally adjusted.
Their results show that including the Google Trends data reduces the errors in predictions by 5 to 20 percent.
Extension to stock markets
We now present our work related to stock market data. Let’s start with a description of the data used.
Data
There are two main datasets. The first one is the economic indicator of interest, the S&P 500 price. And the second one is Google Trends data. These two data sets are time series ranging from early 2004 to nowadays. The data sets are available on daily data points but we will resample them to keep only the last weekly point so that the data has weekly points.
S&P 500
The S&P 500 is a stock market index that measures the stock performance of 500 large companies listed on stock exchanges in the United States. This index is one of the most used to represent US financial markets.
We use the S&P500 data for our analysis, it was downloaded from Yahoo Finance. The data is a table containing the daily values of the S&P500, we have 6 value for each working date (all in $):
- Open: the value of the stock at the opening
- High: the highest value of the stock during the day
- Low: the lowest value of the stock during the day
- Close: the value of the stock at closing
- Adjusted Close: the value of the stock at closing (corrections for stock splits, dividends, and rights offerings)
- Volume: The total amount traded
Let’s start the data exploration for the S&P 500. We will consider the Adjusted Close value for the stock as it is the standard benchmark used by investors to track its performance over time 2.
We first plot the evolution of the Adjusted Close over the year. We will refer to this as the signal.
We can see the big drop in 2008/2009 from the subprime crisis and also around March/April 2020 with the first COVID-19 wave.
Seasonality
We then use a simple decomposition tool from statsmodel to decompose our signal. We choose to use the additive model and our signal is then decomposed as:
with
- Y[t] is the original signal, Adjusted Close for us
- T[t] is the trend signal
- S[t] is the seasonal signal
- e[t] is the residual signal
The results can be seen in the figure below:
There seems to be a monthly seasonality. We can see this from all the recurring up and down in a year. Which seems to correlate with a monthly period.
Autocorrelation
Let’s now look at the partial autocorrelation function of our signal. It will be important to confirm the usage of Auto-Regressive models. The h order (lag) partial autocorrelation is defined by
Now we show the partial autocorrelation function up to the 20th order.
Since the function becomes silent after the first order, then a simple Autoregressive model with a lag of 1 would be sufficient to make predictions 3. However, we will look at more complex models.
First difference
Finally, let’s have a look at the first difference of the signal. This is for each time t we compute the difference of the signal at time t and t-1:
. We show the resulting signal below:
We can see exponential behavior in 2008/2009 and mainly in March/April 2020. We had similar behavior in the residuals in the seasonality decomposition. To counter this we will use the logarithm of the Adjusted Close to have a more smooth behavior. Note that in the original paper, the authors used the logarithm of the target variable so this is consistent with their results. The new signal, which will use in our model, is shown below: