Let's apply everything we have learned about linear regression to a new dataset we collected ourselves. This is stock market data for a few popular tech companies for the previous five years from 2017 to 2022, including some macro-economy data which we will use as independent variables. The data is pre-loaded as a Pandas DataFrame in the variable data. Let's take a look at this dataset.

This dataset contains 7 columns, however Python only displays the first and last 3 columns. The first job a data scientist has when getting a new dataset is to understand what each column represents, so here are the column descriptions.

• date: The date of the stock market data. Covers 5 years from the start of 2017 till the end of 2022.
• aapl: The stock price of Apple Inc.
• msft: The stock price of Microsoft Corporation.
• googl: The stock price of Alphabet Inc.
• interest_rate: The Federal Funds Effective Rate (aka the interest rate) of the US Federal Reserve.
• sp500: The S&P 500 index, which is a stock market index that measures the stock performance of 500 large companies listed on stock exchanges in the United States.
• market_cap: The Wilshire US Large-Cap Total Market Index, which is the total market value of all the large companies listed on stock exchanges in the United States.

Visualizing the data

Let's start by plotting the stock prices of the first company we have data for, Apple. We use the Seaborn library to plot the data.

Great, we can see there is a kind of upward trend in the stock price, this is called a time-trend because it is a trend that changes over time. Next, let's look at the stock prices of all three companies at once. To do this, we will need to map the data from wide to long so we can get the prices together in one column, and we will add a column to indicate which company the price belongs to. We use the melt method on a Pandas DataFrame object to map the data from wide to long, see our Pivot Longer lesson for a review on this method. Then we use the same Seaborn plotting code as before, adding in the hue to draw the lines for each company in different colors.

Linear regression

Great, let's next try to predict the stock price of Apple using linear regression. We will use the same code as in the Prediction with Linear Regression lesson, but we will use the macro-economy data as independent variables: interest_rate, sp500, and market_cap columns as independent variables. In the following code block we run this linear regression, calculate the MSE, and plot the predictions against the actual stock prices.

The MSE is a lot higher than with the iris dataset, but this is to be expected considering the stock market is a lot more volatile than flower growth. Let's get the MSE for the remaining two companies then keep them in mind for comparison with the next modelling strategy we will learn in the next lesson called regression trees.