This work is based on my summer internship at IIT Delhi in 2015. The project derives its base from the machine learning and data mining framework called windML. The windML framework provides various tools for machine learning based analysis of real-time wind energy time series data. This project further develops this framework to implement a Bootstrap Aggregation (Bagging) based system to improve upon the already available spatio-temporal regression models for accurate wind power forecasting. The project is implemented in python using windML, and Sklearn python libraries.
Why Wind Power Forecasting?
With the ever-increasing energy demand and increasing focus towards reducing global warming, the renewable energy sources like wind and solar energy are going to play a crucial role in future energy infrastructure. But this increased penetration of wind energy (and solar for that matter) is not without its problems. The major issue with the wind energy is its high level of uncertainty which can lead to wide-scale instability in the connected power grids. To tackle this problem we need to generate high accuracy prediction model for these energy sources. Further with the availability of a large amount of time series electricity generation data the prediction task is much more solvable than it ever was.
Wind Power Forecasting as a Regression Problem
Wind power forecasting problem for a single turbine can be formulated as a time series regression problem. Depending on the time of forecast it can be classified into short-term (30, 60, 90 minutes ahead), medium-term (Day Ahead, Week Ahead) and long-term forecasting problem. In this case, we would tackle a short-term forecasting problem by using the power data at , ….., and time instants to forecast power at $latex t+\lambda$ time instant. So, at any time instant the power can be seen as a function of power , ….. and . Further, the future power can also be correlated with the difference of power at the previous time instances (, …..and ). Here is called time horizon and is called feature window. So, in this case we have () features to predict power .
Spatio-Temporal Regression model for Wind Power Forecasting
To forecast the power generated by a turbine, the regression model discussed so far uses of its own time series. To further improve the forecast we can use time series data of its neighbor turbines. As these neighbor turbines have to face the same wind (delayed or advanced depending upon the wind direction), their power time series data is easy to correlate with the target turbine. If in a given area, there are neighbor turbines, the total number features are now. So, the power produced by target turbine at time () is a function of features denoted by , where denotes the turbine (varying between to ) and denotes previous time instant (varying between and ).
Bootstrap Aggregation of Regression Models
So far the discussed regression problem can be formulated as:
Here is function that maps the forecasted power to the spatio-temporal features. This function is approximated by a machine-learning model as a function . But being approximate this function is not unique. We can develop different approximate models using different machine-learning techniques (like Support Vector Machines, Random Forest, Neural Network etc.) and by varying hyper-parameters of a single technique. All of these functions show some overfitting resulting in error during the forecast. One way of reducing this overfitting is to apply an ensemble technique called weighted bootstrap aggregation (or weighted bagging) to combine all machine-learning approximated functions. If we train different machine-learning methods to find approximations of the mapping function , the overall regression problem can be formulated as:
Here $latex \alpha_i$ is the weight for approximation .
Bootstrap Aggregation: Python Implementation
class BA_ensemble(object): ''' Variable 'estimators' can be seen as a list of models. Each element of the list is defined as: ['Model_Name', [Model Paramters]] ML Models with parameters: 1. 'SVR': kernal = 'rbf', degree = 3, gamma = 1e-4, C = 100, epsilon = 0.1 2. 'RF': max_depth = 30, n_estimators = 10, n_jobs = -1, random_state = 0 3. 'KNN': n_neighbors = 5, weights = 'uniform', leaf_size = 30 4. 'Ridge': alpha = 0.5 5. 'Bayesian': alpha_1 = 1e-6, alpha_2 = 1e-6 6. 'LR': fit_intercept =True , n_jobs = -1, normalize =False 7. 'GBR': learning_rate = 0.1, n_estimators = 100, max_depth = 3, loss='ls' ''' def __init__(self, train_x, train_y, estimators): ''' Creates and Trains all the models ''' self.reg =  for d in range(len(train_x)): self.X_train = train_x[d] self.Y_train = train_y[d] for i in range(len(estimators)): print i self.reg.append(Estimator(info=estimators[i])) if estimators[i] == 'SVR': self.reg[-1].train(SVR(kernel=estimators[i], degree=estimators[i], gamma = estimators[i] , C = estimators[i], epsilon=estimators[i]).fit(self.X_train,self.Y_train)) elif estimators[i] == 'RF': self.reg[-1].train(RandomForestRegressor(max_depth=estimators[i], n_estimators=estimators[i], n_jobs=estimators[i], random_state=estimators[i]).fit(self.X_train,self.Y_train)) elif estimators[i] == 'KNN': self.reg[-1].train(KNeighborsRegressor(n_neighbors=estimators[i], weights=estimators[i], leaf_size=estimators[i]).fit(self.X_train,self.Y_train)) elif estimators[i] == 'Ridge': self.reg[-1].train(linear_model.Ridge(alpha = estimators[i]).fit(self.X_train,self.Y_train)) elif estimators[i] == 'Bayesian': self.reg[-1].train(linear_model.BayesianRidge(alpha_1 = estimators[i], alpha_2 = estimators[i]).fit(self.X_train,self.Y_train)) elif estimators[i] == 'LR': self.reg[-1].train(linear_model.LinearRegression(fit_intercept = estimators[i], n_jobs = estimators[i], normalize = estimators[i]).fit(self.X_train,self.Y_train)) elif estimators[i] == 'GBR': self.reg[-1].train(GradientBoostingRegressor(learning_rate = estimators[i], n_estimators = estimators[i], max_depth = estimators[i], loss=estimators[i]).fit(self.X_train,self.Y_train)) def predict(self, test_x, test_y): ''' Predicts/Forecasts using all the models ''' self.X_test = test_x self.Y_test = test_y self.y_hats =  for i in range(len(self.reg)): print i self.reg[i].predictions(self.reg[i].regressor.predict(self.X_test)) # Mean Square Error self.reg[i].PI_update('mse', mean_squared_error(self.Y_test, self.reg[i].y_hat)) def aggregate(self, alpha = ): ''' Ensembles (Weighted Bagging/Bootstrap-Aggregation) all the predictions ''' if len(alpha) <span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span><span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span><span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>< len(self.reg): for i in range(len(self.reg)-len(alpha)): alpha.append(1) self.alpha = alpha self.y_hat =  for i in range(len(self.X_test)): self.y_hat.append(0) for j in range(len(self.reg)): self.y_hat[-1] = self.y_hat[-1] + self.alpha[j] * self.reg[j].y_hat[i] self.y_hat[-1] = self.y_hat[-1]/(float(sum(self.alpha))) # Mean Square Error self.mse = mean_squared_error(self.Y_test, self.y_hat)
The above slide shows the prediction made by the weighted bootstrap aggregation strategy in 10 different turbine locations. The red line shows the predicted power while the blue line shows the actual measurements. Clearly, the proposed strategy is able to make predictions close to the actual power value. The above bootstrap aggregation is only using 5 regression models: Random Forest, Ridge Regression, Bayesian Learning, Gradient Boosting Regression and K-Nearest-Neighbors.
The bar graph clearly shows that compared to single regression model the bootstrap aggregation is giving much lower mean square error in all cases.
The above results were taken by using just 5 regression model with equal weights. The prediction accuracy can be further improved by taking more models and optimizing the model weights.