以kaggle上的House Prices: Advanced Regression Techniques为例讲述线性回归
from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV, ElasticNetCV
1、回归模型
（1）一般线性回归：Linear Regression without regularization
from sklearn.linear_model import LinearRegressionlr = LinearRegression()lr.fit(X_train, y_train)model = lrrmse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring = scorer, cv = 10)) #根均方误差y_train_pred = lr.predict(X_train)y_test_pred = lr.predict(X_test)
RMSE on Training set : 0.3887932081355326RMSE on Test set : 0.42716116210823174
（2）岭回归：Linear Regression with Ridge regularization——L2正则化
采用正则化：处理共线性，滤除噪声，防止过拟合；
原理：引入额外的信息偏差，来惩罚极端参数权重；
from sklearn.linear_model import RidgeCVridge = RidgeCV(alphas = [0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6, 10, 30, 60])ridge.fit(X_train, y_train)alpha = ridge.alpha_ridge = RidgeCV(alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, alpha * .85, alpha * .9, alpha * .95, alpha, alpha * 1.05, alpha * 1.1, alpha * 1.15, alpha * 1.25, alpha * 1.3, alpha * 1.35, alpha * 1.4], cv = 10)ridge.fit(X_train, y_train)alpha = ridge.alpha_model = ridgermse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring = scorer, cv = 10)) #根均方误差y_train_rdg = ridge.predict(X_train)y_test_rdg = ridge.predict(X_test)
Ridge RMSE on Training set : 0.11540572328450796Ridge RMSE on Test set : 0.11642721377799556
（3）使用Lasson正则化的线性回归：Linear Regression with Lasso regularization——L1正则化
LASSO：以最小绝对收缩和选择算子为代表，简单地用权重的绝对值之和替换权重的平方。
与L2正则化相反，L1正则化产生稀疏特征向量，使得大多数特征权重为0。
如果高维数据集具有许多不相关的特征，则稀疏性会很有帮助。
from sklearn.linear_model import LassoCVlasso = LassoCV(alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1], max_iter = 50000, cv = 10)lasso.fit(X_train, y_train)alpha = lasso.alpha_lasso = LassoCV(alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, alpha * .85, alpha * .9, alpha * .95, alpha, alpha * 1.05, alpha * 1.1, alpha * 1.15, alpha * 1.25, alpha * 1.3, alpha * 1.35, alpha * 1.4], max_iter = 50000, cv = 10)lasso.fit(X_train, y_train)alpha = lasso.alpha_model = lassormse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring = scorer, cv = 10)) #根均方误差y_train_las = lasso.predict(X_train)y_test_las = lasso.predict(X_test)
Lasso RMSE on Training set : 0.11411150837458062Lasso RMSE on Test set : 0.11583213221750702
（4）使用ElsticNet正则化的线性回归：Linear Regression with ElasticNet regularization——L1和L2正则化
ElasticNet是Ridge和Lasso回归之间的折中，用L1产生稀疏性，用L2克服Lasso的一些限制（例如LASSO不能选择比观察更多的特征）
import sklearn.linear_model import ElasticNetCVelasticNet = ElasticNetCV(l1_ratio = [0.1, 0.3, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 1], alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6], max_iter = 50000, cv = 10)elasticNet.fit(X_train, y_train)alpha = elasticNet.alpha_ratio = elasticNet.l1_ratio_elasticNet = ElasticNetCV(l1_ratio = [ratio * .85, ratio * .9, ratio * .95, ratio, ratio * 1.05, ratio * 1.1, ratio * 1.15], alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6], max_iter = 50000, cv = 10)elasticNet.fit(X_train, y_train)if (elasticNet.l1_ratio_ > 1): elasticNet.l1_ratio_ = 1 alpha = elasticNet.alpha_ratio = elasticNet.l1_ratio_elasticNet = ElasticNetCV(l1_ratio = ratio, alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, alpha * .85, alpha * .9, alpha * .95, alpha, alpha * 1.05, alpha * 1.1, alpha * 1.15, alpha * 1.25, alpha * 1.3, alpha * 1.35, alpha * 1.4], max_iter = 50000, cv = 10)elasticNet.fit(X_train, y_train)if (elasticNet.l1_ratio_ > 1): elasticNet.l1_ratio_ = 1 alpha = elasticNet.alpha_ratio = elasticNet.l1_ratio_model = elasticNetrmse= np.sqrt(-cross_val_score(model, X_train, y_train, scoring = scorer, cv = 10)) #根均方误差y_train_ela = elasticNet.predict(X_train)y_test_ela = elasticNet.predict(X_test)
ElasticNet RMSE on Training set : 0.11411150837458062ElasticNet RMSE on Test set : 0.11583213221750702
此处ratio=1，即完全使用L1正则化，LASSO回归