Generalized Least Squares

In [1]:

<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
<span class="kn">import</span> <span class="nn">statsmodels.api</span> <span class="kn">as</span> <span class="nn">sm</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">statsmodels.iolib.table</span> <span class="kn">import</span> <span class="p">(</span><span class="n">SimpleTable</span><span class="p">,</span> <span class="n">default_txt_fmt</span><span class="p">)</span>

The Longley dataset is a time series dataset:

In [2]:

<span class="n">data</span> <span class="o">=</span> <span class="n">sm</span><span class="o">.</span><span class="n">datasets</span><span class="o">.</span><span class="n">longley</span><span class="o">.</span><span class="n">load</span><span class="p">()</span>
<span class="n">data</span><span class="o">.</span><span class="n">exog</span> <span class="o">=</span> <span class="n">sm</span><span class="o">.</span><span class="n">add_constant</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">exog</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">exog</span><span class="p">[:</span><span class="mi">5</span><span class="p">])</span>

[[      1.       83.   234289.     2356.     1590.   107608.     1947. ]
 [      1.       88.5  259426.     2325.     1456.   108632.     1948. ]
 [      1.       88.2  258054.     3682.     1616.   109773.     1949. ]
 [      1.       89.5  284599.     3351.     1650.   110929.     1950. ]
 [      1.       96.2  328975.     2099.     3099.   112075.     1951. ]]

Let's assume that the data is heteroskedastic and that we know the nature of the heteroskedasticity. We can then define sigma and use it to give us a GLS model

First we will obtain the residuals from an OLS fit

In [3]:

<span class="n">ols_resid</span> <span class="o">=</span> <span class="n">sm</span><span class="o">.</span><span class="n">OLS</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">endog</span><span class="p">,</span> <span class="n">data</span><span class="o">.</span><span class="n">exog</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">()</span><span class="o">.</span><span class="n">resid</span>

Assume that the error terms follow an AR(1) process with a trend:

$\epsilon_i = \beta_0 + \rho\epsilon_{i-1} + \eta_i$

where $\eta \sim N(0,\Sigma^2)$

and that $\rho$ is simply the correlation of the residual a consistent estimator for rho is to regress the residuals on the lagged residuals

In [4]:

<span class="n">resid_fit</span> <span class="o">=</span> <span class="n">sm</span><span class="o">.</span><span class="n">OLS</span><span class="p">(</span><span class="n">ols_resid</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span> <span class="n">sm</span><span class="o">.</span><span class="n">add_constant</span><span class="p">(</span><span class="n">ols_resid</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span><span class="o">.</span><span class="n">fit</span><span class="p">()</span>
<span class="k">print</span><span class="p">(</span><span class="n">resid_fit</span><span class="o">.</span><span class="n">tvalues</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">print</span><span class="p">(</span><span class="n">resid_fit</span><span class="o">.</span><span class="n">pvalues</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>

-1.43902298399
0.173784447884

While we don't have strong evidence that the errors follow an AR(1) process we continue

In [5]:

<span class="n">rho</span> <span class="o">=</span> <span class="n">resid_fit</span><span class="o">.</span><span class="n">params</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>

As we know, an AR(1) process means that near-neighbors have a stronger relation so we can give this structure by using a toeplitz matrix

In [6]:

<span class="kn">from</span> <span class="nn">scipy.linalg</span> <span class="kn">import</span> <span class="n">toeplitz</span>
 
<span class="n">toeplitz</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">))</span>

Out[6]:

array([[0, 1, 2, 3, 4],
       [1, 0, 1, 2, 3],
       [2, 1, 0, 1, 2],
       [3, 2, 1, 0, 1],
       [4, 3, 2, 1, 0]])

In [7]:

<span class="n">order</span> <span class="o">=</span> <span class="n">toeplitz</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">ols_resid</span><span class="p">)))</span>

so that our error covariance structure is actually rho**order which defines an autocorrelation structure

In [8]:

<span class="n">sigma</span> <span class="o">=</span> <span class="n">rho</span><span class="o">**</span><span class="n">order</span>
<span class="n">gls_model</span> <span class="o">=</span> <span class="n">sm</span><span class="o">.</span><span class="n">GLS</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">endog</span><span class="p">,</span> <span class="n">data</span><span class="o">.</span><span class="n">exog</span><span class="p">,</span> <span class="n">sigma</span><span class="o">=</span><span class="n">sigma</span><span class="p">)</span>
<span class="n">gls_results</span> <span class="o">=</span> <span class="n">gls_model</span><span class="o">.</span><span class="n">fit</span><span class="p">()</span>

Of course, the exact rho in this instance is not known so it it might make more sense to use feasible gls, which currently only has experimental support.

We can use the GLSAR model with one lag, to get to a similar result:

In [9]:

<span class="n">glsar_model</span> <span class="o">=</span> <span class="n">sm</span><span class="o">.</span><span class="n">GLSAR</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">endog</span><span class="p">,</span> <span class="n">data</span><span class="o">.</span><span class="n">exog</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">glsar_results</span> <span class="o">=</span> <span class="n">glsar_model</span><span class="o">.</span><span class="n">iterative_fit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">glsar_results</span><span class="o">.</span><span class="n">summary</span><span class="p">())</span>

                           GLSAR Regression Results
==============================================================================
Dep. Variable:                      y   R-squared:                       0.996
Model:                          GLSAR   Adj. R-squared:                  0.992
Method:                 Least Squares   F-statistic:                     295.2
Date:                Tue, 02 Dec 2014   Prob (F-statistic):           6.09e-09
Time:                        12:53:58   Log-Likelihood:                -102.04
No. Observations:                  15   AIC:                             218.1
Df Residuals:                       8   BIC:                             223.0
Df Model:                           6
Covariance Type:            nonrobust
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
const      -3.468e+06   8.72e+05     -3.979      0.004     -5.48e+06 -1.46e+06
x1            34.5568     84.734      0.408      0.694      -160.840   229.953
x2            -0.0343      0.033     -1.047      0.326        -0.110     0.041
x3            -1.9621      0.481     -4.083      0.004        -3.070    -0.854
x4            -1.0020      0.211     -4.740      0.001        -1.489    -0.515
x5            -0.0978      0.225     -0.435      0.675        -0.616     0.421
x6          1823.1829    445.829      4.089      0.003       795.100  2851.266
==============================================================================
Omnibus:                        1.960   Durbin-Watson:                   2.554
Prob(Omnibus):                  0.375   Jarque-Bera (JB):                1.423
Skew:                           0.713   Prob(JB):                        0.491
Kurtosis:                       2.508   Cond. No.                     4.80e+09
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 4.8e+09. This might indicate that there are
strong multicollinearity or other numerical problems.

/home/skipper/.virtualenvs/statsmodels/local/lib/python2.7/site-packages/scipy/stats/stats.py:1205: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=15
  int(n))

Comparing gls and glsar results, we see that there are some small differences in the parameter estimates and the resulting standard errors of the parameter estimate. This might be do to the numerical differences in the algorithm, e.g. the treatment of initial conditions, because of the small number of observations in the longley dataset.

In [10]:

<span class="k">print</span><span class="p">(</span><span class="n">gls_results</span><span class="o">.</span><span class="n">params</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">glsar_results</span><span class="o">.</span><span class="n">params</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">gls_results</span><span class="o">.</span><span class="n">bse</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">glsar_results</span><span class="o">.</span><span class="n">bse</span><span class="p">)</span>

[-3797854.9015      -12.7656       -0.038        -2.1869       -1.1518
       -0.0681     1993.9529]
[-3467960.6325       34.5568       -0.0343       -1.9621       -1.002
       -0.0978     1823.1829]
[ 670688.6993      69.4308       0.0262       0.3824       0.1653
       0.1764     342.6346]
[ 871584.0517      84.7337       0.0328       0.4805       0.2114
       0.2248     445.8287]

Links:

http://statsmodels.sourceforge.net/stable/examples/notebooks/generated/gls.html

doc_statsmodels

2025-01-10 15:47:30

Comments