{"schemaVersion":2,"courseId":"panel","module":{"id":"panel-heterogeneity","title":"1 · Panel data & unobserved heterogeneity","summary":"What panel data is, why repeated observations on the same entities help, and how an unobserved entity effect αᵢ creates correlation that biases naive regression.","stages":[{"type":"read","title":"What makes data 'panel' data","html":"<h2>Panel data: many entities, observed over time</h2><p>A <strong>panel</strong> (or longitudinal) dataset follows the <em>same</em> units &mdash; people, firms, countries, schools &mdash; across <em>multiple</em> time periods. We index every observation by an <strong>entity</strong> i and a <strong>time</strong> t:</p><p class=\"katexish\">yᵢₜ = β₀ + β₁·xᵢₜ + αᵢ + εᵢₜ</p><p>Here <code>yᵢₜ</code> is the outcome for entity i at time t, and <code>xᵢₜ</code> is a regressor that can vary across both i and t. A <em>balanced</em> panel has every entity observed in every period: N entities &times; T periods = N·T rows.</p><h3>The new ingredient: αᵢ</h3><p>The term <strong>αᵢ</strong> is an <em>entity effect</em> &mdash; a quantity that is constant within an entity but differs across entities. Think of a firm's management quality, a person's innate ability, or a country's institutions. It is <strong>unobserved</strong>: we don't have a column for it, yet it shapes y.</p><ul><li><code>εᵢₜ</code> is ordinary idiosyncratic noise (varies over i and t).</li><li><code>αᵢ</code> is fixed for the entity &mdash; it is the same in every period t for that i.</li></ul><div class=\"callout\"><b>Key idea.</b> Panel data's superpower: because we see each entity repeatedly, we can isolate and remove the time-invariant αᵢ even though we never measure it directly. Cross-sectional (one-shot) data cannot do this.</div>","id":"panel-heterogeneity-01","skillIds":["panel.panel-structure"],"estimatedMinutes":6,"difficulty":"advanced","prerequisiteStageIds":[]},{"type":"read","title":"Why αᵢ is dangerous: unobserved heterogeneity","html":"<h2>Unobserved heterogeneity and omitted-variable bias</h2><p>Regression gives unbiased slopes only when the regressor is uncorrelated with everything left in the error term. In the panel model, the <em>composite error</em> is <code>αᵢ + εᵢₜ</code>. The problem appears when the unobserved αᵢ is <strong>correlated with xᵢₜ</strong>:</p><p class=\"katexish\">Cov(xᵢₜ , αᵢ) ≠ 0</p><p>Then αᵢ is just an <em>omitted variable</em> that happens to be bundled with x. A regression of y on x alone will mistakenly credit x with variation that is really due to αᵢ.</p><h3>A concrete story</h3><p>Suppose y = wages, x = years of schooling, and αᵢ = unobserved ability. High-ability people both earn more (αᵢ pushes y up) <em>and</em> stay in school longer (αᵢ correlates with x). A naive wage-on-schooling regression blends the true return to schooling with the ability premium, <strong>overstating</strong> β₁.</p><ul><li>If <code>Cov(x, α) &gt; 0</code>, the naive slope is biased <strong>upward</strong>.</li><li>If <code>Cov(x, α) &lt; 0</code>, it is biased <strong>downward</strong>.</li><li>If <code>Cov(x, α) = 0</code>, there is no bias from α (though it still inflates the variance of the error).</li></ul><div class=\"callout\"><b>Key idea.</b> The whole fixed-vs-random-effects debate is a debate about one quantity: is αᵢ correlated with xᵢₜ? If yes, you need fixed effects.</div>","id":"panel-heterogeneity-02","skillIds":["panel.pooled-bias"],"estimatedMinutes":6,"difficulty":"advanced","prerequisiteStageIds":["panel-heterogeneity-01"]},{"type":"code","title":"Build a balanced panel and look at it","note":"We simulate N=40 entities over T=6 periods. The entity effect alpha is correlated with x (corr≈0.8), so x carries some of alpha's fingerprint. True slope = 2.","code":"import numpy as np, pandas as pd, matplotlib.pyplot as plt\nrng = np.random.default_rng(0)\nN, T = 40, 6\nentity = np.repeat(np.arange(N), T)\ntime   = np.tile(np.arange(T), N)\nalpha  = rng.normal(0, 2, N)            # one entity effect per entity\n# x is correlated with the entity's alpha:\nx = 0.8*alpha[entity] + rng.normal(0, 1, N*T)\ny = 1 + 2.0*x + alpha[entity] + rng.normal(0, 1, N*T)   # true beta1 = 2\ndf = pd.DataFrame({'entity': entity, 'time': time, 'x': x, 'y': y})\nprint('panel shape (rows, cols):', df.shape, ' = N*T =', N*T)\nprint(df.head(8).to_string(index=False))\nprint('\\nCorr(x, alpha_i) =', round(np.corrcoef(x, alpha[entity])[0,1], 3))\n# colour each entity's cloud so the vertical shifts (the alpha_i) are visible\nplt.figure(figsize=(6,3.7))\nfor i in range(0, N, 8):\n    m = df['entity']==i\n    plt.scatter(df.x[m], df.y[m], s=18, label=f'entity {i}')\nplt.xlabel('x'); plt.ylabel('y'); plt.title('each entity = a shifted cloud (its own alpha_i)')\nplt.legend(fontsize=7)","id":"panel-heterogeneity-03","skillIds":["panel.panel-structure"],"estimatedMinutes":10,"difficulty":"advanced","prerequisiteStageIds":["panel-heterogeneity-02"]},{"type":"quiz","title":"Check: what is αᵢ?","prompt":"In yᵢₜ = β₀ + β₁·xᵢₜ + αᵢ + εᵢₜ, what distinguishes αᵢ from εᵢₜ?","choices":["αᵢ varies over time within an entity; εᵢₜ is constant","αᵢ is constant within an entity but differs across entities; εᵢₜ varies over both i and t","αᵢ is observed in the data; εᵢₜ is unobserved","They are identical; the notation is redundant"],"answer":1,"hint":"The 'i' subscript with no 't' is the giveaway.","explain":"αᵢ carries only an i subscript, so it is fixed across all periods of entity i but differs from entity to entity. εᵢₜ carries both subscripts and is the usual time-varying noise. Both are unobserved.","id":"panel-heterogeneity-04","skillIds":["panel.pooled-bias"],"estimatedMinutes":5,"difficulty":"advanced","prerequisiteStageIds":["panel-heterogeneity-03"],"variantId":"panel-heterogeneity-04-core"},{"type":"quiz","title":"Check: when does αᵢ cause bias?","prompt":"A naive regression of y on x (ignoring αᵢ) gives a biased β₁ precisely when:","choices":["αᵢ has large variance","T is small","Cov(xᵢₜ, αᵢ) ≠ 0","εᵢₜ is heteroskedastic"],"answer":2,"hint":"Bias comes from correlation between the regressor and the omitted term.","explain":"Bias arises from omitted-variable correlation: if x is correlated with the omitted αᵢ, the slope absorbs αᵢ's effect. A large but uncorrelated αᵢ only inflates variance, it does not bias the slope.","id":"panel-heterogeneity-05","skillIds":["panel.panel-structure"],"estimatedMinutes":5,"difficulty":"advanced","prerequisiteStageIds":["panel-heterogeneity-04"],"variantId":"panel-heterogeneity-05-core"},{"type":"fillblank","title":"Name the hidden entity effect","prompt":"The time-invariant, entity-specific term αᵢ that you cannot observe — a firm's culture, a person's innate ability — is called the unobserved ___ effect, and it is what biases naive regression when it moves with x.","lead":"You just saw each entity sit at its own height αᵢ, a fixed trait carried in every one of its observations.","accept":["heterogeneity","unobserved heterogeneity","individual heterogeneity","entity heterogeneity"],"hint":"It is the word for entities differing in fixed, unmeasured ways — not 'error', not 'noise'.","explain":"The αᵢ-is-just-noise trap. αᵢ is unobserved <b>heterogeneity</b>: a fixed entity trait, not random scatter. The tempting belief that it is ordinary error fails because error averages to zero and is uncorrelated with x, whereas αᵢ is a persistent level that can correlate with x — that correlation, not its mere presence, is what biases a naive regression. Boundary: if Cov(x, αᵢ)=0 the heterogeneity is harmless to consistency (RE territory); only correlated heterogeneity demands the within transform.","points":15,"id":"panel-heterogeneity-06","skillIds":["panel.pooled-bias"],"estimatedMinutes":4,"difficulty":"advanced","prerequisiteStageIds":["panel-heterogeneity-05"],"variantId":"panel-heterogeneity-06-core"}]}}