Difficulties with Differences (Facebook Expansion Study)
There are stark differences between the elite and selective universities in the earlier phases of Facebook expansion and the colleges that followed later.
One major difficulty with quasi-experimental studies is that conclusions depend on outcome differences between groups that are often far from random and therefore differ from each other greatly from the start. This is true of the Facebook expansion groups examined in Social Media and Mental Health.
Expansion Groups
At the heart of the Social Media and Mental Health study is the classification of U.S. colleges (and universities) into four groups according to the semester during which Facebook became accessible to its students. The four groups range from Spring 04 to Fall 05 expansions.
The authors compiled a list of 775 colleges in the first three expansion groups (Spring 04, Fall 04, Spring 05) — these are listed in Table A.32 of the Appendix.
The last group (Fall 05) consists of all the colleges not in the first three expansion groups; the authors justify this in footnote 15:
For the set of colleges that appear both in our introduction dates dataset and the NCHA survey, the ACHA listed the semesters corresponding to the introduction dates in our dataset. For the set of colleges that appear only in the NCHA dataset, we list the fall of 2005 as the semester in which Facebook was introduced at those colleges. Such imputation is sensible in virtue of the fact that our introduction dates dataset ends after the spring semester of 2005 and that, by the end of 2005, the vast majority of US colleges had been granted access to Facebook.
Differences Among Groups
The authors admit that there are considerable differences between the expansion groups:
Online Appendix Table A.1 shows that colleges in earlier Facebook expansion groups are more selective in terms of test scores, larger, more likely to be on the East Coast, and have more residential undergraduate programs than colleges in later Facebook expansion groups.
Let’s look therefore at Table A.1 ( Summary Statistics by Facebook Expansion Group):
Note: the statistics in Table A.1 are derived from the Integrated Postsecondary Education Data System (IPEDS) — not from the NCHA data set. This is so because the NCHA data set contains no information that could be used to identify any particular college (ACHA maintains privacy of institutions participating in NCHA surveys).
Warning: see Notes below regarding a serious problem with Table A.1 — a problem that does not, however, affect much the issues discussed in this article.
We see especially stark differences between all the expansion groups in test scores and the size of student body.
NCHA Differences Among Groups
Did the expansion groups have similar mental health outcomes before FB expansion?
The answer is no, as we can see in Panel B of Table A.2:
Warning: see Notes below regarding a problem with Table A.2 — a problem that does not, however, affect much the issues discussed in this article.
The differences in pre-2004 mental health across Facebook expansion groups are especially stark when comparing the first Facebook expansion group to the other groups. Indeed in S.D. units the difference is similar to the effect of Facebook expansion in the model preferred by the authors (~0.08).
Why It Matters
Differences between treatment groups complicate determination of effects. If students in elite and competitive schools, which dominate the earlier expansion groups, have a higher prevalence of mental health problems, then this needs to be taken into account.
Difference in differences
One approach is to use a technique called Difference in differences. Imagine we have only two groups, a ‘treatment’ group of elite and competitive colleges with Facebook and a control group of ordinary colleges without Facebook. We may then simply look at the difference in outcome while taking into account the difference before the outcome:
Picture taken from Wikipedia page Difference in differences (Sep 30 2023).
In the picture above, the presumed effect of treatment (Facebook) would be the distance between P2 and Q, where Q is the presumed outcome if there was no treatment.
The ‘difference in differences’ approach, however, requires a strong “parallel trends” assumption, namely that mental health differences between both groups would have continued to remain constant without treatment (Facebook).
Parallel Trends
The parallel trends assumption need not be (even approximately) true: imagine that selective universities (and only those) were raising their academic demands at the time and this was gradually worsening the mental health of their students due to stress; or imagine that the fees of selective schools were escalating rapidly at the time, thus placing greater and greater financial stress on students and their families; or simply imagine that, for some reason unconnected directly with selective universities, the type of students entering these institutions had a declining mental health trend at the time.
Any of the above would mean that students in selective universities could be on a substantially different mental health trajectory compared to students in all the other post-secondary institutions in the U.S.
Heterogeneous Effects
Another concern when treatment groups are non-random is that the results may be valid only for the the population in the treatment group and not for the overall population.
Imagine that Facebook harmed most the students who were already mentally troubled to a degree, while having little effect on others. Since students in the earliest expansion group (Spring 2004) have much worse mental health, estimates of ‘average treatment effect’ may be overly influenced by such students and thus give an effect size much larger than what was true for the entire student body in U.S. post-secondary institutions.
Another possibility is that Facebook was harmful only in the competitive environments of the colleges that dominate the first three expansion groups — in which case once again the effects found within models would apply only to a portion of the overall student population.
Treatment Group (Elite Colleges)
At the core of Difference in differences models are comparisons between students already treated and those not yet treated.
Since in Social Media and Mental Health the ‘treatment’ is attending a college with Facebook access, all such comparisons involve ‘treated’ students in the first two groups.
Colleges in Group 4 gained access the last (Fall 05), so their students can be only in the control group. Group 3 colleges gained access in Spring 05, but it is unknown which students were already exposed to Facebook when they answered the Spring 05 NCHA survey. There is therefore no reliable way to compare ‘treated’ students in Group 3 to ‘untreated students’ in Group 4.
That means only students in Group 1 and Group 2 are eligible for comparisons with untreated students. Groups 1 and 2, however, are dominated by elite schools.
Conclusion
The severe differences between expansion groups mean that determining (and generalizing) the effects of the Facebook expansion is very difficult and may depend on dubious assumptions.
Notes
Errors in Table A.1:
The Number of colleges in column 4 of Table A.1 is 204 even though it should be a far larger number, as it must include all the colleges not among the 775 in the first three expansion groups — and there are thousands of post-secondary institutions in the U.S. (over 7 thousands listed in the IPEDS database). As we will see later, it turns out that columns 3 and 4 correspond to early and late Spring 05 expansions, not Spring and Fall 05 expansions. This means that differences between expansion groups are even starker than implied by the (incorrect) Table A.1.
Error in Table A.2: