Facebook Expansion: Invisible Impacts?
Why is the meteoric rise of social media invisible in 2003-2008 depression trends? And why did authors of the Facebook expansion study completely omit all the relevant mental health trends?
This article is the start of a series dealing with the Facebook expansion study:
In November 2022, a study with the rather grandiose title Social Media and Mental Health was finally published after a long period of anticipation.
There were good reasons for excitement.
First, the study promised to reveal overall population impacts that would include environmental effects (see Notes).
Second, tech corporations are infamous for withholding data needed for independent research, and when they do make some data available, it seems only to ‘friendly’ researcher whose study designs can be expected to produce appropriately ‘friendly’ results. The authors of this study, however, managed to proceed with their analysis without dependence on the Meta corporation.
​
Design
The Social Media and Mental Health (SMMH) study took advantage of the staggered expansion of Facebook from one site (Hardvard) to colleges across the U.S. in four stages, starting with the most elite institutions in spring 2004 and ending with open access to all colleges in fall 2005. The authors separated colleges into four groups according to expansion timing (academic semesters spring 04 to fall 05) and then linked these with results of the National College Health Assessment (NCHA), which is administered bi-annually (spring/fall).
​
Results
The authors found that the rollout of Facebook at a college had a negative impact on student mental health and they estimated this impact as a 2 percentage point increase in the share of students suffering from depression (over a baseline of 25 percent) and as a 3 percentage points increase of severe depression.
The authors stress that these are conservative estimates of immediate effects, the true impacts likely being considerably higher because their analysis indicates that negative effects of Facebook on mental health become stronger with longer exposure to the platform.
These sound like impressive results and were widely cited, even by The U.S. Surgeon General’s Advisory titled Social Media and Youth Mental Health:
​
Mental Health Trends Context
Many readers no doubt wonder how the impact of Facebook expansion fits into the overall impact of social media between 2003 and 2008. After all, Facebook came out only after the spread of Friendster and the immensely more popular (at the time) MySpace while also competing with similar upstarts like Bebo and Hi5.
Furthermore, ‘thefacebook’ (as the site was called during its expansion) was quite primitive and lacked design distinctions that would make it plausibly more harmful than its competitors (features such as the Like Button and News Feed were implemented later).
If the mere rollout of ‘thefacebook’ has had the effect of increasing depression rates by nearly 10%, no doubt the meteoric rise of various other social media platforms must have contributed as well, and the cumulative effect would have had to result in far higher rates of depression circa 2007 than what they were for college students circa 2003.
To understand the impact of Facebook in context of overall social media impacts, we need to examine college mental health trends between 2002 and 2008.
Fortunately, the authors of the Facebook expansion study analyzed precisely the data needed for this, namely the 2000-2008 National College Health Assessment (NCHA); and so we just have to look at the various tables and figures in their paper that describe the relevant mental health trends.
​​
Relevant Mental Health Trends Info
​​
There is none.
​​
Omission of Vital Trends Info
The lack of any data or information on relevant mental health trends within the paper, its online Appendix, and even its Replication Package, is astonishing.
Among the countless figures and tables there is not a single one to describe any mental health developments before 2008.
​
Depression Trend
It turns out that that getting raw NCHA data for 2000-08 is quite difficult — one must send a research proposal to the American College Health Association and await their decision. One may also manually copy numbers out of the 17 separate ACHA-NCHA Reference Group reports 2000-2008, an arduous task and one that does not allow much for demographic controls nor for changes in college participation.
The manual route produces the following chart of prevalence rates for depression:
In the graph the red years (2004-05) correspond to Facebook expansion and also other social media proliferation — yet as we can see, there is no sign of any sustained rise in depression.
In fact prevalence in 2006 and 2007 is smaller than in 2002 and 2003. Furthermore, the overall trend before 2004 was slightly upwards but all this reversed downwards after 2005. That hardly comports with the notion that we should see more depression the further we look after Facebook expansion.
​
Minor Depressive Episodes
Perhaps we can see the impact of social media explosion if we look at minor depressive episodes (mDE), as indicated by the NCHA item asking if a student Felt so depressed that it was difficult to function at some point during the year.
The NCHA prevalence rates for such depressive episodes are far higher than for depression, in fact close to 50% at times, and so should be more immune to noise and more robust in trends.
So let us look at the trend for depressive episodes:
Once again there is no detectable effect of the social media explosion, with the rates in 2006 and 2007 nearly identical to those in 2003 and 2002.
Also: since there is no downward trend 2000-2003, it’s hard to argue that the Facebook expansion prevented decreases of depression that would have occurred otherwise.
Simply put, impacts of the Facebook expansion seem invisible in college depression trends.
​
A Convoluted Measure of Impact
The enigma of the complete omission of any relevant trends by the authors has a companion mystery: the convoluted method of calculating the impacts that were later quoted in The U.S. Surgeon General’s Advisory.
One would expect that to calculate the impact on depression and anxiety, the authors would use the actual measures of depression and anxiety on NCHA.
Instead, the authors undertook their own survey in 2022 in order to predict PHQ-9 and GAD-7 scores based on NCHA mental health results and then used this prediction scheme to estimate Facebook impacts on depression and anxiety circa 2004-08.
Needless to say, this method introduces uncertainties and noise not present in impact estimates based on direct depression and anxiety items offered by NCHA.
​​
Discussion
The complete omission of any relevant mental health trends from the Facebook expansion study is inexplicable to me. The issue is not only why the authors omitted this data, but why the editors of the American Economic Review failed to demand the inclusion of such crucial data, without which the readers are deprived of any context.
Context matters.
First of all, absence of crucial context can minimize our understanding of the phenomena the authors are trying to explain.
Second, absence of context can facilitate and amplify the spread of misinformation engendered by misleading or ambiguous assertions made by the authors.
Third, absence of context reduces our ability to evaluate the plausibility of crucial assumptions made by authors in their calculations of effects and impacts.
We will see in follow-up articles that the Facebook expansion paper suffers from serious flaws that illustrate the three concerns listed above.
​
Conclusion
Depression trends as measured by the National College Health Assessment do not indicate any sustained increases after the social media explosion circa 2004. The authors of the Facebook expansion study inexplicably omit any and all relevant trends data.
​
Notes:
Post-2008 Trends:
Curiously, the authors did take the time to create three graphs for mental health trends after 2008 (see Figure A.1 in the online Appendix) — it is for trends before 2008 that the authors fail to offer a single table or figure. We will deal with Figure A.1 in a follow-up article.
Other NCHA indicators
It may be that NCHA indicators for factors other than depression, such as anxiety or suicidal ideation, did increase. If so it should be a matter of minutes for the authors to create figures from their NCHA data to demonstrate such trends.
Indeed I think it is incumbent upon the authors to publish at least the following data:
a) Separate male and female trends 2000-2008 for crucial NCHA items such as depression, anxiety, and suicidal attempts.
b) Averages of 2002-03 vs. 2006-07 for crucial NCHA measures restricted to colleges that participated in all 8 of the 2002-03 & 2006-07 NCHA surveys (assuming there is a sufficient number of such colleges).
Publication of this data by the authors would no doubt greatly improve the transparency of their paper.
Environmental Effects
An example of environmental effects is second-hand smoke and the impact of violence on witnesses. One environmental effect of screen time may be to reduce the chances of in-person interactions with peers even for teens with little screen time use. Another environmental effect may be the fear of public shaming on social media — it should increase for everyone with increases of social media popularity.
Full citation:
Braghieri, Luca, Ro'ee Levy, and Alexey Makarin. 2022. "Social Media and Mental Health." American Economic Review, 112 (11): 3660-93. DOI: 10.1257/aer.20211218
Notification
This is a classic example of what I like to call "The Incredible Shrinking Effect Size". And it is precisely why honest skeptics are skeptical of any effect sizes that are rather small to begin with, as they are likely to shrink even further upon closer examination to become practically indistinguishable from noise.