Questions of Sampling (Facebook Expansion Study)
The Social Media and Mental Health study is missing important information regarding sampling in the National College Health Assessment surveys.
The conclusions of the Social Media and Mental Health study depend on the results of National College Health Assessment (NCHA) surveys that, in turn, depend on the participation of colleges and students. The issue of sampling is therefore crucial.
Missing College Info
The authors inform us that The NCHA dataset includes the universe of responses to all NCHA survey waves administered between the spring of 2000 and the spring of 2008 and further admit:
The NCHA dataset is an unbalanced panel, in which colleges drop in and out. Specifically, every college in the United States can voluntarily select into any wave of the NCHA survey and is not required to keep administering the survey in subsequent years.
Unfortunately the authors do not include any information on college participation beyond the number of colleges with NCHA data within each expansion group: 40+124+120+136 = 420 (see the last row of Table A.1).
The authors do not specify how many colleges participated in only one survey during the period, nor do they specify how many colleges participated in both a pre-FB survey and a post-FB survey.
The authors also do not reveal the participation history, so in theory one of the NCHA surveys could have consisted of a single campus.
Missing Student Info
Regarding student participation, the authors note that The average response rate across the survey waves for which we have such information is 37 percent (ACHA 2000–2019) but do not include any information on trends in the overall response rates.
The authors further note that Colleges included in the NCHA dataset administered the survey to randomly selected classrooms, randomly selected students, or all students but do not tell us anything else about this, raising the possibility that some of the surveys were dominated by a few colleges that surveyed all their students.
Survey Participation History
We can locate some of the missing participation information at the American College Health Association (ACHA) site:
As we can see, the participation varied wildly, from 8 colleges in Fall 2001 to 117 Spring 2007.
Since some colleges were counted more than once in the 832 total, and the authors report 420 colleges in their total, it seems likely that there were many colleges with only a single survey. It would be helpful if the authors revealed how many colleges had both pre-FB and post-FB survey data.
In view of the above we may have a series of cross-sectional data rather than an ‘unbalanced panel’ in relation to not only students, but also colleges.
Participation Trends
Participation in Fall was very low before 2004 and fluctuated considerably:
Participation in Spring was higher and increased greatly in 2004 and 2006:
We see that post-FB years (2006-08) have triple the participation of pre-FB years (2000-03). Question is, did these rapid increases in participation affect how representative was the sample, and did this in turn affect different expansion groups differently?
Multiple Surveys
There is also information in the table on repeated participation in NCHA surveys, albeit only for the first two decades combined:
If this pattern was similar during the first decade, then out of the 420 participating colleges in 2000-08 nearly 300 participated twice, although a substantially smaller number (perhaps around 100) would have participated both before and after Facebook expansion (since pre-2004 participation was very low).
Why it Matters
Readers may wonder why worry about college participation when there are half-a-million student surveys in the data set. The problem is that models in Social Media and Mental Health depend heavily on the timing of Facebook expansion that in turn depends on participating colleges.
There are at least two concerns.
The first is that rapid changes in participation may affect different expansion groups differently, and those developments may then be misinterpreted as the effects of Facebook expansion since the timing is nearly the same.
The second is that some of the models that the authors use for additional conclusions, such as length-of-exposure effects, may be vulnerable to anomalies that are likelier when survey participation is very low, such as the 8 colleges in the 2001 Fall survey.
Conclusion
Low college participation in early NCHA surveys and the rapid increases in participation around the time of Facebook expansion raise concerns about the methodology in Social Media and Mental Health.