Context and methods of the study
In accordance with SAMHSA funding requirements, participant-level data is collected using the Government Performance and Results Act (GPRA) tool for participants attending TTA services provided by TTCs. [19]. These instruments are approved by the U.S. Office of Management and Budget and help determine the scope, consistency, and quality of the TTC program and do not collect personally identifiable information. The same GPRA instruments are used on all three TTC networks for any type of TTA event.
This study used data collected from two GPRA forms: 1) Event Description Form (DE), an event-level form; and 2) Post-event form, a participant-level form. The Event description form is completed by the sponsoring TTC and includes details of the event, including the total number of attendees in attendance. The Post-event form is collected from attendees within 7 days of the TTA service and gauges information about attendee demographics and satisfaction with the event. Participation in TTA events and production of the Post-event form is completely voluntary. Not all participants complete the Post-event formtherefore, the number differs from the total number of attendees present.
Permission was sought from all 39 TTCs to include their GPRA data in this analysis, of which 38 (97%) agreed to be included. All participant-level data is anonymized, and secondary use of such anonymized data is considered “exempt” or “non-human subject” by the corresponding institutional review boards of the individual authors. GPRA data for TTCs was downloaded and aggregated for two periods: 1) the six-month period before the implementation of the COVID-19 restrictions, from September 1, 2019 to February 28, 2020 (“pre-COVID”); and 2) the six-month period following the implementation of the COVID-19 restrictions, from April 1, 2020 to September 30, 2020 (“during COVID-19”). For the two periods combined, the data collected by the Event description form contained information for a total of 2,257 events and 175,766 participants. The data collected through the Post-event form included a total of 85,528 (49% response rate) responses from unique participants in all 50 U.S. states, the District of Columbia, five U.S. territories (American Samoa, Commonwealth of the Northern Mariana Islands, Guam, Puerto Rico, and the United States Virgin Islands), and three freely associated states (the Republic of the Marshall Islands, the Republic of Palau and the Federated States of Micronesia). See Table 1 for the breakdown of event and participant data collected for the pre-COVID and during COVID-19 periods.
Data Variables
The data extracted from the Event description form included the date the event was held, the duration of the event, the number of attendees attending, and the number of continuing education hours given to attendees for each event. The data extracted from the Post-event form included participant-level variables such as demographics (e.g., race, gender, level of education), occupational discipline, workplace, and workplace postal code. Moreover, the Post-event form included four items to measure attendee satisfaction with TTA events. These questions included an item on overall satisfaction (i.e., “How satisfied were you with the overall quality of this event?”) rated on a 5-point Likert-type scale (Very unsatisfied at Very satisfied)and two items asking participants to indicate their level of agreement on a 5-point Likert-type scale (Totally agree at strongly disagree) related to how the TTC event will help them in their profession (i.e., “I expect this event to benefit my professional development and/or my practice” and “I I will use the information gained from this event to change my current practice”). The fourth element indicated (yes/Nope) whether participants would recommend the training to a colleague.
statistical analyzes
GPRA data was downloaded into Microsoft CSV files and then exported into SPSS version 25 for data analyses. Analyzes included descriptive statistics to compare trends over the two time periods (i.e. before and during COVID-19 restrictions). This included cross-tabulations for nominal and ordinal level data and comparison of means for interval level data. Tests were performed to determine the statistical significance of observed changes, chi-square for cross-tabulations, and t-tests for comparisons of means between pre-COVID and during-COVID means.
Due to the very large sample size, we calculated effect sizes to ensure that we were not only considering statistical significance (which is very likely in a large sample), but also the effect size. Cohen D was calculated to determine the effect sizes of the mean comparisons using the pooled variance estimates, with values interpreted following Cohen’s guidelines such that 0.20 indicated a small effect, 0.50 indicated a moderate effect, and 0.80 indicated a large effect [20]. Phi (used for 2×2 contingency tables) and Cramer’s V coefficients were also calculated as an adjustment to the chi-square significance to account for large sample sizes [21]. Cramer’s V values were interpreted using the following ranges: 0–0.05 = no or very weak association; > 0.05 = weak association; > 0.10 = moderate association; > 0.15 = strong association [21].
Additionally, ArcMap 10.8.1 software was used to show the scope of trainings before and during COVID-19 by mapping zip codes in US states, freely associated states, and territories where TTC training participants were. located in three mutually exclusive groups: 1) postal codes where participants received pre-COVID TTA only; 2) postal codes where participants received the TTA before and during COVID; and 3) postal codes where participants received TTA during COVID only. Also, using advice from Hailu and Wasserman [22]participants’ postcodes were categorized as urban, suburban, or rural using the rural-urban commuting area code classification system [23]. Rural-urban commuting area codes classify U.S. ZIP codes into metropolitan, micropolitan, small town, and rural areas using U.S. Census data on population density, urbanization, and commuting patterns [23]. Descriptive analyzes and matched samples you tests were used to describe changes in the number of participants in urban, suburban and rural postal codes.