- Journal List
- J Int Assoc Provid AIDS Care
- v.20; Jan-Dec 2021
- PMC8274121
As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsem*nt of, or agreement with, the contents by NLM or the National Institutes of Health.
Learn more: PMC Disclaimer | PMC Copyright Notice
J Int Assoc Provid AIDS Care. 2021 Jan-Dec; 20: 23259582211030805.
Published online 2021 Jul 9. doi:10.1177/23259582211030805
PMCID: PMC8274121
PMID: 34242100
Tiffany Chenneville, PhD,1 Hunter Drake, MA,2 Kemesha Gabbidon, PhD,1 Carina Rodriguez, MD,2 and Lisa Hightow-Weidman, MD, MPH3
Author information Article notes Copyright and License information PMC Disclaimer
Abstract
Young men who have sex with men (YMSM) living with HIV experience challenges with retention in care, which negatively affects viral suppression. To address this, researchers piloted Bijou, a program designed to provide health education through electronically delivered behavior and risk reduction modules. Participants were 29 YMSM aged 19-24 living with HIV from the southeastern US. Participants completed pre, post, and 3-month follow-up (3MFU) surveys assessing knowledge, intervention acceptability, satisfaction, self-efficacy, ehealth literacy, and usability. Findings revealed significant improvement in knowledge and e-health literacy from pre-test to post-test but lost significance at 3MFU. Self-efficacy scores did not show significant differences from pre-test to post-test or 3MFU. Participants who completed all modules considered Bijou usable and acceptable; however, many did not complete the program. Findings suggest a need for adaptations to promote knowledge retention, e-health literacy, engagement over time, and research with a larger, more representative sample.
Keywords: HIV, MSM, mobile health, eHealth, youth
What Do We Already Know about This Topic?
There are a variety of mhealth tools available to support HIV linkage to, and retention in, care as well as antiretroviral therapy (ART) adherence among people living with HIV (PLWH); however, mhealth tools targeted for key populations, including young men who have sex with men (YMSM), is limited.
How Does Your Research Contribute to the Field?
This pilot study contributes to the field by presenting data on the potential successes and challenges of implementing mhealth strategies for people living with HIV, particularly for key populations such as YMSM.
What Are the Implications of Your Research toward Theory, Practice, or Policy?
Findings from this study point to the need for further research on mhealth strategies for key populations, such as YMSM living with HIV, in clinical settings to improve retention in care and disease management. This study also highlights the need for more focus on implementation challenges in health research.
The World Health Organization (WHO) has identified young men who have sex with men (YMSM) as a key youth population experiencing a heightened risk of HIV infection. YMSM living with HIV experience challenges with retention in care, which often adversely affects viral suppression efforts. Challenges experienced by youth vary and may be influenced by transmission route. For example, in previous research, youth with perinatally acquired HIV reported several individual (e.g., depression, anxiety) and environmental barriers (e.g., stigma, lack of social support) to retention in care.1-5 Another issue facing youth with perinatally or behaviorally acquired HIV is the transition from pediatric to adult care. Youth typically “age out” of pediatric care at age 24 or 25. Many times these youth are lost to follow-up, which not only affects HIV mortality and morbidity rates but also affects rates of onward HIV transmission. These same challenges affect retention in interventions designed to promote the uptake of HIV services.
Because HIV disproportionately affects Black/African American and Latino populations, YMSM often face intersectional stigma and are affected by compounded health disparities. A needs assessment conducted to inform case management interventions for 61 Black/African American and Latino YMSM aged 18-24 living with HIV in Los Angeles showed that 75% needed stable housing, drug rehabilitation, and/or mental health services.6 Racial and ethnic minority YMSM may experience unique challenges to retention in care. Harper and colleagues surveyed 200 Latino and African American YMSM living in Chicago.7 They found that participants who strongly identified with their ethnic identity yet had internalized negative attitudes toward their HIV status and sexual orientation were significantly more likely to miss their clinic appointments. There have been some interventions targeting ethnic minority YMSM, with the aim of addressing both their physical and psychosocial needs.6,8-10 One outreach program designed to improve retention in care and facilitated by the Los Angeles County Department of Public Health Safety targeted YMSM of color living with HIV.10 Over the first 2 years, only 11% of clients missed their clinic appointments, a significant improvement from previous years.10 Other programs such as Project nGage and Strength Through Youth Livin’ Empowered (STYLE) addressed social support networks as a strategy to facilitate engagement in care.8,10
E-health and Mobile Health Strategies
E-health is the practice of receiving health care services electronically or via the internet. A subcategory of e-health is mobile health. Mobile health (mhealth) is the practice of medical and public health programs via communication devices.11 Findings from a systematic review on the use of mobile technologies to improve chronic disease management and adherence showed mixed results in terms of effectiveness.12 However, several studies involving mhealth strategies with youth living with diabetes mellitus showed significant improvements in adherence and disease management.13,14 Similar studies have demonstrated the utility of mhealth strategies for other pediatric illnesses (e.g., asthma, cancer, obesity/weight control), including HIV.15-17 Catalani and colleagues conducted a systematic review on mobile strategies used to address HIV and found a number of mhealth tools available to support HIV linkage to, and retention in, care as well as antiretroviral therapy (ART) adherence among people living with HIV (PLWH), youth, and young adults; however, evidence was lacking for the use of mhealth tools for key populations including MSM.11 Further, most programs were implemented in Africa or North America, with the primary function of providing alerts and reminders (60%) via text messages to clients. In their systematic review, Lall et al. identified 2 mhealth interventions designed to improve adherence among youth with behaviorally acquired HIV.18 Findings from Muessig et al.’s systematic review of smartphone, internet, and Web 2.0 interventions to address the HIV care continuum revealed trends related to the use of social networking sites, in vivo evaluation and feedback, virtual reality, and gaming strategies.19 Belzer and colleagues conducted a randomized controlled trial, including youth with both perinatally and behaviorally acquired HIV wherein youth received daily medication reminders via telephone.20 Those in the intervention group reported significantly higher levels of adherence and lower viral loads than those in the control group. Other examples of mhealth strategies targeting YMSM include AllyQuest, a gamified smartphone app, and weCare, a social media intervention.21,22
Mhealth strategies may be a useful approach to improve disease management and retention in care among YMSM living with HIV. Therefore, Bijou was developed as a mhealth strategy through an iterative process in collaboration with YMSM.23 Adapted from Arthritis and Chronic Disease Self-Management Programs, Bijou was designed to be delivered in a wholly interactive way and tailored for YMSM.24,25 Bijou is one of the first behavioral interventions of its kind that addresses retention in care and viral suppression behaviors by offering a specially designed interactive electronic health and risk reduction curriculum for YMSM living with HIV. Bijou is delivered via computer, electronic tablet, and/or mobile phone and is designed to help retain YMSM in care and achieve and maintain virologic suppression.
Purpose of Study
The purpose of this study was to examine the impact of the Bijou program on health-related outcomes for YMSM living with HIV in a pilot study. Research questions included: (1) What is the impact of participation in the Bijou program on the following outcomes: HIV health literacy, general health and wellness, personal growth, illness management, relationships, problem-solving/coping, self-efficacy, and E-health literacy? and (2) What level of satisfaction do Bijou users have, and how acceptable do they find the Bijou program?
Methods
Participants and Procedures
This study used existing program evaluation data from Bijou, which was offered to all YMSM living with HIV aged 18-24 (n = 30) and receiving services at an adolescent infectious disease clinic serving predominantly Black/African American, non-Hispanic young men who acquired HIV behaviorally. YMSM were approached during routine clinic visits. Those who wished to enroll completed demographic items and a pre-test on an iPad in the clinic, after which they were provided a link to the Bijou website to complete modules within 6 weeks, but at their own pace and convenience from their personal computer, tablet, or mobile device. Modules ranged in length from 30 to 60 minutes. After completing all modules, YMSM were asked to complete a post-test and 3-month follow-up (3MFU) survey. Pre, post, and 3MFU surveys were administered via Qualtrics survey software and included items assessing Bijou module-related knowledge and intervention acceptability, the General Self-Efficacy Scale, the eHealth Literacy Scale, and a System Usability Scale. A link to post and 3MFU surveys was sent to the participant’s chosen communication method (e.g., text, email, social media messaging). As an incentive, participants were given a $10 gift card after completing the pre-test; another $10 gift card after completing the first 3 modules; a $20 gift card after completing the last 3 modules and the post-test; and another $20 gift card after completing the 3MFU survey. Incentives totaled $60 for participants who completed all modules and surveys.
Bijou Program
The Bijou program consists of 6 electronic self-directed learning modules paired with live or online support sessions with Bijou educators. The modules were designed utilizing game theory elements to be interactive with an emphasis on visual content. For example, the modules include visual cues (e.g., points, completion bars, gem avatar, etc.) for completion and an electronic certificate of completion. Interactive components include activities, journals, and quizzes designed to allow users to apply new knowledge to their own lives. The modules were intended to be broadly appealing to youth. The design of modules is consistent with the cognitive theory of multimedia learning, which suggests the human information processing system includes dual channels (visual and verbal), each with a limited processing capacity.26 The design is also consonant with dual coding theory, which suggests that combining images with text may increase learning and memory retention.27 Module domains and topics were: HIV Health Literacy, General Health and Wellness, Personal Growth, Managing Your Illness, Relationships, and Problem Solving.
Measures
The measures described below were used to assess factors related to disease management and retention in care, as well as usability, acceptability, and satisfaction with Bijou.
Demographic form
The demographic form contained items to collect information on age, gender, race/ethnicity, mode of transmission, and antiretroviral medication use.
Bijou module knowledge
The authors created a 20-question set of items to assess knowledge of the information contained in each of the 6 Bijou modules. For example, the assessment items for the HIV Health Literacy module directly assessed knowledge of the information covered in that module, and the assessment items for the General Health and Wellness module directly assessed knowledge of the information covered in that module. Items included true/false, multiple-choice, and Likert-type items. Combined, overall scores may range from 0 to 78. Cronbach’s alpha for our sample ranged from pre-test to 3MFU as follows: .72 to .99 for HIV Health Literacy; .33 to .99 for General Health and Wellness; .41 to .98 for Personal Growth; .34 to .99 for Managing Your HIV; .65 to .98 for Relationships; .and 62 to .96 for Problem Solving and Coping.
Self-efficacy for managing disease scale
Adapted from the Self-Efficacy for Managing Chronic Disease-6 Item Scale, this 5-item measure is designed to assess how confident the respondent is doing certain activities related to their illness.28,29 For example, one item reads, “How confident are you that you can judge when the changes in your illness mean you should visit a doctor?” Response choices are 1 = Not at all confident to 10 = Totally confident. Scores range from 5 to 50. Cronbach’s alpha for our sample ranged from .81 at pre-test to .97 at 3MFU.
eHealth literacy scale (eHEALS)
Created by Norman and Skinner, this 9-item measure assesses health literacy associated with using the internet for medical purposes.30 The first 8 items use a 5-point Likert scale ranging from 1 = Strongly Agree to 5 = Strongly Disagree. An example of the types of items included in this scale include is “I know how to use the Internet to answer my questions about health.” In addition, there is a question about how useful the respondent finds the internet to be in making decisions about their health. Again, a 5-point Likert scale is used with response choices ranging from 1 = Not useful at all to 5 = Very useful. All items include a “decline to answer” option. Scores ranged from 10 to 50. Cronbach’s alpha for our sample ranged from .73 at pre-test to .90 at 3MFU.
Bijou system usability scale (SUS)
The SUS contained 10 items designed to assess the usability of the Bijou program. Items included statements such as “I felt confident using Bijou” and “I thought Bijou was easy to use.” Response choices use a 5-point Likert scale where 1 = Strongly Disagree, and 5 = Strongly Agree. All items included a “decline to answer” option. See Table 1 for a complete listing of items—Cronbach’s alpha for our sample was .56.
Table 1.
System Usability, Intervention Acceptability, and Program Satisfaction.
| Instrument | Survey Item | n | m | sd |
|---|---|---|---|---|
| System Usabilitya | 11 | 4.13 | 0.54 | |
| I think that I would like to use Bijou frequently. | 4.36 | 0.81 | ||
| I found Bijou unnecessarily complex.* | 3.82 | 1.47 | ||
| I thought Bijou was easy to use. | 4.70 | 0.48 | ||
| I required the support of a technical person to complete Bijou.* | 3.82 | 1.47 | ||
| I found the various functions in Bijou to be well integrated. | 4.73 | 0.47 | ||
| I thought there was too much inconsistency in Bijou.* | 4.18 | 1.17 | ||
| I would imagine that most people would learn to use Bijou very quickly. | 4.18 | 1.17 | ||
| I found Bijou very cumbersome to use.* | 3.10 | 1.37 | ||
| I felt very confident using Bijou. | 4.70 | 0.48 | ||
| I needed to learn a lot of things before I could get going with Bijou.* | 3.73 | 1.49 | ||
| Intervention Acceptabilityb | 11 | 6.52 | 0.50 | |
| I trust the information in Bijou. | 6.55 | 0.52 | ||
| The information in Bijou is easy to understand. | 6.55 | 0.52 | ||
| The information in Bijou is accurate. | 6.64 | 0.51 | ||
| Bijou helps me to quickly find information and support for healthy living. | 6.55 | 0.52 | ||
| Bijou helps me make healthy choices for my life. | 6.45 | 0.69 | ||
| Bijou helps me deal with health challenges that might come up. | 6.55 | 0.69 | ||
| Bijou is useful to me. | 6.36 | 0.67 | ||
| Satisfactionc | 11 | 6.10 | 0.87 | |
| Overall, I am very satisfied with Bijou. | 6.45 | 0.52 | ||
| Using Bijou is very frustrating.* | 5.91 | 1.45 | ||
| I would recommend Bijou to my friends. | 6.09 | 1.22 | ||
| How likely would you be to continue using Bijou if it were available? | 10 | 5.90 | 1.66 | |
Open in a separate window
Note. *Reverse Coded.
a Range: 1-5.
b Range: 1-7.
c Range: 1-7.
Technology use questions-intervention acceptability
Adapted from Horvath and colleagues scale, this 7-item measure assessed participants’ views on the acceptability of Bijou.31 Items included statements like, “I trust the information in Bijou” and “The information in Bijou is easy to understand.” Participants selected their level of agreement using a 7-point Likert scale where 1 = Strongly Disagree and 7 = Strongly Agree. See Table 1 for a complete listing of items. Cronbach’s alpha for our sample was .94.
Satisfaction
Four quantitative items were used to measure participants’ overall satisfaction with Bijou. Sample items included were, “I would recommend Bijou to my friends” and “Overall, I am very satisfied with Bijou.” Participants selected their level of agreement using a 7-point Likert scale where 1 = Strongly Disagree and 7 = Strongly Agree. One item was reverse coded and recoded for analysis. One item assessing the likelihood of continuing the use of Bijou if it were available at the conclusion of the study was scored using a 7-point Likert scale where 1 = Very Unlikely and 7 = Very Likely. See Table 1 for a complete listing of items. Cronbach’s alpha for our sample was .65.
In addition to the quantitative items, open-ended items were used to assess what participants liked, disliked, found memorable, or found lacking in the Bijou program. For example, participants were asked, “Which two things did you like best about Bijou?” and “Are there any features that would make Bijou better?”. See Table 2 for a complete listing of items. Finally, participants were asked open-questions about some of the Bijou features. For example, participants were asked, “What motivated you to submit questions or responses in the Forum section?” and “What motivated you to write in your journal?”. See Table 3 for a complete listing of items.
Table 2.
Participant Responses to Open-Ended Items About Bijou Satisfaction.
| Survey item | Participant response |
|---|---|
| Which two things did you like best about Bijou? | |
| • The videos and the activities | |
| • I liked the section on Bijou where there was a video of a doctor talking about whether to disclose our status in a relationship and if we were to do so, how should we disclose it to our loved ones. | |
| • The information that was presented, and the ease of use. | |
| • That is was a secure website, and also taught me a lot about HIV that I didn’t know. | |
| • The information and review questions | |
| • The section where it talked about how important exercise is and eating healthy is a very important to live a long healthy lifestyle with HIV. | |
| • Easy to use. Very informative | |
| • Complexity. Information provided. Explanatory. | |
| What was most memorable about your experience with Bijou? | |
| • The experience | |
| • I found some videos on Bijou very useful and related to my life, like the videos that talked about how to disclose HIV status. | |
| • The scientific information is very digestible | |
| • The new information I’ve learned. | |
| • the interactive activities | |
| • colorful and knowledgeable (continued) | |
| Which two things did you like the least about Bijou? | |
| • Don’t have any | |
| • Sometimes it glitches weird | |
| • Nothing. | |
| • Needed a more simple back button to return to the module home screen. | |
| Are there any features that would make Bijou better? | |
| • It’s fine the way it is | |
| • I think Bijou is quite good but it would be better if there are more information on what supplements PLHIV should avoid and what is encouraged for PLHIV. | |
| • Needed a more simple back button to return to the module home screen. |
Open in a separate window
Note. Participant responses are direct quotes from post-intervention surveys.
Note. The responses are direct quotes from the participants as entered into the Qualtrics survey system.
Table 3.
Participant Responses to Open-Ended Items About Bijou Features.
| Survey item | Participant response |
|---|---|
| What motivated you to submit questions or responses in the Forum section? | |
| • Just wanted to give my feed back | |
| • I think sharing my personal experience in the Forum section can give others some useful ideas and they may be related to their real life situation. | |
| What motivated you to write in your journal? | |
| • Some questions are very related to my past experiences and writing in the journal can help me to face my fear and problems so I can overcome them. | |
| If you shared anything you learned on Bijou with anyone outside of the study, what information did you share? | |
| • Section 5a | |
| • The various ways to inhibit the virus | |
| • The healthy nutrition. | |
Open in a separate window
Note. The responses are direct quotes from the participants as entered into the Qualtrics survey system.
a Section 5 refers to the Bijou module on relationships.
Data Analyses
Descriptive statistics were used to analyze demographic variables and scores on all measures. Two-tailed paired-sample t-tests were used to compare pre-, post-, and 3-month follow-up test scores on module content in general and by topic, as well as eHealth literacy and self-efficacy. Individual participant data was excluded for measures with incomplete data. Thematic qualitative methods were used to evaluate the usability and acceptability of Bijou.
Ethical Approval and Informed Consent
The University of South Florida’s institutional review board reviewed this study (PRO0032708) and determined it exempt given its use of existing program evaluation data. Patients provided verbal informed consent to participate in the Bijou program.
Results
Participant Demographics
Of 30 YMSM approached, 29 agreed to participate in the Bijou pilot program. However, only 23 completed demographics and a pre-test, 11 completed the post-test, and 8 completed the 3MFU. Participants with complete data for all surveys (n = 8) ranged in age from 19 to 24 (M = 22.00, SD = 2.07). Most participants were Black/African American (50.00%), non-Hispanic (75.00%), and identified as hom*osexual (87.50%). In addition, all participants acquired HIV behaviorally, and most were taking antiretroviral medication (75.00%).
Bijou Module Knowledge
Of the 23 participants who completed the pre-test, 12 (52.17%) completed all the modules, 4 (17.40%) completed only one module, and 7 (30.43%) did not complete any modules. There were no significant differences between those who completed all modules (n = 12) and those who did not (n = 11) on any of the demographic variables. Although 8 participants submitted pre- and post-test surveys, the data from one participant who was an outlier was excluded due to excessive missing data (37%). Findings revealed significant improvement in module content knowledge from pre-test (n = 7, M = 62.43, SD = 8.06, range: 47-70) to post-test (n = 7, M = 67.71, SD = 7.52, range: 53-73), t(6) = 8.205, p < .001. There were significant improvements from pre-test to post-test in HIV health literacy (t(6) = 3.032, p < .012), general health and wellness (t(6) = 3.667, p < .005), personal growth (t(6) = 2.500, p = .023), and managing HIV (t(6) = 2.646, p = .019). Although scores remained higher than pre-test, differences lost significance at 3MFU. See Table 4.
Table 4.
Summary of Results.
| Descriptives | Score improvement | |||||
|---|---|---|---|---|---|---|
| n | Range | m | sd | t | p | |
| Bijou Module Content | 7 | 0-78 | ||||
| Pre-test | 47-70 | 62.43 | 8.06 | |||
| Post-test | 53-73 | 67.71 | 7.52 | 8.205 | < .001 | |
| HIV Health Literacy | 3.032 | .012 | ||||
| General Health and Wellness | 3.667 | < .005 | ||||
| Personal Growth | 2.500 | .023 | ||||
| Managing HIV | 2.646 | .019 | ||||
| eHealth Literacy | 8 | 10-50 | ||||
| Pre-test | 33-45 | 40.63 | 3.96 | |||
| Post-test | 38-50 | 45.00 | 4.69 | 2.768 | .014 | |
| 3-month Follow Up* | 37-50 | 43.25 | 4.89 | 1.649 | .071 | |
| Self-efficacy | 8 | 5-50 | ||||
| Pre-test | 33-50 | 45.38 | 5.37 | |||
| Post-test | 26-50 | 43.75 | 8.43 | −0.437 | .676 | |
| 3-month Follow Up* | 38-50 | 46.50 | 5.32 | 2.224 | .062 | |
Open in a separate window
Note: * 3-month follow up differences are compared to pre-test scores.
Self-Efficacy
Self-efficacy for disease management scores did not show significant differences from pre-test (n = 8, m = 45.38, SD = 5.370, range: 33-50) to post-test (n = 8, m = 43.75, SD = 8.430, range: 26-50) or 3MFU (n = 8, m = 46.50, SD = 5.318, range: 38-50). See Table 4.
E-Health Literacy
E-Health Literacy Scale scores showed significant improvement from pre-test (n = 8, m = 40.63, SD = 3.96, range: 33-45) to post-test (n = 8, m = 45.00, SD = 4.69, range: 38-50), t(7) = 2.768, p = .014. Improvements trended toward significance at 3MFU (n = 8, m = 43.25, SD = 4.89, range: 37-50), t(7) = 1.649, p = .071. See Table 4.
Bijou System Usability Scale (SUS)
The mean score across the 10 items (n = 11) on the Bijou SUS was 4.13 (SD = .54, range: 1-5), indicating a high degree of endorsem*nt of the website’s usability. See Table 1 for average responses to individual items.
Technology Use Questions-Intervention Acceptability
The mean score across the 7 items (n = 11) on the acceptability questionnaire was 6.52 (SD = .50, range: 1-7), indicating that participants rated Bijou as highly acceptable. See Table 1 for average responses to individual items.
Satisfaction
The mean score across the 4 Likert-scale items (n = 11) on the satisfaction scale was 6.10 (SD = .87, range: 1-7), indicating that participants were highly satisfied with Bijou. See Table 1 for average responses to individual items.
When asked what they liked best about Bijou, the most frequent responses concerned ease of use and video content. When asked what was most memorable about Bijou, comments focused on the interactive and colorful design of the intervention and the video content. When asked what they liked least about Bijou, 2 participants described technological issues. Specifically, one participant noted that “sometimes it glitches weird,” and another recommended a “more simple back button to return to the module home screen.” When asked about how Bijou might be improved, one participant made a suggestion about the interface design, and another recommended the inclusion of more information about nutritional supplements for PLWH. Participants also responded to open-ended items about their motivations to use Bijou’s journal and forum features, as well as what they may have shared with people not in the study about their experiences with Bijou. See Table 2 and Table 3 for a list of responses to individual open-ended items.
In addition to the comments above, informal and unsolicited verbal feedback provided to and documented by Bijou facilitators indicated that participants considered Bijou a usable and acceptable intervention, citing ease of use and privacy of completing the program by laptop or smartphone as desirable characteristics. Participants also provided positive feedback about the quality of the module content. Competing demands on time (e.g., work, school, social activities) and illness were cited as reasons for attrition. The program evaluation component (i.e., completing pre, post, and follow-up surveys) also was cited as a reason for attrition from the program. More specifically, participants found the evaluation measures to be tedious and time-consuming. In many cases, participants expressed an intent to complete but never did despite follow-up phone calls, text messages, emails, and/or social media messaging.
Acceptability and Feasibility
Facilitator notes provided additional information about feasibility and acceptability. Most YMSM who failed to complete all the modules did not provide a reason for attrition. Rather, they simply ignored facilitators’ attempts to follow up. However, among the 2 participants who told facilitators they did not plan to complete Bijou, one said they did not wish to mention the reason, and the other indicated the length of the pre-test deterred them from moving forward. Several others described being busy or reported concerns about the time needed to complete Bijou but said they planned to continue even though most did not. Facilitators had to provide login information to many participants multiple times and repeatedly provided links to surveys because participants had lost this information. Despite high attrition rates, some participants offered unsolicited comments about their appreciation for a program like Bijou.
Discussion
The primary objective for developing Bijou was to provide content crucial to YMSM living with HIV in a flexible format that may support retention in care, with the ultimate goal of developing an e-health intervention that can be disseminated broadly. Upon reviewing the program evaluation data, several aspects of program development may have attributed to the positive feedback received by participants who completed the program, including collaboration with youth in the design of the original program and the deliberate incorporation of valuable yet interesting content. Second, the incorporation of interactive components and gamification (i.e., gems) as outlined by Muessig et al. may have also contributed to the positive feedback provided by participants.19 Finally, youth reported confidentiality during program implementation as a positive feature, which is particularly noteworthy as concerns around confidentiality are often described as a barrier to enrollment and retention in e-health programming.19,32
eHealth platforms are uniquely positioned to provide updated content and features which could be used to sustain interest. This pilot data suggest participants were more comfortable using e-health programs and the internet to find health-related data after completing the program. Statistical significance waned after the 3MFU yet remained trending toward significance. Consistent with Muessig et al.’s systematic review, Bijou has incorporated key components such as a focus on e-health literacy and engaging YMSM in the program’s design.19 Overall, continued participatory approaches may be crucial in increasing program success.
The Bijou pilot data suggest that content-driven, self-paced e-health interventions can improve the understanding and retention of HIV-related health education content. Topics addressed by Bijou crucial to living well with HIV included: HIV-related stigma, disclosure planning, medication adherence, social support, self-esteem, HIV basics, safer sex habits, stress, anxiety and depression, financial management, exercise, and nutrition. Overall, the study results were both promising and concerning. The improvements from pre-test to post-test in module-specific content and e-health literacy were positive; however, although scores remained higher at the 3MFU, statistical significance had dwindled, indicating a need for keeping youth engaged with the program beyond simply completing the initial modules. For example, previous findings have suggested that incorporating social support within the e-health intervention may be beneficial for programming addressing sensitive content.33 Although social support from healthcare providers is built into Bijou, most participants did not make use of this feature. In addition to provider support, participant-to-participant interactions may foster both support and accountability, ultimately increasing program retention rates.
Despite its potential use and positive feedback obtained from participants who completed all modules, high attrition raises questions about the acceptability and feasibility of this e-health intervention. Of course, attrition concerns are not unique to this study. For example, Bennett et al. revealed that African American participants tended to experience higher attrition rates than other racial/ethnic groups for e-health interventions largely due to insecure internet access.34 This may have been an issue in our study given that the clinic where data was obtained serves predominantly Black/African American patients. However, because attempts to contact participants who failed to complete all modules were largely unsuccessful, it is difficult to state with confidence the reasons for high attrition in our sample.
Nonetheless, available data suggest the program evaluation component (i.e., pre, post, and follow-up surveys) may have been a deterrent to completion. In addition, some participants seemed to have difficulty keeping up with login details. The former suggests the need for succinct program evaluation strategies that are not overly burdensome for participants. The latter may suggest that mobile apps are better suited than web-based e-health interventions for this population. Indeed, Baumel & Kane found that real-world user engagement of self-guided e-health interventions is affected by product design.35 A mobile app would at least address the barrier created by participants not remembering the website URL.
Poor uptake of Bijou speaks to the challenges of retaining YMSM in care, generally, and in clinical intervention programs, specifically. Although there is a demonstrated need for e-health interventions for YMSM living with HIV, there are barriers to the retention of YMSM in HIV care and clinical programming.36 Based on findings from a qualitative study, Beach and colleagues described 5 themes, which map onto Bronfenbrenner’s ecological model, to explain the barriers and facilitators to retention in HIV care and clinical intervention programming: stigma, health system characteristics, community outreach, intersectionality, and geography.37 Further, they recommend multi-level approaches that take into consideration these barriers and facilitators to intervention planning with YMSM living with HIV.37 It may be useful to partner with community organizations to deliver e-health interventions, although technology capacity-building and staff training may be needed to support such organizations.38
Limitations and Future Directions
Given the small sample size, quantitative findings can in no way be generalized to the e-health uptake practices of YMSM. Instead, this data will inform further improvement in Bijou implementation and may be useful for similar studies. A larger, more representative sample would be needed to generalize findings to a broader population of YMSM. Also, as described above, there were high attrition rates, which adversely impacts the estimates and interpretations provided. To address this, strategies that complement rather than compete with the interests and responsibilities of YMSM living with HIV will need to be developed.32 James et al.32 suggest that researchers or, in this case, clinicians tailor recruitment, retention, and incentive strategies to the health condition and population of interest (e.g., YMSM living with HIV). To avoid competing demands, one strategy in clinical settings might be to offer patients the opportunity to engage in the ehealth intervention from an iPad or other mobile device while waiting to see healthcare providers. Finally, there was not an in-depth assessment of barriers experienced by participants. Qualitative approaches (e.g., focus groups) may provide a more layered contextual understanding of barriers experienced by YMSM, including structural (e.g., poverty), social (i.e., enacted stigma), and individual (e.g., intrinsic motivation) factors. Such approaches may help identify additional domains in which the Bijou program can be expanded before a larger trial is implemented. Therefore, considerations for the challenges experienced by participants must be carefully addressed prior to large-scale implementation.
Conclusion
Despite limitations, pilot data suggest promise for Bijou as a mhealth strategy for improving e-literacy among YMSM. These findings also demonstrate the need to revisit implementation strategies with particular attention to improving attrition rates among YMSM, which may include providing mhealth programs as complementary to participants’ interests. Further exploration into the use of interactive components, gamification, social support, and improved incentives may prove valuable. Findings speak to the need for ongoing education, given that initial gains appear to decrease with time. Future research is needed to test the efficacy of Bijou using a larger and more representative sample.
Acknowledgments
We would like to acknowledge Allison Nall, Alexandra Cario, and Marlayna Cromedy for their contributions to this project.
Authors’ Note: Data from this study was presented at the 2019 American Psychological Association’s Annual Convention in Chicago, IL, USA.
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by HRSA-15-022 Ryan White HIV/AIDS Program Part D Grants for Coordinated HIV Services and Access; and H12HA24835 Health Resources and Services Administration/Ryan White HIV/AIDS Program Part D-Women, Infants, Children, and Youth (WICY) Supplemental Funding, HIV Care Innovation.
ORCID iDs: Tiffany Chenneville 
https://orcid.org/0000-0001-5598-9387
Hunter Drake https://orcid.org/0000-0001-5223-080X
Kemesha Gabbidon https://orcid.org/0000-0001-9900-4308
References
1. Abramowitz S, Koenig LJ, Chandwani S, et al. Characterizing social support: global and specific social support experiences of HIV-infected youth. AIDS Patient Care STDS. 2009;23(5):323–330. [PubMed] [Google Scholar]
2. Giannattasio A, Officioso A, Continisio GI, et al. Psychosocial issues in children and adolescents with HIV infection evaluated with a World Health Organization age-specific descriptor system. J Dev Behav Pediatr. 2011;32(1):52–55. [PubMed] [Google Scholar]
3. Petersen I, Bhana A, Myeza N, et al. Psychosocial challenges and protective influences for socio-emotional coping of HIV+ adolescents in South Africa: a qualitative investigation. AIDS Care. 2010;22(8):970–978. [PMC free article] [PubMed] [Google Scholar]
4. Reisner MSL, Mimiaga MJ, Skeer MM, Perkovich MB, Johnson CV, Safren SA. A review of HIV antiretroviral adherence and intervention studies among HIV–infected youth. Top HIV Med. 2009;17(1):14. [PMC free article] [PubMed] [Google Scholar]
5. Williams PL, Storm D, Montepiedra G, et al. Predictors of adherence to antiretroviral medications in children and adolescents with HIV infection. Pediatrics. 2006;118(6):e1745–1757. [PubMed] [Google Scholar]
6. Wohl AR, Garland WH, Wu J, et al. A youth-focused case management intervention to engage and retain young gay men of color in HIV care. AIDS Care. 2011;23(8):988–997. [PubMed] [Google Scholar]
7. Harper GW, Fernandez IM, Bruce D, Hosek SG, Jacobs RJ; Adolescent Medicine Trials Network for HIVAI. The role of multiple identities in adherence to medical appointments among gay/bisexual male adolescents living with HIV. AIDS Behav. 2013;17(1):213–223. [PMC free article] [PubMed] [Google Scholar]
8. Bouris A, Voisin D, Pilloton M, et al. Project nGage: Network supported HIV care engagement for younger black men who have sex with men and transgender persons. J AIDS Clin Res. 2013;4:10.4172/2155–6113.1000236. [PMC free article] [PubMed] [Google Scholar]
9. Hightow-Weidman LB, Smith JC, Valera E, Matthews DD, Lyons P. Keeping them in “STYLE”: finding, linking, and retaining young HIV-positive black and Latino men who have sex with men in care. AIDS Patient Care STDS. 2011;25(1):37–45. [PMC free article] [PubMed] [Google Scholar]
10. Magnus M, Jones K, Phillips G, II, et al. Characteristics associated with retention among African American and Latino adolescent HIV-positive men: results from the outreach, care, and prevention to engage HIV-seropositive young MSM of color special project of national significance initiative. J Acquir Immune Defic Syndr. 2010;53(4):529–536. [PubMed] [Google Scholar]
11. Catalani C, Philbrick W, Fraser H, Mechael P, Israelski DM. mHealth for HIV treatment & prevention: a systematic review of the literature. Open AIDS J. 2013;7:17–41. [PMC free article] [PubMed] [Google Scholar]
12. Hamine S, Gerth-Guyette E, Faulx D, Green BB, Ginsburg AS. Impact of mHealth chronic disease management on treatment adherence and patient outcomes: a systematic review. J Med Internet Res. 2015;17(2):e52. [PMC free article] [PubMed] [Google Scholar]
13. Franklin VL, Waller A, Pagliari C, Greene SA. A randomized controlled trial of sweet talk, a text-messaging system to support young people with diabetes. Diabet Med. 2006;23(12):1332–1338. [PubMed] [Google Scholar]
14. Kumar VS, Wentzell KJ, Mikkelsen T, Pentland A, Laffel LM. The DAILY (Daily Automated Intensive Log for Youth) trial: a wireless, portable system to improve adherence and glycemic control in youth with diabetes. Diabetes Technol Ther. 2004;6(4):445–453. [PubMed] [Google Scholar]
15. Kosse RC, Bouvy ML, de Vries TW, Koster ES. Effect of a mHealth intervention on adherence in adolescents with asthma: a randomized controlled trial. Respir Med. 2019;149:45–51. [PubMed] [Google Scholar]
16. Ramsey WA, Heidelberg RE, Gilbert AM, Heneghan MB, Badawy SM, Alberts NM. eHealth and mHealth interventions in pediatric cancer: a systematic review of interventions across the cancer continuum. Psychooncology. 2020;29(1):17–37. [PubMed] [Google Scholar]
17. Darling KE, Sato AF. Systematic review and meta-analysis examining the effectiveness of mobile health technologies in using self-monitoring for pediatric weight management. Child Obes. 2017;13(5):347–355. [PubMed] [Google Scholar]
18. Lall P, Lim SH, Khairuddin N, Kamarulzaman A. Review: an urgent need for research on factors impacting adherence to and retention in care among HIV-positive youth and adolescents from key populations. J Int AIDS Soc. 2015;18(2 Suppl 1):19393. [PMC free article] [PubMed] [Google Scholar]
19. Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep. 2015;12(1):173–190. [PMC free article] [PubMed] [Google Scholar]
20. Belzer ME, Naar-King S, Olson J, et al. The use of cell phone support for non-adherent HIV-infected youth and young adults: an initial randomized and controlled intervention trial. AIDS Behav. 2014;18(4):686–696. [PMC free article] [PubMed] [Google Scholar]
21. Hightow-Weidman L, Muessig K, Knudtson K, et al. A gamified smartphone app to support engagement in care and medication adherence for HIV-positive young men who have sex with men (AllyQuest): development and pilot study. JMIR Public Health Surveill. 2018;4(2):e34. [PMC free article] [PubMed] [Google Scholar]
22. Tanner AE, Song EY, Mann-Jackson L, et al. Preliminary impact of the weCare social media intervention to support health for young men who have sex with men and transgender women with HIV. AIDS Patient Care STDS. 2018;32(11):450–458. [PMC free article] [PubMed] [Google Scholar]
23. Kirschke-Schwartz H, Kreer C.Bijou: An interactive and engaging online, mobile-friendly, self-management program for HIV+ YMSM. Annual Ryan White Conference;
24. Lorig KR, Ritter P, Stewart AL, et al. Chronic disease self-management program: 2-year health status and health care utilization outcomes. Med Care. 2001;39(11):1217–1223. [PubMed] [Google Scholar]
25. Lorig KR, Ritter PL, Laurent DD, Plant K. The internet-based arthritis self-management program: a one-year randomized trial for patients with arthritis or fibromyalgia. Arthritis Rheum. 2008;59(7):1009–1017. [PubMed] [Google Scholar]
26. Mayer RE. Cognitive theory of multimedia learning. In: Mayer R, ed. The Cambridge Handbook of Multimedia Learning. Cambridge University Press; 2005:31–48. [Google Scholar]
27. Paivio A. Mental representations: a dual coding approach. Oxford University Press; 1990. [Google Scholar]
28. Ritter PL, Lorig K. The English and Spanish self-efficacy to manage chronic disease scale measures were validated using multiple studies. J Clin Epidemiol. 2014;67(11):1265–1273. [PubMed] [Google Scholar]
29. Lorig KR, Sobel DS, Ritter PL, Laurent D, Hobbs M. Effect of a self-management program on patients with chronic disease. Eff Clin Pract. 2001;4(6):256–262. [PubMed] [Google Scholar]
30. Norman CD, Skinner HA. eHEALS: the eHealth literacy scale. J Med Internet Res. 2006;8(4):e27. [PMC free article] [PubMed] [Google Scholar]
31. Horvath KJ, Carrico AW, Simoni J, Boyer EW, Amico KR, Petroll AE. Engagement in HIV medical care and technology use among stimulant-using and nonstimulant-using men who have sex with men. AIDS Res Treat. 2013;2013:121352. [PMC free article] [PubMed] [Google Scholar]
32. James DC, Harville C, II, Sears C, Efunbumi O, Bondoc I. Participation of African Americans in e-Health and m-Health studies: a systematic review. Telemed J E Health. 2017;23(5):351–364. [PubMed] [Google Scholar]
33. Gaysynsky A, Romansky-Poulin K, Arpadi S. “My YAP Family”: analysis of a Facebook group for young adults living with HIV. AIDS Behav. 2015;19(6):947–962. [PubMed] [Google Scholar]
34. Bennett GG, Steinberg DM, Stoute C, et al. Electronic health (eHealth) interventions for weight management among racial/ethnic minority adults: a systematic review. Obes Rev. 2014;15(Suppl 4):146–158. [PubMed] [Google Scholar]
35. Baumel A, Kane JM. Examining predictors of real-world user engagement with self-guided ehealth interventions: analysis of mobile apps and websites using a novel dataset. J Med Internet Res. 2018;20(12):e11491. [PMC free article] [PubMed] [Google Scholar]
36. Anand T, Nitpolprasert C, Kerr SJ, et al. A qualitative study of Thai HIV-positive young men who have sex with men and transgender women demonstrates the need for eHealth interventions to optimize the HIV care continuum. AIDS Care. 2017;29(7):870–875. [PubMed] [Google Scholar]
37. Beach LB, Greene GJ, Lindeman P, et al. Barriers and facilitators to seeking HIV services in Chicago among young men who have sex with men: perspectives of HIV service providers. AIDS Patient Care STDS. 2018;32(11):468–476. [PMC free article] [PubMed] [Google Scholar]
38. Ventuneac A, Li DH, Mongrella MC, et al. Exploring potential implementation barriers and facilitators of the SMART program, a stepped-care package of eHealth HIV prevention interventions for adolescent men who have sex with men. Sex Res Social Policy. 2020;17(3):378–388. [PMC free article] [PubMed] [Google Scholar]
Articles from Journal of the International Association of Providers of AIDS Care are provided here courtesy of SAGE Publications
