Administration for Children and Families Youth Programs – Sexual Orientation and Gender Identity (SOGI) Items

OMB No.: 0970-0355

Expiration Date: XX/XX/XXXX

Instrument 2. SOGI Measures Pretest:

Youth Virtual Cognitive Interview and Focus Group Discussion Protocol

THE PAPERWORK REDUCTION ACT OF 1995

This collection of information is voluntary and will be used to provide the Administration for Children and Families with information to help refine sexual orientation and gender identity measures. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0355, Exp: XX/XX/XXXX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Melissa Thomas at mthomas@mathematica-mpr.com

Fall 2024

SOGI Measures: Virtual Cognitive Interview and Focus Group

Discussion Protocol

Note to reviewer: The session will be a discussion about suggested survey questions on gender identity and sexual orientation, to gather feedback from youth about the questions and response options. Youth will begin the session by reviewing the questions (Appendix C). For the virtual cognitive interviews, the questions will be displayed on the screen for participants to review. For the in-person focus groups, youth will be provided a document including the questions. This will be followed by a group discussion about the questions. The same general protocol will be used for both the virtual cognitive interviews and in-person focus groups. Distinctions for each group are indicated with “VIRTUAL” OR “IN-PERSON” below.

Check in participants

1. RECORD ATTENDANCE IN A STAND-ALONE FILE THAT CAN BE DELETED/SHREDDED

2. ONCE EVERYONE HAS ARRIVED, START THE INTRODUCTION

Introduction

Thanks for helping us today! My name is [NAME], and this is my colleague, [NAME]. We work for a research company called Mathematica. Mathematica was hired by the Administration for Children and Families, which is within the U.S. Department of Health and Human Services, part of the federal government that works to support the health and well-being of people, including youth.

We’re here today because we want to get your feedback and thoughts about some questions that could be used on future surveys with people your age. We want to learn how well these survey questions capture the experiences and characteristics of youth in grades 7-12. Your participation today will help us make the questions better for when other youth take the surveys. You can help us make sure the questions are clear and use words that people will understand, make sense based on your experiences, and help us learn if there may be better ways to ask the questions. Although we don’t expect you to speak for everyone, we consider each of you an expert in how well these questions work for people your age. Thank you again for your help.

Our session today will take no more than an hour. In a few minutes, I’m going to ask you to review some questions. You do not have to answer the questions for yourself – simply review them and consider your thoughts on the questions. After everyone is done reviewing the questions, we will have a group discussion so I can hear your thoughts and opinions about them. We will walk through the questions together and you should chime in anytime to provide your thoughts and opinions.

Informed assent information [VIRTUAL: collected at time of parent consent/ IN-PERSON: collected in-person at arrival]

Before we get started, I’d like to be sure you are aware of a few things. [VIRTUAL: Before being allowed to join the virtual group/IN-PERSON: When you arrived] those of you who are under the age of 18 were asked to sign a form called an assent form, and your parent or caregiver has already provided permission for you to participate in this discussion. Those of you who are 18 or older were asked to sign a consent form agreeing to participate prior to this discussion. I want to make sure all of you know your rights for participating in this discussion today.

• First, your participation is completely voluntary. That means you can decide if you want to participate and can stop participating at any time without any reason or consequence.

• Your responses in the discussion will not be shared with anyone outside of the study team.

• During the discussion, you can decide not to answer any questions that I ask the group.

• We ask that everyone in this group respect each other’s privacy by NOT talking about the details of our discussion with anyone outside the group. However, we want everyone to understand that because this is a group discussion, we cannot guarantee your privacy of what you say in front of each other.

• When we share what we learned during today’s discussion we will NOT use your name or any other information that could identify you. We will only share information about you if required by law—that means that if we learn that you have been hurt or are in danger or if you tell us that you plan to seriously hurt yourself or someone else, then, by law, we must report it to the appropriate legal authorities to make sure you are safe.

• We would like to audio/video-record and take notes during our discussion. This is to help us remember all of your thoughts and ideas to make improvements to the survey questions. We will not share this recording with anyone outside of our study team and will destroy it once the study is finished. If anyone does not want to be recorded, we will not record the discussion. Does everyone agree to be recorded? CONFIRM EVERYONE’S ANSWER. IF ANYONE DISAGREES, SESSION WILL NOT BE RECORDED.

Does anyone have any questions at this point?

Review survey questions

Let’s get started. First, I would like for you to review some questions on sexual orientation and gender identity. We are not asking you to give your personal answers to these questions – we just want you to review them and give us feedback on how clear they are, how youth may think about them when responding, and how they could be improved.

[VIRTUAL: I will show the questions on the screen and ask you to take a few minutes to review the questions. Then I’ll ask the group some follow-up questions.] [IN-PERSON: I’ll pass out a list of questions for you to review. As you review the questions, if you have any thoughts or questions, please note them on the paper and we will ask about them during the discussion.

ALL: We are interested in any and all feedback you have for us, so please feel free to share any thoughts you have. There are no right or wrong answers. Are there any questions? PAUSE FOR QUESTIONS. Let’s begin.

VIRTUAL: BEGIN POSTING QUESTIONS FOR PARTICIPANTS TO VIEW ON SCREEN. FOLLOW UP WITH QUESTIONS BELOW.

IN PERSON: Go ahead and begin reviewing the sheet of questions. When you are done, please turn the paper over so I know you’ve completed.

Discussion group guidelines

We understand that participating in a group like this might be new to some of you. Before we begin our discussion, we want to establish a few group rules. Please feel free to add to this if you have other things we should consider as a group:

• There are no right or wrong answers to our questions. We just want to know your true opinions—all feedback is welcome!

• Remember to be respectful of everyone’s opinions, even if they are different from your own. We are really interested to hear if you have different thoughts and opinions about these questions, and welcome that discussion.

• Please do not share what we discuss today outside of this room. It is up to each of you to respect one another’s privacy and honor this request. 

1. Let’s go around and each say your first name and your favorite (movie, food, TV show). You can also say your pronouns if you would like. If you feel more comfortable using a “fake” name, you can do that. We will not record this part of the discussion.

2. Some of the topics we talk about today may be things you are not used to talking about. If we ask anything that you don’t want to answer or that makes you feel uncomfortable – please just let me know and we can skip that question.

3. As we go through specific questions, you don’t need to tell us what your answer is or would be. Our goal for today is to get your feedback on how these questions are worded and your thought process for these questions, so you can help us improve the questions.

4. IN PERSON: Each of you have a green and red card. We will use these throughout the discussion to get feedback on the questions you liked and did not like. For example, sometimes I might ask you to look at a specific question and ask everyone to hold up the green card if you like the question or the red card if you don’t like it, then we can discuss why folks felt the way they did. Remember there is no right or wrong answer, and we are not trying to get everyone to agree on something. It’s okay if everyone has different thoughts and ideas. It is possible you may not feel either way about a question, and that is okay too. If that is the case, you don’t need to hold up a card.

5. VIRTUAL: We are going to ask you to sometimes tell us your opinion by clicking on the thumbs up or thumbs down emojis. Let’s practice doing this together right now. Move your cursor to the bottom of your screen, to the Reactions button. If you don’t see it, look for 3 dots, and click on those 3 dots. You will then see some emojis pop up just above the Reactions button. Click on the thumbs up. Great, looks like everyone did that. Now click on the thumbs down. [PROVIDE SUPPORT IF NEEDED]. Perfect. We will use these throughout the discussion to get feedback on the survey questions you liked and did not like. For example, sometimes I might ask you to look at a specific question and ask everyone to click the thumbs up emoji if you like the question or the thumbs down emoji if you don’t like it, then we can discuss why folks felt the way they did. Remember there is no right or wrong answer and we are not trying to get everyone to agree on something. It’s okay if everyone has different thoughts and ideas. It is possible you may not feel either way about a question, and that is okay too. If that is the case, you don’t need to click on an emoji.

6. Do you have any questions before we begin or anything to add to our group rules?

DATA COLLECTION PROTOCOL

VIRTUAL COGNITIVE INTERVIEWS AND IN-PERSON FOCUS GROUP DISCUSSIONS

BEGIN DISCUSSION. REMIND PARTICIPANTS SESSIONS ARE BEING RECORDED. CHECK THAT RECORDING HAS BEGUN.

High level questions

Note to reviewer: Because participants in the virtual discussions will review the questions one at a time, we will ask the high-level questions at the end of the discussion. Participants in the in-person focus group discussion will see review all of the questions at once and we will ask the high-level questions at the beginning of the discussion.

We have some questions about the general process of answering these questions:

1. Overall, do you think people your age will feel comfortable providing honest answers to these questions? Why or why not?

2. Do you think it will be easy or difficult for people your age to answer these questions? If difficult: What do you think makes it difficult?

3. Do you think these questions make sense, to you and others your age? Which ones don’t make sense for others your age?

4. Were there any words or phrases that were confusing or seemed overly complicated? If so, which ones? What was confusing about them? What words or phrases would you use instead?

5. Are there any words or phrases that seemed out of date or not the way people your age might phrase things? If so, which ones? What words or phrases would you use instead?

Individual item questions

We’re going to go question by question and I’d like to hear your thoughts on each question. VIRTUAL: For question 1 (display on screen); IN-PERSON: Please look at question 1…

6. Looking at the question itself – was it clear what was being asked? If not, what was unclear?

7. How confident are you that others your age will understand what this question is asking?

8. Let’s look now at the answer choices listed here. Thinking about yourself and others your age, do you think everyone would be able to choose an answer here? Is there an answer for everyone here?

9. How confident are you that your peers know what each of these answer options means? Are they clear? If not: what about them is confusing? Are there any that are missing?

10. Is there anything that we didn’t discuss that would improve this question or the responses offered?

REPEAT THIS PROCESS FOR EACH OF THE QUESTIONS WITHIN A GIVEN TOPIC (SEXUAL ORIENTATION OR GENDER IDENTITY).

Comparison questions

Now that we have reviewed each individual question, I want you to look at them next to each other and tell me what you think (VIRTUAL: DISPLAY SEXUAL ORIENTATION Q1 AND Q2; THEN REPEAT QUESTIONS FOR THE “WINNER” VS. Q3; THEN THE “WINNER” VS. Q4, ETC.) REPEAT FOR GENDER IDENTITY QUESTIONS.

1. Of the different question choices, which do you think is the better question? Why?

2. Of the different question choices, which do you think is the easiest for people to answer? Why?

3. Which do you think most youth would be most comfortable answering? Why?

4. Which do you think would be most difficult to answer? Why?

Wrap-up

Those were all the questions I had, but I want to make sure we don’t miss any more feedback from you to help us improve these questions.

Please take a moment to look back through the survey questions. Do you have any additional comments? Did we miss anything in our discussion that you would like to mention now? This can be about your general thoughts, feedback, or opinions on the surveys and the experience of answering the questions, rather than about specific questions. PAUSE FOR COMMENTS.

Thank you so much for joining the discussion today and sharing your thoughts with us. [VIRTUAL: If you have any follow up questions, we added our email address in the chat and can stay on a few more moments. Otherwise, you will receive an email with a thank you note and your gift card as a thank you for participating.] [IN-PERSON: You may now leave and to receive your gift card as a thank you for participating.]

Thanks, and have a wonderful [day/afternoon/evening]!