Evaluation of the Feasibility and Perceived Value of Integrating Learn the Signs. Act Early. Developmental Monitoring Resources in Early Head Start

Many children with developmental disabilities are not identified before age 3 years of age, preventing them from being able to fully benefit from early intervention services. Early childhood educators, particularly those in Early Head Start (EHS) programs, are important partners in the early identification of children with developmental delays. Learn the Signs. Act Early. (LTSAE) is a program of the U.S. Centers for Disease Control and Prevention that provides free developmental monitoring resources to increase the early identification of children with developmental delays and disabilities. This paper presents findings from the first evaluation of the use of LTSAE resources in EHS, which was conducted across four states and 11 EHS programs from fall 2018 through spring 2019. Surveys (n = 448) and interviews (n = 39) with EHS management, staff, and parents indicated that LTSAE resources were valued and accepted, and their use in EHS considered feasible. Importantly, families and staff reported the LTSAE materials provided shared language to help them more effectively discuss development. These findings inform EHS and other early education programs that wish to enhance developmental monitoring, screening, and referral.

Early identification of developmental disabilities is critical for children and their families to have access to early intervention services that have been shown to improve skills and outcomes, reduce challenging behaviors, and enable families to develop strategies and obtain resources needed for successful family functioning (Bradshaw et al., 2015;Estes et al., 2015;Noyes-Grosser et al., 2018). Early intervention involves an array of services and supports that are employed to assist children with developmental delays or disabilities (Centers for Disease Control & Prevention [CDC], 2021). Furthermore, early intervention can reduce the need for more costly interventions over time (Cidav et al., 2017). Based on nationally representative survey data from 2009 to 2017, about 17% of U.S. children aged 3-17 years, or 1 in 6 children, had a developmental disability such as autism spectrum disorder or attention deficit hyperactivity disorder (Zablotsky et al., 2019). Every U.S. state and territory has a program for providing low or no cost early intervention services through Part C of the Individuals with Disabilities Education Act (IDEA) for children 0-3 years of age. Early intervention services may include speech therapy, physical therapy, and other types of services based on the needs of the child and family (CDC, 2021). However, many children are not identified before age 3 years preventing them from being able to fully benefit from these services. In one longitudinal birth cohort study, only 10% of children eligible for Part C early intervention received services by the time they were 24 months old (Rosenberg et al., 2008).
Early identification is a process that includes developmental monitoring (also called surveillance), developmental screening, referral, and evaluation for services (Barger et al., 2018a, b). The American Academy of Pediatrics (AAP) first issued guidance in 2006, and again in 2020, regarding best practices for early identification through developmental surveillance and screening  in pediatric primary health care. Developmental surveillance is a process that occurs during health supervision visits. It consists of reviewing milestone checklists and developmental history, asking parents about concerns, assessing strengths and risks, observing the child, documenting the findings, and obtaining and sharing results with others (e.g., early childhood educators, WIC providers, home visitors, etc.). Developmental screening involves the use of standardized screening tools at specific ages and whenever surveillance indicates a developmental concern. The guidelines from the AAP emphasize the importance of both developmental surveillance and screening in identifying developmental delays and disabilities and have been shown to be more effective when practiced together (Barger et al., 2018a, b). Despite the guidance, many children do not receive surveillance or screening from their healthcare provider. The 2016 U.S. National Survey of Children's Health found that only 37% of parents of children 9-35 months of age reported that their healthcare provider performed developmental surveillance during visits and only 30% reported having completed a developmental screening in the past 12 months. Additionally, just 19% of parents reported their children received both developmental monitoring and screening (Hirai et al., 2018). Healthcare provider reports of screening are higher than parent reports with 63% of general pediatricians indicating that they always or almost always conduct developmental screening with their patients . It is unclear whether recall bias, lack of family engagement in the screening process, or other factors explain this discrepancy.
Early childhood education programs are well positioned to improve early identification and support children who are not receiving developmental monitoring or screening through their healthcare provider. Based on 2011 U.S. census data, 12.4 million or 61.3% of children under the age of 5 years were in a childcare arrangement or preschool setting and almost a quarter of those were cared for in a child care center, nursery school, preschool, or Head Start program (Laughlin, 2013). Research has shown that early childhood educators (ECEs) can accurately identify children with developmental delays even without specialized training, although this ability improves with more specialized training (Branson & Bingham, 2017;Zhang et al., 2019).
Although many ECE programs are undertaking efforts that improve early identification of children with developmental disabilities, there is limited information available with respect to the extent of these activities. One webbased survey of childcare center directors in a midwestern state showed that 16% conducted developmental screening in their programs and 54% viewed screening as part of the ECE's professional role (Boh & Johnson, 2018). A national survey of 489 ECEs found that ECEs commonly monitor development (86%), but not explicitly for the purpose of identifying developmental concerns and often without the use of a tool like a milestone checklist. Of note, over 15% of ECEs surveyed did not know where to refer families if they suspected a delay (Chödrön et al., 2019). There is support for ECE early identification activities and federal initiatives such as Birth To Five: Watch Me Thrive! have aimed to educate ECEs and other early childhood professionals about best practices for developmental monitoring, screening, and referral to services (U.S. Department of Health & Human Services & U.S. Department of Education, 2014). Some federally funded programs, such as Head Start and Early Head Start, encourage early identification practices through technical assistance and program standards.
In the U.S., Head Start is a national program to promote school readiness through high quality child and family development services for low-income pregnant women and families (Administration for Children & Families [ACF], 2019a). Early Head Start (EHS) programs are particularly important for early identification, as they serve infants and toddlers up to the age of 3 years. EHS programs are guided by the Head Start Performance Standards, which require that at least 10% of children enrolled are eligible for services under IDEA. EHS programs are mandated to complete or obtain a current developmental screening to identify concerns regarding a child's developmental, behavioral, motor, language, social, cognitive, and emotional skills within 45 calendar days of a child's enrollment. Further, programs must engage parents as their children's primary teachers and nurturers and implement intentional strategies to engage parents in their children's learning and development (ACF, 2019b). EHS services may be center-based or delivered through a home visitation model. Additionally, EHS programs may enter into a childcare collaboration to deliver EHS services to children enrolled in a non-EHS child care program (ACF, 2020). Children in EHS have been shown to be more likely to receive Part C services, but gaps in early identification may remain. Children in the EHS Research andEvaluation Project, 1996-2001, were significantly more likely to have received Part C services than those in the control group.
Though a majority of children in EHS (87%) were determined to be at developmental risk, only 4.7% of the total sample had received Part C services (Peterson et al., 2004). Another study found that a third of children were determined to be at developmental risk both in the original study when they were between 0-3 years of age and at follow up when they were 3-5 years of age, but never received early intervention services (Peterson et al., 2011).
Learn the Signs. Act Early. (LTSAE) is a program of the CDC that aims to increase the early identification of children with developmental delays and disabilities so children and their families can access early intervention services. The program offers free milestone checklists and other resources to facilitate developmental monitoring, support developmental screening, and encourage early action regarding developmental concerns. LTSAE milestone content is based on two AAP resources, Bright Futures: Guidelines for Health Supervision of Infants, Children, and Adolescents and Caring for Your Baby and Young Child: Birth to Age 5 (Hagan et al., 2008;Shelov & Altmann, 2009). Content has been adapted for parents as the primary audience and has been tested with parents for readability and comprehension (Raspa et al., 2015). The LTSAE program differs from other child development resources because it intentionally includes milestones that reflect what most children can do by given ages, as opposed to an average age, to better identify developmental delays. Many other milestone resources include milestones that are at the 50th percentile, meaning that half of children would be expected to attain a milestone at that age and half would not. This helps parents get a sense of their child's developmental trajectory but is less helpful for identifying when they should be concerned about a missed milestone. The LTSAE milestone checklists are available online as printable PDFs or through the mobile device Milestone Tracker app. The program also offers supporting materials including children's books, brochures, posters, training videos, and fact sheets on seeking help for developmental concerns and how providers can discuss developmental concerns with families (see www. cdc. gov/ ActEa rly/ Mater ials) (See Fig. 1). Program materials have been evaluated in a number of settings including health care (Gadomski et al., 2018) and child care (Chödrön et al., 2020;Graybill et al., 2016), but neither feasibility nor impact of LTSAE in EHS has been evaluated. In a setting like EHS with program standards on screening, family engagement, and enrolling children eligible for services under IDEA, it is unclear whether LTSAE monitoring resources complement or duplicate resources and processes that may be already in place for early identification. Furthermore, there are no published studies on the parents' perspectives of developmental monitoring and screening in EHS. This study aimed to assess the feasibility and perceived value of integrating LTSAE resources in select EHS programs across four states and was conducted from fall 2018 through spring 2019. The evaluation questions were as follows (1) How can LTSAE be integrated into EHS programs? (2) Does the use of LTSAE in EHS influence developmental monitoring attitudes or behaviors of staff and parents? (3) How does the use of LTSAE in EHS influence the communication between parents and EHS staff around development? (4) To what extent does the use of LTSAE in EHS influence parent engagement with developmental screening and referrals for a developmental concern?

Study Design
The study employed a convergent, parallel mixed-method research design (Creswell & Plano Clark, 2011). A onegroup, posttest-only design was used for the quantitative strand of the study that included the collection of parent, staff, and management survey data. A practical qualitative approach was utilized for the qualitative strand of the study (Stufflebeam & Coryn, 2014). Institutional Review Boards at the University of Alabama at Birmingham, Indiana University, University of Iowa, and the University of Wisconsin-Madison approved, exempted, or determined that the project did not constitute research.

Procedures
EHS programs were recruited from May to July 2018 and the study took place from August 2018 to March 2019. EHS programs were recruited through existing partnerships with the study team members and 48 were invited to participate in the study. In total, 11 EHS programs agreed to participate in the study, 7 operating in rural and 4 in urban areas. As seen in Table 1, programs operated multiple center-based, home visiting, and childcare collaborative programs. Most programs were able to distribute LTSAE materials to families in August or September 2018 at the beginning of the program year. Practical considerations such as program capacity to begin integrating materials and LTSAE materials delivery timing influenced the start dates. The duration of integration of materials in EHS programs ranged from 8 to 26 weeks. The survey and qualitative data collection took place from February to March 2019.

Integration of LTSAE
The study team and EHS program directors and managers collaboratively developed options for how EHS programs could integrate the LTSAE materials into existing practices. Either the study team or EHS management led trainings for staff on these options for integrating LTSAE materials. These trainings varied in length, with the average study team-conducted training lasting 52 min and the average EHS-led training lasting 15 min. All participating EHS programs used the milestone checklists, most often the printable single sheet and Milestone Moments Booklet versions. Some EHS programs elected to use promotional items such as children's books, posters, and flyers to promote the Milestone Tracker App in addition to the milestone checklists. All EHS programs are required to use a curriculum, but the specific curriculum and accompanying parent handouts in use at the time of LTSAE integration was unknown. Programs had the options of providing LTSAE materials in packets at the beginning of the program year, at parentteacher conferences, home visits, and at special parent events throughout the year. Staff were encouraged to ask parents to complete the milestone checklists and talk with families about them. If concerns about a child's development arose, staff were to follow their program's approved procedures for follow-up and referral.

Participants
English-speaking parents, staff, and management participated in surveys, interviews, and focus groups (see Table 2 for total participants by methodology). Across the four states, 25 focus groups were conducted, including 6 in Alabama, 10 in Indiana, 7 in Iowa, and 2 in Wisconsin. Fifteen individual interviews were conducted with six in Alabama, seven in Iowa, and two in Wisconsin. More than 90% of survey, focus group, and interview respondents were female. Most parents who responded to the survey described themselves as White (49%) or Black/African American (41%) and 12% indicated Hispanic or Latino origin. For the focus groups and interview participants, again most parents identified as White (53%) or Black/ African American (40%). The race/ ethnicity of staff and managers in the focus groups and interviews was mostly White (67%) and Black/ African American (26%), with 6% indicating Hispanic or Latino origin. See Tables 3 and 4 for additional characteristics of respondents.

Surveys
Three surveys (parent, staff, and management) were developed for this study. The surveys were administered using the Qualtrics survey application and could be completed using a computer, tablet, or smartphone. EHS staff also distributed the parent survey in paper form as needed. A designated EHS contact provided the team with process data during integration including numbers of milestone checklists and LTSAE materials shared with families as well as discussions staff had with families about completed checklists.

Focus Groups and Interviews
The study team conducted focus groups and interviews using semi-structured interview guides tailored for parents, staff, and managers. All interview guides included questions about experience with LTSAE materials and recommendations. Example questions included, "Thinking about a time when you used the materials, what was your experience using them?" and "What would you say to other Early Head Start programs that are interested in using the LTSAE materials?" The staff interview guide probed further on feasibility and  The manager interview guide also asked about noticeable changes in staff and parents and alignment to EHS program, as shown in the following example: "Please share specific ways the materials and training aligned or not, with existing EHS performance standards and existing practices throughout the program." Parent focus groups were often conducted during parent events. Staff focus groups were often held in place of regularly scheduled staff meetings. Management interviews took place with program managers at each program site who were directly involved with LTSAE integration. Staff focus groups and management interviews were held in person or via video-or tele-conference. In three instances, management focus groups were conducted instead of interviews for participants' convenience.

Measures
Attitudes toward feasibility and the perceived value of materials were measured through a combination of survey items. The staff and management survey question on ease of use of LTSAE components was "Since having received the LTSAE materials, how easy or difficult has it been to do the following activities?" (e.g., "Hand out LTSAE materials to parents" and "Review completed milestone checklists with parents"). The response options were on a 1 to 4 Likert scale (i.e., very easy, easy, difficult, very difficult). Barriers to integrating LTSAE materials were assessed through this multiple-choice question on the staff survey, "Which of the following barriers to using LTSAE did you experience?" Opinions pertaining to the continued use of LTSAE in EHS were captured using this yes or no survey question to staff, managers, and parents, "In your opinion, should your EHS program continue to offer LTSAE materials to families?" The extent that staff and managers felt that the LTSAE materials affected parent education and communication about children's development was measured through a question that asked respondents to "indicate how much you agree or disagree with the following statements" (e.g., "Inclusion of LTSAE in our program helped staff feel more confident having conversations with parents about milestones"). Response options were on a 1 to 4 Likert scale (i.e., strongly agree, agree, disagree, strongly disagree). Parent opinions about their abilities and confidence tracking and discussing milestones as well as acting on developmental concerns was measured through a question that asked parents to indicate "how much you agree or disagree the following statements. Because of these materials:" (e.g., "I feel more confident tracking my child's developmental milestones.") Response options were on a 1 to 4 Likert scale (i.e., strongly agree, agree, disagree, strongly disagree).

Analysis
Descriptive statistics were conducted for all quantitative items. Qualitative data obtained from parent and staff focus groups and management interviews and focus groups were analyzed using coding and thematic analysis facilitated by templates designed to address the project evaluation questions. Different templates were developed for each participant category (i.e., parent, staff, and management). The templates mapped the responses pertaining to the focus group and interview questions and probes to the program evaluation questions. The study team reviewed each transcript, applied a priori codes to text, and entered a summary of the coded text by state into the templates along with direct quotes that exemplified the findings. To ensure consistent coding and summarizing across team members, validity checks were conducted by having 3 to 4 team members code and summarize the same transcripts. Any errors or disagreements were resolved through a group review and necessary revisions were made to the codes and summaries. Then, the template summaries by state were consolidated into overall project summaries presented by evaluation question. The overall summaries were then developed into a narrative that was reviewed for accuracy by the study team.

LTSAE Integration
Process data indicate that across all EHS programs sites, staff distributed 1165 milestone checklists, 1456 Milestone Moments Booklets, and 1385 Milestone Tracker app flyers to parents during the months of LTSAE integration. During this time staff across all sites had 993 conversations with parents using completed milestone checklists. Additionally, from the staff survey, nearly all staff (99%) reported sharing the LTSAE materials with families and 70% reported reviewing checklists with most parents. More than half (58%) of staff members reported that they had completed a developmental milestone checklist for a child (separately from parents).

Survey Results
In total, 448 surveys from managers, staff, and parents were included in the study, with 36% from Alabama (n = 162), 21% from Indiana (n = 93), 22% from Iowa (n = 99), and 21% from Wisconsin (n = 94). Response rates for the managers, staff, and parent surveys were 71%, 61%, and 38% respectively. EHS staff surveyed were primarily teachers (55%) or home-based visitors (35%). EHS management was more diverse and often served multiple roles. These categories are therefore not mutually exclusive and included Directors (16), Child Development and Education Managers (5), Disability Services Managers (4), Health Services Manager (3), Teachers (2), Family and Community Partnerships Manager (1), Family Engagement Specialist (1), other managers and supervisors (3), and other (1). About 90% of home visiting staff and 94% of centerbased staff found the LTSAE activities very easy or easy to integrate into their work with families (see Fig. 2). Similarly, most managers reported that it was very easy (35%) or easy (56%) to integrate LTSAE materials with staff in their programs. Managers provided narrative responses to support their answers on ease of integrating materials and the most common responses were that LTSAE materials aligned with their program, explained developmental milestones to parents, and were easy for the staff to use and understand. Three managers (9%) mentioned that it was difficult to make time to integrate the materials; one manager (4%) reported that staff integrated the materials but some parents did not use them, and another (4%) reported that staff were initially confused about how to integrate the materials.
Most managers (74%) reported that their program did not experience any barriers with the use of the LTSAE materials, while some (11%) reported they did experience barriers or were unsure (14%). Managers described the barriers to using LTSAE materials in narrative responses. Two managers (6%) said there was not enough time to use the materials, one (4%) reported that parents were not interested in using the information, and one (4%) reported that their staff thought the materials were inaccurate, believing the checklists were intended to reflect average ages of milestone attainment. Of staff who reported at least one barrier (55%), the four most common barriers identified were a lack of time to use the materials (26%), conflicting demands (26%), feeling that parents were unreceptive to materials (24%), and low literacy levels among parents (24%).
Most staff and managers strongly agreed or agreed with statements about the use of LTSAE increasing staff confidence in communicating with parents about development and developmental concerns, improvements to referral processes, as well as improvements in parent's ability to complete developmental screenings (see Fig. 3). Most parents (93%) reported that, because of the LTSAE materials, they talked to their child's teacher or home visitor about their child's developmental milestones. Most parents (72%) also agreed that they felt more comfortable talking to their child's teacher or home visitor about any concerns they had about developmental milestones. The majority agreed that, because of the materials, they felt more confident tracking their child's milestones, knew which milestones their child should have reached, felt confident talking to their child's teacher about milestones, and knew what to do if they had a developmental concern (Fig. 4).
Overall, 28% of parents had a concern about their child's development and followed up with a professional about their concern. Half of the parents surveyed (51%) reported that the materials helped them identify a concern about how their child plays, learns, speaks, acts, or moves. Of these parents, about 68% said their teacher or home visitor suggested that they talk to someone (e.g., a doctor or other professional) about their concern and most of these parents (80%) reported that they did so. Of those who reported speaking with a professional about their concern, 52% thought the materials were very helpful when talking to their doctor, 26% thought they were helpful, 1% thought they were unhelpful, and 1% were not sure. Thirty-two parents provided narrative  responses to explain what happened when they spoke to a professional about their concern. Reported outcomes for these conversations were that 28% of parents received additional information about their concern, 28% were referred to another professional, 25% were currently receiving help for their concern, 16% reported that their child was diagnosed with a developmental or physical delay, and 3% reported that they talked to a doctor but the doctor did not identify a developmental delay. Most parents (92%), staff (85%), and managers (97%) indicated that their program should continue to offer LTSAE materials to families. Managers explained through narrative responses why their program should continue to offer LTSAE materials to families and the most common responses were that materials help parents track and understand their children's development, were a useful resource for teachers to communicate with families, and generally enhanced the program.

Acceptability and Usefulness of LTSAE Materials
Overall feedback from parents, staff, and managers on the use of LTSAE in EHS was positive. Staff consistently noted the materials helped them set goals with families and talk through next steps, in addition to serving as a reference point on typical development during parent conferences and meetings. Staff noted that parents who already had developmental concerns were more likely to utilize the materials than those whose children seemed to be developing on time. Staff, especially at center-based programs, did struggle to find enough time to implement the use of the checklists and other materials. For some families, language barriers and parental illiteracy prevented use of materials. Still, managers did not consistently report resistance from staff regarding implementing LTSAE materials with families.
We are always hesitant to try something new because it ends up being a lot more work, but this didn't.

-EHS Manager
There were some diverse opinions about how well the materials aligned with other EHS materials and curriculum. A few managers mentioned a discrepancy between the milestone information in the LTSAE materials and the milestones that EHS staff typically look for as part of their curriculum. In contrast, some managers, a few parents, and one staff member noted that LTSAE appeared redundant with milestone information in existing EHS program curriculum. However, many managers reported that LTSAE materials were well-aligned and helped meet EHS performance standards, especially aligning with education for staff and parents, fostering parent confidence and skills, and promoting family engagement.
I foresee us continuing to use the materials because it aligns with our curriculum and it provides that 'boost' to our parents to identify developmental milestones on their own.-EHS Manager Overall, respondents in all three categories expressed positive views on the milestone content of the LTSAE materials. Parents mentioned that having information on developmental milestones is important and useful. Staff noted that the materials bolstered their own knowledge of developmental milestones and served as a helpful aid in explaining developmental milestones to parents/caregivers. Managers also reported an increase in parent confidence related to child development knowledge and ability to talk with others about their child's development in those parents who had used the materials to track milestones.  Fig. 4 Parent responses about tracking and discussing milestones after using the "Learn the Signs. Act Early." materials in Early Head Start When they approached us with these milestones, told us how to read it, how to go by it, and how to use it, it was very, very helpful.-EHS Parent The materials gave families the language to talk to providers.-EHS Manager

Areas for Improvement
Some areas for improvement related to the training on and timing of distribution of LTSAE materials were noted. In some cases, unclear expectations and some staff members not receiving the training resulted in confusion among staff. Some suggested holding parent-specific trainings and others suggested providing more training for staff in the form of hands-on activities and booster sessions so the time between implementation and training was shorter. Staff and managers shared that introducing materials mid-program year was not optimal, particularly because developmental screenings occur at the beginning of the year in EHS programs. A suggestion was made that receiving materials prior to the start of the school year would allow the staff to become familiar with the materials before the screenings that occur at the beginning of the year. Another theme was that some parents felt overwhelmed if several LTSAE materials were given at one time, particularly given the high number of EHS materials families typically received.
I don't think anything did not work well. It's just like, some families don't have the time when you bring them a huge pack full of things to go through. That's why I think maybe smaller components [of the materials] at smaller times would give us a better chance to spread it out over the month and talk about it at each visit…-EHS Staff member.

Supporting Communication, Screening, and Referral
Staff and managers consistently reported that they felt materials helped increase parent-staff communication and comfort among parents during discussions about child development by providing a framework and language for staff and parents to use. Staff across all states felt the materials helped parents engage more with their children and complemented EHS's focus on family engagement. Staff across all states reported that LTSAE materials increased their confidence in talking through developmental milestones with parents because the materials offered an evidence-based, credible resource that validated or supported their child's development concerns.

Discussion
Results from this project show that, overall, the integration of LTSAE resources was generally valued, accepted, and considered feasible across a variety of EHS settings in several states. Ninety-three (28%) participating parents reported following up on a developmental concern by talking to a professional, with eight (2%) reporting that their child began receiving services for that concern and five (1.5%) that their child had received a diagnosis. EHS managers, staff and parents agreed that the LTSAE materials should continue to be used in EHS settings and were helpful for monitoring development, developmental screening, and referral. While previous work has described how Head Start and EHS professionals view developmental monitoring as an important practice (Chödrön et al., 2019), this is the first study to examine parent perspectives on the use of developmental monitoring tools in EHS. Two of the most salient findings are that parents felt more knowledgeable about developmental milestones and families and staff felt they had a shared language to meaningfully discuss development because of receiving the materials. This is important because parents are able to better notice and express concerns if they have knowledge and information about typical child development behaviors (Marshall et al., 2016). Moreover, having more meaningful conversations about development is a key aspect of quality family engagement, which is a goal of EHS programs (Hubel et al., 2017). Although the number of parents who reported on the outcomes of conversations with professionals about development concerns was small, it is encouraging to see that conversations took place that led to their children receiving services. While results are promising, there are important lessons for future integration of these resources in practice. First, before disseminating LTSAE materials, managers can determine how LTSAE best meets program needs, performance standards, and complements their existing curriculum and screening and referral procedures. Second, LTSAE resources at the beginning of the program year or at the time of enrollment is ideal to coincide with developmental screenings and allow for staff to have an orientation to the resources during training before the school year begins. The CDC provides a free, one-hour online training course that EHS programs can integrate into this orientation, Watch Me! Celebrating Milestones and Sharing Concerns (www. cdc. gov/ Watch MeTra ining). Third, allowing for gradual introduction of materials through ongoing interactions with parents may prevent families from feeling overwhelmed with material and provide an opportunity for teachers and parents to review materials together as they are introduced. Finally, EHS programs may benefit from integrating LTSAE into specific screening and referral protocols prior to first using the materials with families. In light of previous research indicating that many early childhood educators are not aware of appropriate referrals to make for developmental concerns (Branson & Bingham, 2017;Chödrön et al., 2019) it may be important to clarify and train staff on existing referral protocols to ensure consistency. It is important to develop a procedure to re-screen if there is a concern, refer families to early intervention and their healthcare provider, connect with services, and follow up on referrals before introducing LTSAE materials to ensure that family concerns are appropriately met.

Limitations
The generalizability of these findings into other early childhood education settings may be limited as EHS has specific requirements and structures which may differ from other ECE programs. While conducting the project in multiple EHS programs and in four states allowed for collection of diverse opinions and experiences with LTSAE, the project had several limitations that affect how generalizable these findings may be to other EHS programs. While all parents of enrolled children were offered LTSAE materials, not all parents received the survey. Resource constraints limited the data collection to English-speaking parents. It is possible that non-English speaking parents had different experiences of the program not represented in these results. Additionally, two programs experienced challenges distributing the parent survey and did not offer it to all parents of children enrolled. The number and characteristics of parents in the EHS programs who did not receive the survey were not available to the study team. Also, the response rate to the parent survey was low, as some programs found it challenging to motivate parents to complete the survey. Social desirability may have influenced self-report of behaviors such as following up on developmental concerns. Further, the participating EHS programs may not be representative of all EHS programs. The programs voluntarily participated in this project because managers were interested in utilizing the LTSAE materials and felt their program had some level of capacity to integrate materials and participate in data collection. These findings may be different in programs with different degrees of support from management and capacity to initiate a new activity. While some participants reported that parents received too many materials at once, the project could not control exactly when parents received LTSAE materials and did not have full information about other EHS materials that may have been shared at the same time. Variation across programs is a significant limitation. There were some notable differences in survey results between states and by program implementation time. However, the study design and inconsistency of the findings limit the ability to determine whether these differences are driven by state or population differences, length of program implementation, training or other factors (e.g., type of program, method of implementation). Also, while some staff reported low literacy as a barrier, the literacy level of parents who participated was not assessed. Lastly, the study used a single group, posttest-only design to accommodate the study timeframe and avoid overburdening participants with data collection. However, this design alone has considerable internal validity threats (Shadish et al., 2001), limiting the inferences that could be made about the program specifically causing change in behavior, attitudes or screening and referral outcomes.

Future Directions
To understand the impact of LTSAE, randomized controlled studies are needed that could more fully evaluate integration and LTSAE's ability to improve early identification and children's developmental outcomes. These studies are needed in EHS as well as other ECE settings and would ideally include parents who speak English, Spanish, and other languages. The challenges presented in this project to conducting evaluation research in real program settings, such as variability in how programs integrated LTSAE, data collection challenges, and brief study timeframes, could be addressed by working with only one or two ECE program partners with a single procedure for using LTSAE over two or more school years. More research is needed across early childhood program settings including EHS on early identification processes to target improvements and ensure children are accessing services as early as possible (Barger et al., 2018a, b). ECEs across early learning settings have an important role in early identification of developmental delays and disabilities and many are implementing practices aimed to identify children earlier (Boh & Johnson, 2018;Branson & Bingham, 2017;Ford & Johnson, 2017;Peterson et al., 2011). However, there is a dearth of research on the extent these activities are occurring across the field, and even less study about outcomes on early identification, child health, and wellbeing. Describing these processes would further the understanding of the needs of ECEs and their contributions to early identification in communities. Future controlled studies examining implementation of programs such as LTSAE could help fill these gaps if conducted over a sufficient period of time to measure change in developmental monitoring, developmental screening, referrals for developmental delay, and child outcomes related to early intervention.

Conclusions
This project was the first to examine the feasibility and value of integrating the use of LTSAE resources in EHS settings in the U.S. It was uncertain whether participants would value LTSAE in EHS settings where developmental screening is routine. This project showed that managers, staff and parents valued the LTSAE materials because they are designed for families and offered a common language as a foundation for discussions about development. Although some challenges were identified, results suggest integrating the use of LTSAE resources in a variety of EHS settings was generally feasible and acceptable, and might enhance EHS developmental monitoring, screening, and referral.
Author Contributions All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by JP, FS, ET, AEB, RS-H, CJ, SV, and GC. The first draft of the manuscript was written by JA and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.