Inactivating mutations in chromodomain helicase DNA binding protein 7 (CHD7) cause CHARGE syndrome, a severe multiorgan system disorder of which Isolated gonadotropin-releasing hormone (GnRH) deficiency (IGD) is a minor feature. Recent reports have described predominantly missense CHD7 alleles in IGD patients, but it is unclear if these alleles are relevant to causality or overall genetic burden of Kallmann syndrome (KS) and normosmic form of IGD. To address this question, we sequenced CHD7 in 783 well-phenotyped IGD patients lacking full CHARGE features; we identified nonsynonymous rare sequence variants in 5.2% of the IGD cohort (73% missense and 27% splice variants). Functional analyses in zebrafish using a surrogate otolith assay of a representative set of these CHD7 alleles showed that rare sequence variants observed in controls showed no altered function. In contrast, 75% of the IGD-associated alleles were deleterious and resulted in both KS and normosmic IGD. In two families, pathogenic mutations in CHD7 coexisted with mutations in other known IGD genes. Taken together, our data suggest that rare deleterious CHD7 alleles contribute to the mutational burden of patients with both KS and normosmic forms of IGD in the absence of full CHARGE syndrome. These findings (i) implicate a unique role or preferential sensitivity for CHD7 in the ontogeny of GnRH neurons, (ii) reiterate the emerging genetic complexity of this family of IGD disorders, and (iii) demonstrate how the coordinated use of well-phenotyped cohorts, families, and functional studies can inform genetic architecture and provide insights into the developmental biology of cellular systems.