Early life stress (ES) increases vulnerability to psychopathology and impairs cognition in adulthood. These ES-induced deficits are associated with lasting changes in hippocampal plasticity. Detailed information on the neurobiological basis, the onset, and progression of such changes and their sex-specificity is currently lacking but is required to tailor specific intervention strategies. Here, we use a chronic ES mouse model based on limited nesting and bedding material from postnatal day (P) 2–9 to investigate; (1) if ES leads to impairments in hippocampus-dependent cognitive function in adulthood and (2) if these alterations are paralleled by changes in developmental and/or adult hippocampal neurogenesis. ES increased developmental neurogenesis (proliferation and differentiation) in the dentate gyrus (DG) at P9, and the number of immature (NeurD1+) cells migrating postnatally from the secondary dentate matrix, indicating prompt changes in DG structure in both sexes. ES lastingly reduced DG volume and the long-term survival of developmentally born neurons in both sexes at P150. In adult male mice only, ES reduced survival of adult-born neurons (BrdU/NeuN+ cells), while proliferation (Ki67+) and differentiation (DCX+) were unaffected. These changes correlated with impaired performance in all learning and memory tasks used here. In contrast, in female mice, despite early alterations in developmental neurogenesis, no lasting changes were present in adult neurogenesis after ES and the cognitive impairments were less prominent and only apparent in some cognitive tasks. We further show that, although neurogenesis and cognition correlate positively, only the hippocampus-dependent functions depend on changes in neurogenesis, whereas cognitive functions that are not exclusively hippocampus-dependent do not. This study indicates that chronic ES has lasting consequences on hippocampal structure and function in mice and suggests that male mice are more susceptible to ES than females. Unraveling the mechanisms that underlie the persistent ES-induced effects may have clinical implications for treatments to counteract ES-induced deficits.