Highly active antiretroviral therapy (HAART) suppresses viral loads in 71% of people living with HIV globally, while failing to bring adequate immune reconstitution in nearly one-third of them. We hypothesize that the persistence of latent HIV reservoirs in specific memory T cell subsets contributes to impaired immune recovery. We conducted a case-control study to estimate differences in the HIV-1 proviral DNA across memory CD4-positive T cell subsets between participants with CD4 counts of over 500 cells/µL (immune responders or IRs) and those with counts of less than 350 cells/µL (immune non-responders or INRs) with sustained viral suppression. Latent HIV reservoirs (LRs) were detected in at least one memory T cell subset in 48.33% of total participants. Latent reservoirs were more frequent among INRs than among IRs (65.38% versus 35.29%, P = 0.02), particularly in the effector memory T cell subset (34.6% in INRs versus 8.8% in IRs, P = 0.02). Thus, despite long-term viral suppression with HAART, the persistence of latent reservoirs in memory T cells is associated with poor CD4-positive T cell recovery. Emerging classes of antiretroviral agents that target latent viral pools may enhance immune restoration and bring us closer to finding an HIV cure.