Background: Due to the rapid aging of the South Korean population, neurological diseases such as dementia are increasing. Many studies have reported that the incidence of dementia is associated with environmental factors along with age. Objectives: This study analyzed the association between cognitive function and ten heavy metals in the body: arsenic, aluminum, chromium, manganese, cobalt, nickel, iron, copper, zinc, and lead. Methods: From 2018 to 2019, a total of 120 participants who suffered from cognitive impairment were recruited for this study. Blood and urine samples were collected and analyzed for heavy metal concentrations using an inductively coupled plasma mass spectrometer. Demographic information was obtained through face-to-face questionnaires completed by a trained investigator. Cognitive function was evaluated with the Korean version of the Mini-Mental State Examination and the Korean version of the Boston Name Waiting Test. The associations between cognitive function scores and heavy metal concentrations were analyzed using multiple logistic regression analysis. Results: The average age of the 120 participants was 72.7 years, and 69.2% were female. The mean of the MMSE-K and K-BNT scores were 22.9 and 37.9, respectively. The geometric mean of aluminum (Al) was 8.42 μg/L. MMSE-K was associated with iron (Fe), but the significance was removed in the logistic regression based on 24 points. K-BNT was significantly associated with aluminum and the odds ratio for K-BNT above 38 decreased by 45% as the aluminum concentration increased. Conclusions: The association between aluminum and the K-BNT score indicated that aluminum is associated with language-related cognitive decline. Based on this result, further study will be conducted by considering co-exposure effects of heavy metals including aluminum.