High-glycemic foods, which cause rapid spikes in blood sugar, can negatively impact both memory and overall brain health.