This study aimed to characterize peak exercise cardiac function and thigh muscle fatty infiltration and their relationships with VO2peak among anthracycline-treated breast cancer survivors (BCS). BCS who received anthracycline chemotherapy ~ 1 year earlier (n = 16) and matched controls (matched-CON, n = 16) were enrolled. Resting and peak exercise cardiac function, myocardial T1 mapping (marker of fibrosis), and thigh muscle fat infiltration were assessed by magnetic resonance imaging, and VO2peak by cycle test. Compared to matched-CON, BCS had lower peak SV (64 ± 9 vs 57 ± 10 mL/m2, p = 0.038), GLS (− 30.4 ± 2.2 vs − 28.0 ± 2.5%, p = 0.008), and arteriovenous oxygen difference (16.4 ± 3.6 vs 15.2 ± 3.9 mL/100 mL, p = 0.054). Mediation analysis showed: (1) greater myocardial T1 time (fibrosis) is inversely related to cardiac output and end-systolic volume exercise reserve; (2) greater thigh muscle fatty infiltration is inversely related to arteriovenous oxygen difference; both of which negatively influence VO2peak. Peak SV (R2 = 65%) and thigh muscle fat fraction (R2 = 68%) were similarly strong independent predictors of VO2peak in BCS and matched-CON combined. Post-anthracyclines, myocardial fibrosis is associated with impaired cardiac reserve, and thigh muscle fatty infiltration is associated with impaired oxygen extraction, which both contribute to VO2peak.