Great point. You need concrete for wind, solar, and li-ion battery storage too (including pumped hydro), but out of those I'd say pumped hydro is the only one that remotely compares in the amount of concrete needed for construction.
So purely looking at the emissions from materials needed to build these power sources, renewables have the edge due to less concrete. These emissions might show up elsewhere in raw material extraction like with silicon for solar, and then the rare earth metals needed for generators in wind, all the lithium/nickel/cobalt needed for batteries, etc., but I want to say that the Life Cycle Assessments (LCAs) from places like the National Renewable Energy Laboratory (NREL) in the US or the International Energy Agency (IEA) worldwide have taken that into account and still show that renewables + storage are cheaper on a carbon basis compared to fossil fuels and nuclear.