No, I've tried with that enabled as well, same issue. This isn't before a layer change, it's when it goes from extruding the walls to extruding the infill.
But even without infill this happens. It's the end of the outer wall that doesn't connect to the beginning of the outer wall and it's not really fixed by reducing seam gap, so I'm not sure what's actually causing it.