Yes -- which is why people in-the-know who are training neural nets periodically get frustrated and say "You know what, this thing's not producing the results I want, it did something really wrong as a matter of fact. Fuck it, I'm going to stop applying a gradient in the direction of better results, until it gets its act together."
Oh and also the neural net is physically in charge of all of our lives in this example