Walk the directories with readdir, and don't stat everything we can find. Thanks to davej for the public humiliation reminding me to go back and re-fix this one.