User-agent: Screaming Frog SEO Spider Disallow: User-agent: * Disallow: /wp-admin/ Disallow: /go/ Disallow: /nzb/