User-agent: * Disallow: /_private/ Disallow: /_vti_log/ Disallow: /aspnet_client/ Disallow: /db/ Disallow: /error/ Disallow: /_ScriptLibrary/ Disallow: /images/ Disallow: /sound/ Disallow: /pdf/ User-agent: psbot Disallow: /_private/ Disallow: /_vti_log/ Disallow: /_ScriptLibrary/ Disallow: /bp/ Disallow: /cgi-bin/ Disallow: /_vti_bin/ Disallow: /db/ Disallow: /aspnet_client/ Disallow: /images/ Disallow: /images/ User-agent:classroombot Disallow: /_private/ Disallow: /_vti_log/ Disallow: /_vti_bin/ Disallow: /db/ Disallow: /error/ Disallow: /aspnet_client/ Disallow: /images/ Disallow: /_ScriptLibrary/ Disallow: /cgi-bin/ Disallow: /pdf/ User-Agent: Googlebot Disallow: /cgi-bin Disallow: /test/ Disallow: /images/ Disallow: *.cgi User-agent: Mediapartners-Google* Disallow: /cgi-bin Disallow: /test/ Disallow: /images/ User-Agent: Googlebot-Image Disallow: /images/ User-agent: asterias Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: AttributorCorporationDMCABot Disallow: * User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle * Disallow: / User-agent: Bitacle* Disallow: / User-agent: Bitacle Disallow: / User-agent: BaiDuSpider Disallow: / User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html) Disallow: / User-agent:classroombot Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: DoCoMo Disallow: / User-Agent: datacha0s/2.0 Disallow: / User-Agent: eagle Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot Disallow: *? User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Gigabot Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: ia_archive Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver/1.6 Disallow: / User-agent: IconSurf Disallow: / User-agent: IBM_Planetwide Disallow: / User-agent: ichiro/2.0 (ichiro@nttr.co.jp) Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: Mozilla/4.0 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 Disallow: / User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) Disallow: / User-agetn: Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T) Disallow: / User-agent: MSIECrawler Disallow: / User-agent: moget/2.1 (moget@goo.ne.jp) Disallow: / User-Agent: MJ12bot Disallow: / User-agent: Moget Disallow: / User-agent: Nutch Disallow: / User-agent: Openbot Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: psbot Disallow: / User-agent: Robozilla Disallow: / User-agent: Scrubby Disallow: / User-agent: scooter Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: slurp@inktomi.com Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: SlySearch Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: Teoma Disallow: / User-agent: TurnitinBot/2.0 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: TurnitinBot/1.5 Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Tutorial Crawler Disallow: / User-agent: Titan Disallow: / User-agent: Twiceler-0.9+http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler Disallow: / User-agent: vscooter Disallow: / User-agent: Webcollage Disallow: / User-agent: Web Downloader/6.9 Disallow: / User-agent: Web Downloader/5.7 Disallow: / User-agent: Web Downloader/4.5 Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: Wget/1.9+cvs-stable (Red Hat modified) Disallow: / User-agent: Wget/1.9.1 Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Yahoo-MMCrawler/3.x Disallow: / User-agent: Yahoo-Blogs Disallow: / User-agent: Yahoo-Blogs/v3.9 Disallow: / User-agent: YahooYSMcm Disallow: / User-agent: YahooYSMcm/2.0.0 Disallow: / User-agent: YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/) Disallow: / User-agent: YahooSeeker Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: y!j-bsc Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: Y!J Disallow: / User-agent: Y!J/1.0 Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: /