# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no Allow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CloudflareBrowserRenderingCrawler Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content # Major Search Engines - Allow with restrictions User-agent: Googlebot User-agent: bingbot User-agent: slurp User-agent: duckduckbot User-agent: yandexbot User-agent: baiduspider User-agent: applebot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: anthropic-ai Disallow: /api/ Disallow: /admin/ Disallow: /logout Disallow: /privacy Disallow: /terms-and-conditions Crawl-delay: 1 # Social Media Preview Crawlers - Allow for marketing User-agent: facebookexternalhit User-agent: twitterbot User-agent: linkedinbot Allow: / # Archive Services - Allow User-agent: ia_archiver Allow: / # AI Training Bots - Block User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: bytespider Disallow: / User-agent: bytedance Disallow: / # Malicious/Unwanted Bots User-agent: semrushbot Disallow: / User-agent: ahrefsbot Disallow: / User-agent: dotbot Disallow: / User-agent: whatcms Disallow: / User-agent: rogerbot Disallow: / User-agent: blexbot Disallow: / User-agent: linkfluence Disallow: / User-agent: mj12bot Disallow: / User-agent: aspiegelbot Disallow: / User-agent: domainstatsbot Disallow: / User-agent: cincraw Disallow: / User-agent: nimbostratus Disallow: / User-agent: httrack Disallow: / User-agent: serpstatbot Disallow: / User-agent: megaindex Disallow: / User-agent: semanticbot Disallow: / User-agent: cocolyzebot Disallow: / User-agent: domcopbot Disallow: / User-agent: traackr Disallow: / User-agent: bomborabot Disallow: / User-agent: linguee Disallow: / User-agent: webtechbot Disallow: / User-agent: clickagy Disallow: / User-agent: sqlmap Disallow: / User-agent: internet-structure-research-project-bot Disallow: / User-agent: seekport Disallow: / User-agent: awariosmartbot Disallow: / User-agent: onalyticabot Disallow: / User-agent: buck Disallow: / User-agent: riddler Disallow: / User-agent: sbl-bot Disallow: / User-agent: df Disallow: / User-agent: pubmatic Disallow: / User-agent: bvbot Disallow: / User-agent: sogou Disallow: / User-agent: barkrowler Disallow: / User-agent: embed.ly Disallow: / User-agent: semantic-visions Disallow: / User-agent: voluumdsp Disallow: / User-agent: wc-test-dev-bot Disallow: / User-agent: gulperbot Disallow: / User-agent: moreover Disallow: / User-agent: ltx71 Disallow: / User-agent: accompanybot Disallow: / User-agent: mauibot Disallow: / User-agent: moatbot Disallow: / User-agent: seznambot Disallow: / User-agent: zagbot Disallow: / User-agent: ccbot Disallow: / User-agent: tweetmemebot Disallow: / User-agent: paperlibot Disallow: / User-agent: livelapbot Disallow: / User-agent: slurp Disallow: / User-agent: sottopop Disallow: / User-agent: mail.ru_bot Disallow: / User-agent: rasabot Disallow: / User-agent: anderspinkbot Disallow: / User-agent: zoominfobot Disallow: / User-agent: castlebot Disallow: / User-agent: linkdexbot Disallow: / User-agent: coccocbot Disallow: / User-agent: yacybot Disallow: / User-agent: isec_bot Disallow: / User-agent: flockbrain Disallow: / User-agent: csscheck Disallow: / User-agent: surdotlybot Disallow: / User-agent: contxbot Disallow: / User-agent: relemindbot Disallow: / User-agent: pulno Disallow: / User-agent: zombiebot Disallow: / User-agent: dmasslinksafetybot Disallow: / User-agent: linkpadbot Disallow: / User-agent: aaabot Disallow: / User-agent: mtrobot Disallow: / User-agent: snappreviewbot Disallow: / User-agent: scrapy Disallow: / User-agent: bytespider Disallow: / User-agent: bytedance Disallow: / User-agent: mozlila Disallow: / User-agent: nessus Disallow: / User-agent: binlar Disallow: / User-agent: blackwidow Disallow: / User-agent: blekkobot Disallow: / User-agent: blowfish Disallow: / User-agent: bullseye Disallow: / User-agent: bunnys Disallow: / User-agent: butterfly Disallow: / User-agent: careerbot Disallow: / User-agent: casper Disallow: / User-agent: checkpriv Disallow: / User-agent: cheesebot Disallow: / User-agent: cherrypick Disallow: / User-agent: chinaclaw Disallow: / User-agent: choppy Disallow: / User-agent: clshttp Disallow: / User-agent: cmsworld Disallow: / User-agent: copernic Disallow: / User-agent: copyrightcheck Disallow: / User-agent: cosmos Disallow: / User-agent: crescent Disallow: / User-agent: cy_cho Disallow: / User-agent: datacha Disallow: / User-agent: diavol Disallow: / User-agent: discobot Disallow: / User-agent: dittospyder Disallow: / User-agent: dotnetdotcom Disallow: / User-agent: dumbot Disallow: / User-agent: emailcollector Disallow: / User-agent: emailsiphon Disallow: / User-agent: emailwolf Disallow: / User-agent: extract Disallow: / User-agent: eyenetie Disallow: / User-agent: seokicks Disallow: / User-agent: futuribot Disallow: / User-agent: netpeakcheckerbot Disallow: / User-agent: yellowbrandprotectionbot Disallow: / User-agent: datagnionbot Disallow: / User-agent: uptime-kuma Disallow: / User-agent: peer39 Disallow: / User-agent: crawler Disallow: / User-agent: claudebot Disallow: / User-agent: fidget Disallow: / User-agent: my-tiny Disallow: / User-agent: flaming Disallow: / User-agent: flashget Disallow: / User-agent: flicky Disallow: / User-agent: foobot Disallow: / User-agent: g00g1e Disallow: / User-agent: getright Disallow: / User-agent: gigabot Disallow: / User-agent: go-ahead-got Disallow: / User-agent: gozilla Disallow: / User-agent: grabnet Disallow: / User-agent: grafula Disallow: / User-agent: harvest Disallow: / User-agent: heritrix Disallow: / User-agent: icarus6j Disallow: / User-agent: jetbot Disallow: / User-agent: jetcar Disallow: / User-agent: jikespider Disallow: / User-agent: kmccrew Disallow: / User-agent: leechftp Disallow: / User-agent: libweb Disallow: / User-agent: linkextractor Disallow: / User-agent: linkscan Disallow: / User-agent: linkwalker Disallow: / User-agent: loader Disallow: / User-agent: masscan Disallow: / User-agent: miner Disallow: / User-agent: majestic Disallow: / User-agent: mechanize Disallow: / User-agent: morfeus Disallow: / User-agent: moveoverbot Disallow: / User-agent: netmechanic Disallow: / User-agent: netspider Disallow: / User-agent: nicerspro Disallow: / User-agent: nikto Disallow: / User-agent: ninja Disallow: / User-agent: nutch Disallow: / User-agent: octopus Disallow: / User-agent: pagegrabber Disallow: / User-agent: planetwork Disallow: / User-agent: postrank Disallow: / User-agent: purebot Disallow: / User-agent: pycurl Disallow: / User-agent: python Disallow: / User-agent: queryn Disallow: / User-agent: queryseeker Disallow: / User-agent: radian6 Disallow: / User-agent: radiation Disallow: / User-agent: realdownload Disallow: / User-agent: scooter Disallow: / User-agent: seekerspider Disallow: / User-agent: semalt Disallow: / User-agent: siclab Disallow: / User-agent: sindice Disallow: / User-agent: sistrix Disallow: / User-agent: sitebot Disallow: / User-agent: siteexplorer Disallow: / User-agent: sitesnagger Disallow: / User-agent: skygrid Disallow: / User-agent: smartdownload Disallow: / User-agent: snoopy Disallow: / User-agent: sosospider Disallow: / User-agent: spankbot Disallow: / User-agent: spbot Disallow: / User-agent: stackrambler Disallow: / User-agent: stripper Disallow: / User-agent: sucker Disallow: / User-agent: surftbot Disallow: / User-agent: finditanswersbot Disallow: / User-agent: vebidoobot Disallow: / User-agent: coibotparser Disallow: / User-agent: anybot Disallow: / User-agent: dingtalkbot Disallow: / User-agent: echobot Disallow: / User-agent: popscreen Disallow: / User-agent: vuhuvbot Disallow: / User-agent: marketwirebot Disallow: / User-agent: hypestat Disallow: / User-agent: whatyoutypebot Disallow: / User-agent: mixrankbot Disallow: / User-agent: xforce Disallow: / User-agent: smtbot Disallow: / User-agent: thesis-research-bot Disallow: / User-agent: sux0r Disallow: / User-agent: suzukacz Disallow: / User-agent: suzuran Disallow: / User-agent: takeout Disallow: / User-agent: teleport Disallow: / User-agent: telesoft Disallow: / User-agent: true_robots Disallow: / User-agent: turingos Disallow: / User-agent: vampire Disallow: / User-agent: vikspider Disallow: / User-agent: voideye Disallow: / User-agent: webleacher Disallow: / User-agent: webreaper Disallow: / User-agent: webstripper Disallow: / User-agent: webvac Disallow: / User-agent: webviewer Disallow: / User-agent: webwhacker Disallow: / User-agent: winhttp Disallow: / User-agent: wwwoffle Disallow: / User-agent: woxbot Disallow: / User-agent: xaldon Disallow: / User-agent: xxxyy Disallow: / User-agent: yamanalab Disallow: / User-agent: yioopbot Disallow: / User-agent: youda Disallow: / User-agent: zeus Disallow: / User-agent: zmeu Disallow: / User-agent: zune Disallow: / User-agent: zyborg Disallow: / User-agent: lanaibot Disallow: / User-agent: metadataparser Disallow: / User-agent: go-http-client Disallow: / User-agent: daum Disallow: / User-agent: libwww Disallow: / User-agent: guzzlehttp Disallow: / User-agent: expert-html Disallow: / User-agent: geedobot Disallow: / User-agent: newspaper Disallow: / User-agent: peacockmedia Disallow: / User-agent: gobuster Disallow: / User-agent: expanseinc Disallow: / User-agent: crawling Disallow: / User-agent: tineye Disallow: / User-agent: damieng Disallow: / User-agent: scpitspi Disallow: / User-agent: screaming Disallow: / User-agent: babbar Disallow: / User-agent: scalaj Disallow: / User-agent: turnitin Disallow: / User-agent: blackbox Disallow: / User-agent: okhttp Disallow: / User-agent: externalhit Disallow: / User-agent: dataforseo Disallow: / User-agent: semrush Disallow: / User-agent: contentking Disallow: / User-agent: siteauditbot Disallow: / User-agent: botify Disallow: / User-agent: cxense Disallow: / User-agent: revvim Disallow: / User-agent: colly Disallow: / User-agent: github Disallow: / User-agent: img2dataset Disallow: / User-agent: petalbot Disallow: / User-agent: whizebot Disallow: / User-agent: projectdiscovery Disallow: / User-agent: datenbutler Disallow: / User-agent: seebot.org Disallow: / User-agent: fuseonbot Disallow: / User-agent: vipnytt Disallow: / User-agent: pandalytics Disallow: / User-agent: ninjbot Disallow: / User-agent: gowikibot Disallow: / User-agent: 360spider Disallow: / User-agent: acapbot Disallow: / User-agent: acoonbot Disallow: / User-agent: ahrefs Disallow: / User-agent: alexibot Disallow: / User-agent: asterias Disallow: / User-agent: attackbot Disallow: / User-agent: backdorbot Disallow: / User-agent: becomebot Disallow: / User-agent: kauaibot Disallow: / # Sitemap Sitemap: https://vulnetix.com/sitemap.xml