#crawlers — Public Fediverse posts

I like the new single from Crawlers.

Apple Music: https://music.apple.com/gb/album/cool/6781042262?i=6781042263

Spotify: https://open.spotify.com/track/7BKTCgTQvXFBdCDBtODvQc

#Music #Crawlers

I like the new single from Crawlers.

Apple Music: https://music.apple.com/gb/album/cool/6781042262?i=6781042263

Spotify: https://open.spotify.com/track/7BKTCgTQvXFBdCDBtODvQc

#Music #Crawlers

Win one of the TEN copies of Vampire Crawlers game by Poncle in #js13k 2026!

https://js13kgames.com/2026/prizes#vampirecrawlers

#gamedev #gamedevjs #Poncle #Vampire #Crawlers #VampireCrawlers #Poncle #Games

Win one of the TEN copies of Vampire Crawlers game by Poncle in #js13k 2026!

https://js13kgames.com/2026/prizes#vampirecrawlers

#gamedev #gamedevjs #Poncle #Vampire #Crawlers #VampireCrawlers #Poncle #Games

ICYMI: AI crawlers hit sites 50,000 times per human visit, Cloudflare data shows: AI crawlers reach up to 50,000 visits per referred reader, Cloudflare data shows, as a Munich court strips Google of its liability shield over AI Overviews. https://ppc.land/ai-crawlers-hit-sites-50-000-times-per-human-visit-cloudflare-data-shows/ #AI #Crawlers #DataAnalysis #Cloudflare #Google

ICYMI: AI crawlers hit sites 50,000 times per human visit, Cloudflare data shows: AI crawlers reach up to 50,000 visits per referred reader, Cloudflare data shows, as a Munich court strips Google of its liability shield over AI Overviews. https://ppc.land/ai-crawlers-hit-sites-50-000-times-per-human-visit-cloudflare-data-shows/ #AI #Crawlers #DataAnalysis #Cloudflare #Google

#Development #Launches
ua-tracer · Inspect what a user agent actually does https://ilo.im/16e91f

_____
#UserAgents #Crawlers #AI #Websites #Development #WebDev #Frontend #CSS #JavaScript #Backend

#Development #Launches
ua-tracer · Inspect what a user agent actually does https://ilo.im/16e91f

_____
#UserAgents #Crawlers #AI #Websites #Development #WebDev #Frontend #CSS #JavaScript #Backend

#Design #Inspirations
This page left intentionally blank · “In our age of generative AI, a blank page is a deliberate act!” https://ilo.im/16e912

_____
#Print #Content #Websites #Blogs #SmallWeb #IndieWeb #AI #Crawlers #UxDesign #WebDesign

#Design #Inspirations
This page left intentionally blank · “In our age of generative AI, a blank page is a deliberate act!” https://ilo.im/16e912

_____
#Print #Content #Websites #Blogs #SmallWeb #IndieWeb #AI #Crawlers #UxDesign #WebDesign

#Cloudflare gives #AI #crawlers a September deadline: pay publishers or get blocked - https://thenextweb.com/news/cloudflare-block-ai-crawlers-pay-publishers "From 15 September, Cloudflare will block crawlers that harvest content for AI training from any page carrying ads, unless the owner opts in, and pay publishers when their work shapes an AI answer. "

#Cloudflare gives #AI #crawlers a September deadline: pay publishers or get blocked - https://thenextweb.com/news/cloudflare-block-ai-crawlers-pay-publishers "From 15 September, Cloudflare will block crawlers that harvest content for AI training from any page carrying ads, unless the owner opts in, and pay publishers when their work shapes an AI answer. "

ICYMI: Microsoft Clarity now flags robots.txt violations inside Bot Analytics: Microsoft Clarity now surfaces robots.txt violations in Bot Analytics, showing publishers which AI crawlers break access rules and what content they target. https://ppc.land/microsoft-clarity-now-flags-robots-txt-violations-inside-bot-analytics/ #MicrosoftClarity #BotAnalytics #SEO #WebAnalytics #Crawlers

#Business #Guides
What server logs reveal that SEO tools miss · How to uncover hidden SEO issues at scale https://ilo.im/16dm9u

_____
#Google #Bing #SEO #SearchEngines #Crawlers #LogFiles #WebAnalytics #Development #WebDev #Backend

#Business #Guides
What server logs reveal that SEO tools miss · How to uncover hidden SEO issues at scale https://ilo.im/16dm9u

_____
#Google #Bing #SEO #SearchEngines #Crawlers #LogFiles #WebAnalytics #Development #WebDev #Backend

ICYMI: New York passes bill forcing AI crawlers to identify themselves to news sites: New York's Assembly passed A11292 on June 5, 2026, requiring AI crawlers to disclose identity and purpose to news publishers or face $15,000-per-day penalties. https://ppc.land/new-york-passes-bill-forcing-ai-crawlers-to-identify-themselves-to-news-sites/ #NewYork #AI #Crawlers #News #Legislation

ICYMI: New York passes bill forcing AI crawlers to identify themselves to news sites: New York's Assembly passed A11292 on June 5, 2026, requiring AI crawlers to disclose identity and purpose to news publishers or face $15,000-per-day penalties. https://ppc.land/new-york-passes-bill-forcing-ai-crawlers-to-identify-themselves-to-news-sites/ #NewYork #AI #Crawlers #News #Legislation

#Business #Reports
Bots have overtaken humans on the internet · “The open web is now being swamped by AI agents.” https://ilo.im/16dlpj

_____
#Bots #Crawlers #WebTraffic #WebAnalytics #AI #Websites #Development #WebDev #Frontend #Backend

#umustbekidding #AI "#training #crawlers made up ~67.5% of observed AI bot volume in 2025" the #internet in 2026: more #bot ( #AI searching the #web) #traffic than #human traffic #wtf https://dwaves.de/2026/06/04/the-internet-in-2026-more-bot-ai-searching-the-web-traffic-than-human-traffic-wtf/ #wtf #howcanitbe #omg #answers ?

#umustbekidding #AI "#training #crawlers made up ~67.5% of observed AI bot volume in 2025" the #internet in 2026: more #bot ( #AI searching the #web) #traffic than #human traffic #wtf https://dwaves.de/2026/06/04/the-internet-in-2026-more-bot-ai-searching-the-web-traffic-than-human-traffic-wtf/ #wtf #howcanitbe #omg #answers ?

#umustbekidding #AI "#training #crawlers made up ~67.5% of observed AI bot volume in 2025" the #internet in 2026: more #bot ( #AI searching the #web) #traffic than #human traffic #wtf https://dwaves.de/2026/06/04/the-internet-in-2026-more-bot-ai-searching-the-web-traffic-than-human-traffic-wtf/ #wtf #howcanitbe #omg #answers ?

#Anti #AI #Software / #Website Protection / AI #poison

https://iocaine.madhouse-project.org/

iocaine features:

> No sympathy towards #crawlers

Stand in the way of AI but not your visitors

> Lightweight = Very little #CPU and #RAM

Tries its best to require little resources from your visitors too.

If you can serve #static content, you'll be able to run #iocaine too.

#Anti #AI #Software / #Website Protection / AI #poison

https://iocaine.madhouse-project.org/

iocaine features:

> No sympathy towards #crawlers

Stand in the way of AI but not your visitors

> Lightweight = Very little #CPU and #RAM

Tries its best to require little resources from your visitors too.

If you can serve #static content, you'll be able to run #iocaine too.

"Our new analysis shows that more than 340 local news sites across the United States are now limiting the #InternetArchive’s ability to access and preserve their stories.

Researchers, historians, and citizens around the world rely on the web archives of #localnews sites to do their work.

“Blocking the Internet Archive’s web #crawlers threatens one of the most effective ways that we capture and store news content for the long term.”"

https://www.niemanlab.org/2026/05/more-than-340-local-news-outlets-are-limiting-the-internet-archives-access-to-their-journalism/

"Our new analysis shows that more than 340 local news sites across the United States are now limiting the #InternetArchive’s ability to access and preserve their stories.

Researchers, historians, and citizens around the world rely on the web archives of #localnews sites to do their work.

“Blocking the Internet Archive’s web #crawlers threatens one of the most effective ways that we capture and store news content for the long term.”"

https://www.niemanlab.org/2026/05/more-than-340-local-news-outlets-are-limiting-the-internet-archives-access-to-their-journalism/

Fucking #crawlers of #meta #facebook #apple #google are easting bandwidth and creating nonsense. In the last 3 days meta developer crawlers alone ate up 850GB+ #bandwidth. Assholes.

Fucking #crawlers of #meta #facebook #apple #google are easting bandwidth and creating nonsense. In the last 3 days meta developer crawlers alone ate up 850GB+ #bandwidth. Assholes.

Crawlers – Roadside Attractions and Saban Films pick up the new spider-infested horror film. Details here https://www.liveforfilm.com/2026/05/04/crawlers-roadside-attractions-and-saban-films-pick-up-the-new-spider-infested-horror-film/

#Crawlers #MatildaLutz #film #spiders #horror

Crawlers – Roadside Attractions and Saban Films pick up the new spider-infested horror film. Details here https://www.liveforfilm.com/2026/05/04/crawlers-roadside-attractions-and-saban-films-pick-up-the-new-spider-infested-horror-film/

#Crawlers #MatildaLutz #film #spiders #horror

Spider-Horror Pic ‘Crawlers’ Acquired By Roadside Attractions & Saban Films
#Acquisitions #News #Crawlers #RoadsideAttractions #SabanFilms

https://deadline.com/2026/04/crawlers-roadside-attractions-1236877140/

Spider-Horror Pic ‘Crawlers’ Acquired By Roadside Attractions & Saban Films
#Acquisitions #News #Crawlers #RoadsideAttractions #SabanFilms

https://deadline.com/2026/04/crawlers-roadside-attractions-1236877140/

Blocking AI crawlers cost news publishers 7% of traffic, study finds: A Wharton and Rutgers study finds news publishers who blocked LLM crawlers lost 7% of weekly traffic in 6 weeks, with no measurable content protection gains. https://ppc.land/blocking-ai-crawlers-cost-news-publishers-7-of-traffic-study-finds/ #AI #crawlers #NewsPublishers #TrafficLoss #ContentProtection

Blocking AI crawlers cost news publishers 7% of traffic, study finds: A Wharton and Rutgers study finds news publishers who blocked LLM crawlers lost 7% of weekly traffic in 6 weeks, with no measurable content protection gains. https://ppc.land/blocking-ai-crawlers-cost-news-publishers-7-of-traffic-study-finds/ #AI #crawlers #NewsPublishers #TrafficLoss #ContentProtection

We're live!

Avali Week AND 420? Hell yeah!
$145 flat colors and $25 crawlers!
See ya there!

https://fweeblies.art

#twitch #stream #avaliweek #420 #crawlers #avali

We're live!

Avali Week AND 420? Hell yeah!
$145 flat colors and $25 crawlers!
See ya there!

https://fweeblies.art

#twitch #stream #avaliweek #420 #crawlers #avali

Oh look even more crawlers lmao

#crawlers #fresnonightcrawler #cryptidapril

Oh look even more crawlers lmao

#crawlers #fresnonightcrawler #cryptidapril

ICYMI: Google rewrites Googlebot's rulebook: 2MB limits, IP moves, and what crawlers really are: Google today published two blog posts revealing Googlebot's true architecture as a shared SaaS platform, a 2MB fetch limit, and a new IP ranges directory path. https://ppc.land/google-rewrites-googlebots-rulebook-2mb-limits-ip-moves-and-what-crawlers-really-are/ #Google #Googlebot #SEO #Crawlers #WebMaster

#Development #Explainers
Inside Googlebot · How Google’s crawl system decides which content gets indexed https://ilo.im/16btho

_____
#Business #Google #SearchEngine #SEO #Crawlers #Content #RobotsTxt #Development #WebDev #Frontend

#Development #Explainers
Inside Googlebot · How Google’s crawl system decides which content gets indexed https://ilo.im/16btho

_____
#Business #Google #SearchEngine #SEO #Crawlers #Content #RobotsTxt #Development #WebDev #Frontend

Quo Vadis, Crawlers? Progress and what’s next on safeguarding our infrastructure https://diff.wikimedia.org/2026/03/26/quo-vadis-crawlers-progress-and-whats-next-on-safeguarding-our-infrastructure/ #AI, #AIDataCrawlers, #Crawlers, #Infrastructure, #Knowledge, #KnowledgeAsAService, #Scraping, #ScrapingBots, #WebScraping, #WikimediaFoundation, #WikimediaProjects

Quo Vadis, Crawlers? Progress and what’s next on safeguarding our infrastructure https://diff.wikimedia.org/2026/03/26/quo-vadis-crawlers-progress-and-whats-next-on-safeguarding-our-infrastructure/ #AI, #AIDataCrawlers, #Crawlers, #Infrastructure, #Knowledge, #KnowledgeAsAService, #Scraping, #ScrapingBots, #WebScraping, #WikimediaFoundation, #WikimediaProjects

Fresh on my #blog: "There's a difference between 'scraping' and 'retrieving'". I have a dilemma; one is extractive, the other accessibility. What do I do?
#ArtificialIntelligence #LLMs #crawlers #ethics #AIethics
thomasrigby.com/posts/theres-a-difference-between-scraping-and-retrieving/

52467 requests in about 8h from #OpenAI #crawlers with no sign of stopping.

#Development #Findings
Markdown, llms.txt, and AI crawlers · Do Markdown and llms.txt matter for your website? https://ilo.im/16b5qb

_____
#Business #SEO #SearchEngines #AI #Crawlers #Content #Website #Markdown #LlmsTxt #RobotsTxt

#Development #Reports
Google lists Googlebot file limits · Do Google’s crawling limits affect your website? https://ilo.im/16adna

_____
#Business #Google #SearchEngine #Crawlers #Googlebot #Files #HTML #PDF #WebDev #Frontend

#Development #Reports
Google lists Googlebot file limits · Do Google’s crawling limits affect your website? https://ilo.im/16adna

_____
#Business #Google #SearchEngine #Crawlers #Googlebot #Files #HTML #PDF #WebDev #Frontend

#Business #Initiatives
AI’s free web scraping days may be over · Say hello to RSS’s younger, tougher brother https://ilo.im/166s9q

_____
#Web #Publishing #Website #Blog #Content #AI #Crawlers #Payments #RSL #RSS #RobotsTxt

Hey does anyone know if there's still a working zip bomb style exploit that can be deployed on a static site/JS (or as a asset/resource)? Specifically to target web scrapers and AI bullshit? The second any server goes online now it's immediately bombarded by stupid numbers of requests.

#hacking #aislop #crawlers #webscraping #webcrawler #robots #zipbomb #zipbombing #exploit #robotstxt #server #scraper