#googlebot — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #googlebot, aggregated by home.social.
-
SEO трафик на сайте есть, а заявок нет. Я полез в серверные логи и посмотрел, кто на самом деле «ходит» на сайт
У сайта рос органический трафик, позиции были в порядке, а заявок всё равно не хватало. Клиент задал простой вопрос: «Если людей стало больше, где заявки?» Я полез не только в Метрику, но и в access.log — и там уже начался отдельный зоопарк: Googlebot, YandexBot, GPTBot, Ahrefs, Semrush, MJ12bot, DataForSEO и Chrome, который вёл себя совсем не как человек. В статье покажу, как отличать визиты от серверных запросов, проверять настоящего Googlebot, разбирать crawler'ов и решать, кого вообще стоит пускать на коммерческий сайт через robots.txt .
https://habr.com/ru/articles/1070350/
#SEO #техническое_SEO #серверные_логи #accesslog #robotstxt #Googlebot #YandexBot #вебкраулеры #роботный_трафик #UserAgent
-
Momenteel lijkt het wel alsof het halve internet los gaat. De logfiles staan vol met troep... Ik denk een overblijfsel van een recente zwakheid in #WordPress. Hopelijk waait de boel snel weer over.
Tussen alle troep, ben ik dan wel weer blij om deze bot te zien:
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Ga maar eens lekker de inhoud van mijn blogs indexeren ja 👍
-
Momenteel lijkt het wel alsof het halve internet los gaat. De logfiles staan vol met troep... Ik denk een overblijfsel van een recente zwakheid in #WordPress. Hopelijk waait de boel snel weer over.
Tussen alle troep, ben ik dan wel weer blij om deze bot te zien:
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Ga maar eens lekker de inhoud van mijn blogs indexeren ja 👍
-
ICYMI: Google's expiry tag forces a re-crawl, and it won't fix 50% of 404 hits: Classifieds site cycles 10,000 listings monthly on 24-72 hour lifespans against 5,000 stable URLs. Half of Googlebot hits land on 404s. What the tag misses. https://ppc.land/googles-expiry-tag-forces-a-re-crawl-and-it-wont-fix-50-of-404-hits/ #SEO #Googlebot #404Errors #Webmasters #DigitalMarketing
-
ICYMI: Google's expiry tag forces a re-crawl, and it won't fix 50% of 404 hits: Classifieds site cycles 10,000 listings monthly on 24-72 hour lifespans against 5,000 stable URLs. Half of Googlebot hits land on 404s. What the tag misses. https://ppc.land/googles-expiry-tag-forces-a-re-crawl-and-it-wont-fix-50-of-404-hits/ #SEO #Googlebot #404Errors #Webmasters #DigitalMarketing
-
Google's expiry tag forces a re-crawl, and it won't fix 50% of 404 hits: Classifieds site cycles 10,000 listings monthly on 24-72 hour lifespans against 5,000 stable URLs. Half of Googlebot hits land on 404s. What the tag misses. https://ppc.land/googles-expiry-tag-forces-a-re-crawl-and-it-wont-fix-50-of-404-hits/ #SEO #Googlebot #404errors #DigitalMarketing #WebDevelopment
-
Google's expiry tag forces a re-crawl, and it won't fix 50% of 404 hits: Classifieds site cycles 10,000 listings monthly on 24-72 hour lifespans against 5,000 stable URLs. Half of Googlebot hits land on 404s. What the tag misses. https://ppc.land/googles-expiry-tag-forces-a-re-crawl-and-it-wont-fix-50-of-404-hits/ #SEO #Googlebot #404errors #DigitalMarketing #WebDevelopment
-
Cap Puckhaber shares how Cloudflare block Googlebot can impact your website's impressions and visibility in Google Search Console.
https://websitejockey.com/cloudflare-blocking-googlebot-bluehost-403-error/
#cloudflare #googlebot #google #googlesearchconsole #seo #webmanagement
-
Cap Puckhaber shares how Cloudflare block Googlebot can impact your website's impressions and visibility in Google Search Console.
https://websitejockey.com/cloudflare-blocking-googlebot-bluehost-403-error/
#cloudflare #googlebot #google #googlesearchconsole #seo #webmanagement
-
Cloudflare: из «трубы» — в вахтёра и кассира
(про трубу в заголовке я немного зря - CF всегда были средством доставки, но любили их и за защиту; другими словами, надо бы писать про "трубу с ситечком", но в заголовок такое не ложится) Сама новость: Cloudflare, всем нам хорошо знакомый провайдер CDN и WAF, теперь будет не просто выделять в статистике посещений ИИ-ботов, но и разделять их по категориям, разрешая одним то, что запрещено другим. Ну и да, прямо замаячила тема оплаты за доступ. Brave new world, как по мне! Cloudflare разделила ИИ-ботов на три категории: Search , Agent и Training . Владельцы сайтов теперь могут независимо разрешать или блокировать поисковую индексацию, обращения агентов от имени пользователей и сбор данных для обучения моделей. Для каждой категории доступны три режима: разрешить , заблокировать везде или заблокировать только на страницах с рекламой . В результате Cloudflare уже не только пропускает трафик и отфильтровывает мусор. Она начинает решать, кто и зачем может читать сайт, а следом собирается брать оплату за проход.
https://habr.com/ru/articles/1058108/
#Cloudflare #ИИботы #вебкраулеры #WAF #CDN #BotBase #robotstxt #Googlebot #Pay_Per_Crawl #монетизация_контента
-
Почему Google не индексирует страницы, хотя технически всё в порядке
У меня есть сайт на Next.js. Часть страниц индексируется почти сразу. Часть застряла в статусе «Обнаружено, не проиндексировано» уже две недели. Самое неприятное в том что все страницы технически одинаковые. Тот же фреймворк, тот же сервер, тот же sitemap. Расскажу, как я перебирал гипотезы одну за другой, и что в итоге осталось. Читать разбор
https://habr.com/ru/articles/1052898/
#google #seo #googlebot #sitemap #sitemapxml #robotstxt #searchconsoler #nextjs #caching
-
Is it now okay to consider GoogleBot hostile and just 403 them away?
-
Is it now okay to consider GoogleBot hostile and just 403 them away?
-
After Google’s announcement that they will start showing AI results rather than links in search results a lot of people showed interest in blocking them from scanning their sites.
If you want to do this rather block their IP ranges than use robots.txt. I recommend blocking all Google Cloud IP ranges as well. I only see malicious bot traffic from there.
Here is a good resource.https://www.searchengineworld.com/googles-full-list-of-googlebot-ip-addresses-spider-ip-addresses
#google #googlebot #googleai #hosting #AI #seo #googlesearch
-
After Google’s announcement that they will start showing AI results rather than links in search results a lot of people showed interest in blocking them from scanning their sites.
If you want to do this rather block their IP ranges than use robots.txt. I recommend blocking all Google Cloud IP ranges as well. I only see malicious bot traffic from there.
Here is a good resource.https://www.searchengineworld.com/googles-full-list-of-googlebot-ip-addresses-spider-ip-addresses
#google #googlebot #googleai #hosting #AI #seo #googlesearch
-
Пять неочевидных вещей, которые я узнал, запуская кино-соцсеть: от robots.txt-ловушки до 24-мерной математики вкуса
Последние полгода я работаю над VibeMuvik — кино-соцсетью с рецензиями, дебатами и синхронным просмотром фильмов. Одна из тех штук, которые «ну вроде несложно», пока не начинаешь копать. Эта статья — про неожиданные находки . Не про «как я выбрал стек» (скучно) и не про «туториал по WebRTC» (и без меня есть). Это пять ситуаций, в которых я споткнулся, обнаружил что-то интересное, и подумал «об этом стоит рассказать — другим пригодится». Поехали.
https://habr.com/ru/articles/1027876/
#robotstxt #SEO #WebRTC #Nextjs #IndexNow #sitemap #Googlebot #Cinema_DNA #синхронный_просмотр #рекомендательные_системы
-
FYI: Google rewrites Googlebot's rulebook: 2MB limits, IP moves, and what crawlers really are: Google today published two blog posts revealing Googlebot's true architecture as a shared SaaS platform, a 2MB fetch limit, and a new IP ranges directory path. https://ppc.land/google-rewrites-googlebots-rulebook-2mb-limits-ip-moves-and-what-crawlers-really-are/ #Googlebot #SEO #WebCrawlers #DigitalMarketing #SaaS
-
FYI: Google rewrites Googlebot's rulebook: 2MB limits, IP moves, and what crawlers really are: Google today published two blog posts revealing Googlebot's true architecture as a shared SaaS platform, a 2MB fetch limit, and a new IP ranges directory path. https://ppc.land/google-rewrites-googlebots-rulebook-2mb-limits-ip-moves-and-what-crawlers-really-are/ #Googlebot #SEO #WebCrawlers #DigitalMarketing #SaaS
-
ICYMI: Google rewrites Googlebot's rulebook: 2MB limits, IP moves, and what crawlers really are: Google today published two blog posts revealing Googlebot's true architecture as a shared SaaS platform, a 2MB fetch limit, and a new IP ranges directory path. https://ppc.land/google-rewrites-googlebots-rulebook-2mb-limits-ip-moves-and-what-crawlers-really-are/ #Google #Googlebot #SEO #Crawlers #WebMaster
-
Inside Googlebot: demystifying crawling, fetching, and the bytes we process: https://developers.google.com/search/blog/2026/03/crawler-blog-post. A great post that clarifies how #Googlebot does its business for people in the #SEO community.
-
Inside Googlebot: demystifying crawling, fetching, and the bytes we process: https://developers.google.com/search/blog/2026/03/crawler-blog-post. A great post that clarifies how #Googlebot does its business for people in the #SEO community.
-
Google rewrites Googlebot's rulebook: 2MB limits, IP moves, and what crawlers really are: Google today published two blog posts revealing Googlebot's true architecture as a shared SaaS platform, a 2MB fetch limit, and a new IP ranges directory path. https://ppc.land/google-rewrites-googlebots-rulebook-2mb-limits-ip-moves-and-what-crawlers-really-are/ #Google #Googlebot #SEO #WebCrawlers #DigitalMarketing
-
Google rewrites Googlebot's rulebook: 2MB limits, IP moves, and what crawlers really are: Google today published two blog posts revealing Googlebot's true architecture as a shared SaaS platform, a 2MB fetch limit, and a new IP ranges directory path. https://ppc.land/google-rewrites-googlebots-rulebook-2mb-limits-ip-moves-and-what-crawlers-really-are/ #Google #Googlebot #SEO #WebCrawlers #DigitalMarketing
-
FYI: Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #DigitalMarketing #SaaS
-
FYI: Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #DigitalMarketing #SaaS
-
ICYMI: Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #Crawling #SaaS #DigitalMarketing
-
ICYMI: Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #Crawling #SaaS #DigitalMarketing
-
Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #SaaS #DigitalMarketing
-
Googlebot is not a program - Google engineers finally explain what it really is: Google engineers reveal Googlebot is a misnomer for a central SaaS crawling platform serving dozens of products, with a 15 MB default file size limit and geo-crawling constraints. https://ppc.land/googlebot-is-not-a-program-google-engineers-finally-explain-what-it-really-is/ #Googlebot #SEO #WebCrawling #SaaS #DigitalMarketing
-
FYI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
FYI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
ICYMI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
ICYMI: Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
Google's secret crawl logic, finally explained in one page: Google published a new web crawling overview on March 3, 2026, detailing how Googlebot discovers, renders, and manages site access across 30+ years of web indexing. https://ppc.land/googles-secret-crawl-logic-finally-explained-in-one-page/ #Google #SEO #WebCrawling #Googlebot #DigitalMarketing
-
#Development #Reports
Google lists Googlebot file limits · Do Google’s crawling limits affect your website? https://ilo.im/16adna_____
#Business #Google #SearchEngine #Crawlers #Googlebot #Files #HTML #PDF #WebDev #Frontend -
#Development #Reports
Google lists Googlebot file limits · Do Google’s crawling limits affect your website? https://ilo.im/16adna_____
#Business #Google #SearchEngine #Crawlers #Googlebot #Files #HTML #PDF #WebDev #Frontend -
Testing tool simulates Google's 2MB HTML limit as SEO professionals assess crawling impact: Dave Smart added 2MB truncation feature to Tame the Bots fetch tool on February 6, enabling technical SEO professionals to simulate Googlebot's reduced file size limits. https://ppc.land/testing-tool-simulates-googles-2mb-html-limit-as-seo-professionals-assess-crawling-impact/ #SEO #GoogleBot #HTML #Crawling #DigitalMarketing
-
Testing tool simulates Google's 2MB HTML limit as SEO professionals assess crawling impact: Dave Smart added 2MB truncation feature to Tame the Bots fetch tool on February 6, enabling technical SEO professionals to simulate Googlebot's reduced file size limits. https://ppc.land/testing-tool-simulates-googles-2mb-html-limit-as-seo-professionals-assess-crawling-impact/ #SEO #GoogleBot #HTML #Crawling #DigitalMarketing