#noindex — Public Fediverse posts

SEO‑поиск в Next.js App Router, индексируемые landing pages из фильтров и URL

Поиск в Next.js App Router связывает несколько механизмов — Server Components, dynamic routes, searchParams , metadata и клиентские фильтры. Если в проекте одна форма и несколько результатов, состояние можно держать в useState . В каталоге или агрегаторе каждый фильтр меняет URL, набор данных и правила индексации. В статье на примере учебного проекта EventMap — агрегатора событий, разберем, как спроектировать систему фильтрации, которая генерирует чистые, индексируемые URL для поисковых роботов и обновляет выдачу при изменении фильтров. Страницы вроде «Концерты в Москве» или «Выставки в Петербурге» могут работать как отдельные landing pages. Для них нужны серверный HTML, постоянный URL, свой title , description , h1 , canonical и отдельная политика index или noindex . Произвольные запросы, сортировки и глубокая пагинация остаются частью поиска и не попадают автоматически в индекс. Поиск построен через App Router, optional catch‑all route, серверный рендер результатов и общий набор функций для URL, metadata, canonical и sitemap. Поиск по каталогу создаёт отдельное состояние для каждой комбинации фильтров. Категория, город, дата, сортировка, поисковая строка и номер страницы попадают в URL. При десяти категориях, пятидесяти городах и пяти вариантах сортировки получается 2500 комбинаций без учёта пагинации и свободного текста. Интерфейс должен обработать все допустимые комбинации. В поисковый индекс обычно попадает только часть из них. Страница категории music , страница города london и сочетание music/london могут содержать устойчивую подборку. Запрос ?q=free+jazz+tonight , сортировка по цене и восьмая страница выдачи относятся к текущему действию пользователя. Фильтры без отдельной политики создают большое пространство URL. Google описывает эту проблему в документации по faceted navigation. Изменение каждого параметра создаёт новый адрес, краулер тратит запросы на многочисленные комбинации и позже обнаруживает, что многие из них не содержат самостоятельного материала. ( Google for Developers )

https://habr.com/ru/articles/1063570/

#nextjs #seo #typescript #react #javascript #app_router #server_components #canonical #sitemap #noindex

@jvnebnunny @site_admin

There was a dude a few weeks ago who vibe coded an account to automatically share posts across fediverse' without the posters permission, using tags. It was suggested that adding #NoIndex to a users bio would prevent this. Not sure if it worked as the account was quickly blocked on the server I'm on, however I suspect this is the same person.

@jvnebnunny @site_admin

There was a dude a few weeks ago who vibe coded an account to automatically share posts across fediverse' without the posters permission, using tags. It was suggested that adding #NoIndex to a users bio would prevent this. Not sure if it worked as the account was quickly blocked on the server I'm on, however I suspect this is the same person.

I put together a list of accounts that are based on the #WestCoast for an early #FollowFriday! Let me know if you want to be added to the list [or removed].

Alissa Azar @alissaazar
Syria📍PDX
Journalist covering police & far-right violence in the PNW • She/They • Antifascist • Neurodivergent

AI6YR Ben @ai6yr
The account of AI6YR Ben, on his own server. Very eclectic selection of posts. Good weather info.

JustBob 🪖🇺🇸♒️️ @bob
The silence is deafening. Always hearing about this country or that one, this group or that one but the silence continues for #homeless people.

Indivisible SF @IndivisibleSF
San Franciscans standing indivisible for a progressive agenda in CA-11 and CA-15.

Led By Gilded Fools @MHowell
White guy living in Indian Country (Suquamish aka Seattle, WA)
#Wilderness, #AntiFacist, #LGBTQ 🏳️‍🌈🏳️‍⚧️ ally, #BLM, #ProChoice, #GunReform #noindex #nobots #noAI

Mungen Cakes ✅️ @Mungencakes
Loose cannon, Fuckstar, the antifa Ray Stevens, #AltText evangelist. Unceded Alsea land.

The Needling @TheNeedling
Seattle's Only Real Fake News (Satire/Parody)

Pete's Mom 🍉 🇺🇦 ☮️ @Petesmom
ANTIFA, ACAB / FUCK ICE
Black Lives Matter! Immigrant Lives Matter! Trans Lives Matter! I believe in DEI! “In diversity there is beauty and there is strength.” - Maya Angelou

Rachael L @rachael
I live in Seattle with my family and work in tech. I'm also disabled (mostly deaf & blind on one side). I ride the bus, walk or bike most of the time and I care a lot about our transportation system working for everyone. This account is for Seattle, urbanism & regional political stuff!

San Francisco Baykeeper @sfbaykeeper
Defending #SFBay & holding polluters accountable since 1989

Seattle Bike Blog @seabikeblog
Pedaling news up Seattle’s hills since 2010.

Youth Liberation Front Seattle @seattleylf
The Seattle Division of the PNWYLF, a collective in an autonomous network of antifascist students and youth [Note: They have not posted in a while.]

#FollowerPower #PNW #California #Oregon #WashingtonState #SeattleWA #SanFrancisco #PortlandOR #PacificCoast

I put together a list of accounts that are based on the #WestCoast for an early #FollowFriday! Let me know if you want to be added to the list [or removed].

Alissa Azar @alissaazar
Syria📍PDX
Journalist covering police & far-right violence in the PNW • She/They • Antifascist • Neurodivergent

AI6YR Ben @ai6yr
The account of AI6YR Ben, on his own server. Very eclectic selection of posts. Good weather info.

JustBob 🪖🇺🇸♒️️ @bob
The silence is deafening. Always hearing about this country or that one, this group or that one but the silence continues for #homeless people.

Indivisible SF @IndivisibleSF
San Franciscans standing indivisible for a progressive agenda in CA-11 and CA-15.

Led By Gilded Fools @MHowell
White guy living in Indian Country (Suquamish aka Seattle, WA)
#Wilderness, #AntiFacist, #LGBTQ 🏳️‍🌈🏳️‍⚧️ ally, #BLM, #ProChoice, #GunReform #noindex #nobots #noAI

Mungen Cakes ✅️ @Mungencakes
Loose cannon, Fuckstar, the antifa Ray Stevens, #AltText evangelist. Unceded Alsea land.

The Needling @TheNeedling
Seattle's Only Real Fake News (Satire/Parody)

Pete's Mom 🍉 🇺🇦 ☮️ @Petesmom
ANTIFA, ACAB / FUCK ICE
Black Lives Matter! Immigrant Lives Matter! Trans Lives Matter! I believe in DEI! “In diversity there is beauty and there is strength.” - Maya Angelou

Rachael L @rachael
I live in Seattle with my family and work in tech. I'm also disabled (mostly deaf & blind on one side). I ride the bus, walk or bike most of the time and I care a lot about our transportation system working for everyone. This account is for Seattle, urbanism & regional political stuff!

San Francisco Baykeeper @sfbaykeeper
Defending #SFBay & holding polluters accountable since 1989

Seattle Bike Blog @seabikeblog
Pedaling news up Seattle’s hills since 2010.

Youth Liberation Front Seattle @seattleylf
The Seattle Division of the PNWYLF, a collective in an autonomous network of antifascist students and youth [Note: They have not posted in a while.]

#FollowerPower #PNW #California #Oregon #WashingtonState #SeattleWA #SanFrancisco #PortlandOR #PacificCoast

Même sujet, autre point de vue.

« Quand une civilisation n’est plus que cendres et poussières, déclara-t-il, il ne reste plus que l’art. Des images, des mots, de la musique. Des structures imaginatives. C’est ça qui fait sens, pour les hommes, s’entend. »

ATWOOD, Margaret - Le dernier homme

#Art, #dessin, #aquarelle, #watercolor, #lineart, #urbansketcher, #happy #nobot #noindex

Voici le vieux tracteur à péniches près de la cantine des italiens.

«Aucune route frayée ne sillonait le voisinage, et, pour atteindre notre heureuse retraite, il fallait repousser le feuillage de milliers d’arbres forestiers et anéantir la gloire de milliers de fleurs parfumées.»

ALLAN POE, Edgar - Histoires grotesques et sérieuses -Eléonora

#Art, #dessin, #aquarelle, #watercolor, #lineart, #urbansketcher, #LaLouvière, #régionducentre, #Hainaut, #belgique, #happy #nobot #noindex

Maison près de l'ascenseur N°1, La Louvière.

" Cette cité sans pittoresque, sans végétation et sans âme finit par sembler reposante, on s’y endort enfin."
Camus, Albert - La Peste

#Art, #dessin, #aquarelle, #watercolor, #lineart, #urbansketcher, #LaLouvière, #régionducentre, #Hainaut, #belgique, #happy #nobot #noindex

Maison près de l'ascenseur N°1, La Louvière.

" Cette cité sans pittoresque, sans végétation et sans âme finit par sembler reposante, on s’y endort enfin."
Camus, Albert - La Peste

#Art, #dessin, #aquarelle, #watercolor, #lineart, #urbansketcher, #LaLouvière, #régionducentre, #Hainaut, #belgique, #happy #nobot #noindex

Ascenseur N°1 à La Louvière

«Les cendres du monde défunt emportées çà et là dans le vide sur les vents froids et profanes. Emportées au loin et dispersées et emportées encore plus loin. Toute chose coupée de son fondement. Soutenue par un souffle, tremblante et brève. Si seulement mon coeur était de pierre.»
McCARTHY, Cormack - La Route

#Art, #dessin, #aquarelle, #watercolor, #lineart, #urbansketcher, #LaLouvière, #régionducentre, #Hainaut, #belgique, #happy #nobot #noindex

Ascenseur N°1 à La Louvière

«Les cendres du monde défunt emportées çà et là dans le vide sur les vents froids et profanes. Emportées au loin et dispersées et emportées encore plus loin. Toute chose coupée de son fondement. Soutenue par un souffle, tremblante et brève. Si seulement mon coeur était de pierre.»
McCARTHY, Cormack - La Route

#Art, #dessin, #aquarelle, #watercolor, #lineart, #urbansketcher, #LaLouvière, #régionducentre, #Hainaut, #belgique, #happy #nobot #noindex

#HolosDiscover currently blocks #Bluesky accounts bridged via #BridgyFed, because AT Protocol has no equivalent to the indexable flag.
The bridge is opt-in, but bridged users cannot signal refusal of third-party indexing.
Standard opt-out mechanisms remain functional: block @HolosDiscover, #nobot or #noindex in bio, mention with "unfollow".

What's your opinion?

#HolosDiscover currently blocks #Bluesky accounts bridged via #BridgyFed, because AT Protocol has no equivalent to the indexable flag.
The bridge is opt-in, but bridged users cannot signal refusal of third-party indexing.
Standard opt-out mechanisms remain functional: block @HolosDiscover, #nobot or #noindex in bio, mention with "unfollow".

What's your opinion?

@evan more profile tags found in the wild..

Updated #Botiquette issue on Codeberg in #fediverse ideas repo to track this..

https://codeberg.org/fediverse/fediverse-ideas/issues/33#issuecomment-13221306

#nobot #noindex #noarchive #nosearch #nobridge #noai

@evan more profile tags found in the wild..

Updated #Botiquette issue on Codeberg in #fediverse ideas repo to track this..

https://codeberg.org/fediverse/fediverse-ideas/issues/33#issuecomment-13221306

#nobot #noindex #noarchive #nosearch #nobridge #noai

Should users have to opt-out with #NoBot #NoIndex #NoAI or can we have nice things?

#mastoadmin #moderation

Should users have to opt-out with #NoBot #NoIndex #NoAI or can we have nice things?

#mastoadmin #moderation

#PSA I plan to sign into https://Zeitgeist.blue on Saturday 4 April to see how it works. This is a new client that uses "bring your own LLM key" as an approach to distill/curate Fediverse timelines. If you want to opt out of any of your posts going through personal LLM inference, please unfollow me and block me by then.

**Update**: you can also add #nobots, #noindex or #noai to your profile and Zeigeist will ignore your posts: https://github.com/seldo/zeitgeist/commit/90b09ad6f683d7059ef4a5c4a3883704a7e71057. I hope you do this instead of blocking me, but both should work :)

I am going to use an Anthropic API key, and my understanding is that your posts would *not* end up in a corpus used for training after I do this; this is inference only, and in principle I will be the only one seeing the results.

If you have any concerns or need an extension for some reason, do reach out! I can also then let you know once I stop using the tool so you can optionally remove the block.

Context: https://www.loomio.com/d/ASGoq9TQ/comment/3334596

#PSA I plan to sign into https://Zeitgeist.blue on Saturday 4 April to see how it works. This is a new client that uses "bring your own LLM key" as an approach to distill/curate Fediverse timelines. If you want to opt out of any of your posts going through personal LLM inference, please unfollow me and block me by then.

**Update**: you can also add #nobots, #noindex or #noai to your profile and Zeigeist will ignore your posts: https://github.com/seldo/zeitgeist/commit/90b09ad6f683d7059ef4a5c4a3883704a7e71057. I hope you do this instead of blocking me, but both should work :)

I am going to use an Anthropic API key, and my understanding is that your posts would *not* end up in a corpus used for training after I do this; this is inference only, and in principle I will be the only one seeing the results.

If you have any concerns or need an extension for some reason, do reach out! I can also then let you know once I stop using the tool so you can optionally remove the block.

Context: https://www.loomio.com/d/ASGoq9TQ/comment/3334596

@seldo they were not honouring #noindex to begin with.

@seldo they were not honouring #noindex to begin with.

@chad For what it is worth, the dev of that AI timeline tool says it now supports several ways to stop it.

Zeitgeist now:
* respects the `noindex` preference if set on your account
* respects #noindex or #nobots if those are in your bio
* sends a unique user-agent ("Zeitgeist/1.0") if you want to block the client entirely.

Edit to add: It also says it respects the #noai tag in profile bios as of yesterday:
https://github.com/seldo/zeitgeist/commit/90b09ad6f683d7059ef4a5c4a3883704a7e71057

@maddad @mastodonmigration

@chad For what it is worth, the dev of that AI timeline tool says it now supports several ways to stop it.

Zeitgeist now:
* respects the `noindex` preference if set on your account
* respects #noindex or #nobots if those are in your bio
* sends a unique user-agent ("Zeitgeist/1.0") if you want to block the client entirely.

Edit to add: It also says it respects the #noai tag in profile bios as of yesterday:
https://github.com/seldo/zeitgeist/commit/90b09ad6f683d7059ef4a5c4a3883704a7e71057

@maddad @mastodonmigration

Zeitgeist now:
* respects the `noindex` preference if set on your account
* respects #noindex or #nobots if those are in your bio
* sends a unique user-agent ("Zeitgeist/1.0") if you want to block the client entirely.

Zeitgeist now:
* respects the `noindex` preference if set on your account
* respects #noindex or #nobots if those are in your bio
* sends a unique user-agent ("Zeitgeist/1.0") if you want to block the client entirely.

@seldo @janl I sure am a tedious bastard. And proud of it.

Anyway, is the concept of Zeitgeist (compliments on the apt name, right on the nose) some clob of code spoonfeeding you a plausible bullshit summary of what's been happening using a personal LLM? No harm, no foul. At least not to me.

You're an adult, and solely responsible for any dissociation and brainrot caused.

OR! BUT! WAIT!
Is Zeitgeist going to consume the feed of everyone who subscribes and thus give Anthropic or Copilot access to content of the rest of us by unwitting proxies? Including people who post Follower Only, or even Private Mentions?

[takes look at the code]

Yes, this is giving Anthropic/ Copilot access as if a user:

https://github.com/seldo/zeitgeist/blob/main/app/api/mastodon/auth/route.ts

I see no filtering out of Private Mentions and Follower Only posts:

https://github.com/seldo/zeitgeist/blob/main/app/api/mastodon/timeline/route.ts

And yes, you'll shove it right into Anthropic/ Copilot:

https://github.com/seldo/zeitgeist/blob/main/app/api/summarize/route.ts

For the "tedious" part: I cannot find evidence you gave thought to what impact your fun idea might have on others. You also weren't aware of the #noindex, #notbots tags, or why people might want those to be respected.

I know, I know, no fun, right? You probably mean no harm. But what you want to do, and what you're actually doing, are not always the same thing. Especially not if you don't read the room you just barged into.

For the "bastard" part: I'll refrain from ending this with a bunch of prompt injections in my status, aka your postsText. :flan_wink:

@seldo @janl I sure am a tedious bastard. And proud of it.

Anyway, is the concept of Zeitgeist (compliments on the apt name, right on the nose) some clob of code spoonfeeding you a plausible bullshit summary of what's been happening using a personal LLM? No harm, no foul. At least not to me.

You're an adult, and solely responsible for any dissociation and brainrot caused.

OR! BUT! WAIT!
Is Zeitgeist going to consume the feed of everyone who subscribes and thus give Anthropic or Copilot access to content of the rest of us by unwitting proxies? Including people who post Follower Only, or even Private Mentions?

[takes look at the code]

Yes, this is giving Anthropic/ Copilot access as if a user:

https://github.com/seldo/zeitgeist/blob/main/app/api/mastodon/auth/route.ts

I see no filtering out of Private Mentions and Follower Only posts:

https://github.com/seldo/zeitgeist/blob/main/app/api/mastodon/timeline/route.ts

And yes, you'll shove it right into Anthropic/ Copilot:

https://github.com/seldo/zeitgeist/blob/main/app/api/summarize/route.ts

For the "tedious" part: I cannot find evidence you gave thought to what impact your fun idea might have on others. You also weren't aware of the #noindex, #notbots tags, or why people might want those to be respected.

I know, I know, no fun, right? You probably mean no harm. But what you want to do, and what you're actually doing, are not always the same thing. Especially not if you don't read the room you just barged into.

For the "bastard" part: I'll refrain from ending this with a bunch of prompt injections in my status, aka your postsText. :flan_wink:

Why do I live in a world where I have to put this shit in my bio:

#nobot #noindex #noarchive #nosearch #nobridge #noai

in my profile, and pray that the techbro's bullshit bot respects it? Whatever happened to asking people? 🙄

Why do I live in a world where I have to put this shit in my bio:

#nobot #noindex #noarchive #nosearch #nobridge #noai

in my profile, and pray that the techbro's bullshit bot respects it? Whatever happened to asking people? 🙄

@CoolerPseudonym @apps https://discover.holos.social/how-it-works says it all

so basically if you have any of these things set, then you're opted out.
So if EITHER
- you have 'indexable' turned off
- have follow approval required on
- have #nobot or #noindex in your bio
then you're opted out from holos

@CoolerPseudonym @apps https://discover.holos.social/how-it-works says it all

so basically if you have any of these things set, then you're opted out.
So if EITHER
- you have 'indexable' turned off
- have follow approval required on
- have #nobot or #noindex in your bio
then you're opted out from holos

#HolosDiscover is up to shenanigans again. Rather than making their scraper / indexer truly opt-in, they're again using @[email protected] to follow accounts that don't have search engine indexing turned off and don't have #NoBot and #NoIndex in their profiles.

I recommend blocking @[email protected] at minimum.

As a sidebar, I would support the existence of an actual opt-in index for reasons I have discussed previously. This is not that.

#Fediblock

#HolosDiscover is up to shenanigans again. Rather than making their scraper / indexer truly opt-in, they're again using @[email protected] to follow accounts that don't have search engine indexing turned off and don't have #NoBot and #NoIndex in their profiles.

I recommend blocking @[email protected] at minimum.

As a sidebar, I would support the existence of an actual opt-in index for reasons I have discussed previously. This is not that.

#Fediblock

@nodebb
I'm trying to figure out why my posts are being repeated on your platform, despite my account having #nobot #noindex, etc.

Care to comment on why you don't appear to respect people's requests for privacy?

@nodebb
I'm trying to figure out why my posts are being repeated on your platform, despite my account having #nobot #noindex, etc.

Care to comment on why you don't appear to respect people's requests for privacy?

JSON Column Search Takes 47 SECONDS Breaking Black Friday?!

JSON SEARCH NIGHTMARE! Can't index JSON columns in older MySQL! Every product search scans 2 MILLION rows! 47-second queries! Server MELTS on Black Friday! $4.7M revenue LOST! Watch the horror!

#sql #sqldisaster #json #performance #noindex #slowquery #sqlfails #blackfriday #sqlshorts #databasedisaster #sqlwtf #jsonsearch

https://www.youtube.com/watch?v=O0jCjaAsNNA

JSON Column Search Takes 47 SECONDS Breaking Black Friday?!

JSON SEARCH NIGHTMARE! Can't index JSON columns in older MySQL! Every product search scans 2 MILLION rows! 47-second queries! Server MELTS on Black Friday! $4.7M revenue LOST! Watch the horror!

#sql #sqldisaster #json #performance #noindex #slowquery #sqlfails #blackfriday #sqlshorts #databasedisaster #sqlwtf #jsonsearch

https://www.youtube.com/watch?v=O0jCjaAsNNA

#HolosDiscover is back, from a clean slate.
A #Fediverse search engine that uses standard #ActivityPub federation. It follows you like any account, respects indexable flags, #nobot, #noindex, locked accounts. Deletions, edits, blocks are processed instantly through ActivityPub.
You have full control. Block it, mention it with "unfollow", or disable indexing in your settings.
Source code under AGPL-3.0 on #Codeberg.

Details: https://discover.holos.social/how-it-works

Account: @HolosDiscover

#HolosDiscover is back, from a clean slate.
A #Fediverse search engine that uses standard #ActivityPub federation. It follows you like any account, respects indexable flags, #nobot, #noindex, locked accounts. Deletions, edits, blocks are processed instantly through ActivityPub.
You have full control. Block it, mention it with "unfollow", or disable indexing in your settings.
Source code under AGPL-3.0 on #Codeberg.

Details: https://discover.holos.social/how-it-works

Account: @HolosDiscover

@apps

So this dead horse flinched again

"With #HolosDiscover we checked multiple criteria before indexing: "indexable" enabled, account not locked, no #nobot or #noindex in bio, not in opted-out list, only public posts."

I'll copy-pasta with only slight edits:

My entire point (all the noise notwithstanding) focused on

Default opt-in versus default opt-out

This is an agent --> recipient transaction

Default opt-in: the recipient is opted into (and receives) the agent's action whether the recipient --> knows of <-- the action or not

Default opt-out: the recipient is opted out of (and cannot receive) the action whether the recipient --> knows of <-- the action or not

Neither default opt-in nor default opt-out have any logical meaning if

--> THE RECIPIENT DOES NOT KNOW OF THE AGENT <--

in advance

There was no mechanism for prior notification *before* indexing

People would have had to stumble on what you're doing, by how, exactly?

How dose the recipient learn of what you've done *before* you do it?

@apps

So this dead horse flinched again

"With #HolosDiscover we checked multiple criteria before indexing: "indexable" enabled, account not locked, no #nobot or #noindex in bio, not in opted-out list, only public posts."

I'll copy-pasta with only slight edits:

My entire point (all the noise notwithstanding) focused on

Default opt-in versus default opt-out

This is an agent --> recipient transaction

Default opt-in: the recipient is opted into (and receives) the agent's action whether the recipient --> knows of <-- the action or not

Default opt-out: the recipient is opted out of (and cannot receive) the action whether the recipient --> knows of <-- the action or not

Neither default opt-in nor default opt-out have any logical meaning if

--> THE RECIPIENT DOES NOT KNOW OF THE AGENT <--

in advance

There was no mechanism for prior notification *before* indexing

People would have had to stumble on what you're doing, by how, exactly?

How dose the recipient learn of what you've done *before* you do it?

With #HolosDiscover we checked multiple criteria before indexing: "indexable" enabled, account not locked, no #nobot or #noindex in bio, not in opted-out list, only public posts. Every deletion, edit or block was processed instantly via #ActivityPub.
Google uses that same "indexable" flag but ignores everything else, keeps deleted content cached for weeks.
We shut it down after pushback. Was that the right call? Don't hesitate to share, this concerns the whole Fediverse.

With #HolosDiscover we checked multiple criteria before indexing: "indexable" enabled, account not locked, no #nobot or #noindex in bio, not in opted-out list, only public posts. Every deletion, edit or block was processed instantly via #ActivityPub.
Google uses that same "indexable" flag but ignores everything else, keeps deleted content cached for weeks.
We shut it down after pushback. Was that the right call? Don't hesitate to share, this concerns the whole Fediverse.

@apps

[here we go again]

"Only public posts from consenting users are indexed. We respect every signal available."

But there is the entire problem in a nutshell:

What about people who have never heard of Holos Discovery and who never will unless they receive a Follow request?

Allegedly only consenting users are indexed

I have #NoIndex #Nobot set on my profile and I have to approve Follows

But during the convo I had earlier that person said:

"Your content has been removed from our index and you won't be contacted again."

To which I replied:

"So you've --> already <-- "scraped my content" without my knowledge or permission..."

I then asked about second-party scraping -- someone I Follow and interact with has been "Discovered" and Indexed -- and thus my content has been indexed too, without my knowledge or consent

At that, they never returned to answer

The core point is, historically a lot of people have not wanted to found, scraped, and indexed for unknown third-party use

Here we go again...