#fulltextsearch — Public Fediverse posts

I have noticed that a while ago, several new Postgres extensions regarding text search have surfaced that all use BM25. Obviously if you read their documentation each one is the ultimate solution to all text search problems.

Is anyone aware of a comparison between the various extensions?

I would be mainly interested in a feature comparison, articles that point out the sweet spots and the pitfalls.

#PostgreSQL #FullTextSearch #BM25

I have noticed that a while ago, several new Postgres extensions regarding text search have surfaced that all use BM25. Obviously if you read their documentation each one is the ultimate solution to all text search problems.

Is anyone aware of a comparison between the various extensions?

I would be mainly interested in a feature comparison, articles that point out the sweet spots and the pitfalls.

#PostgreSQL #FullTextSearch #BM25

A few weeks ago, we migrated our website from WordPress to Hugo: 20 days, 360 commits, and roughly 1,580 redirect rules. You can read the full story in the corresponding blog post. And yes, there was one little thing missing since the migration. 🙈 A full-text search feature... but we have that now, too. 🚀 https://nine.ch/en/blog/from-wordpress-to-hugo/ #website #fulltextsearch #feature #swisshosting #nine

A few weeks ago, we migrated our website from WordPress to Hugo: 20 days, 360 commits, and roughly 1,580 redirect rules. You can read the full story in the corresponding blog post. And yes, there was one little thing missing since the migration. 🙈 A full-text search feature... but we have that now, too. 🚀 https://nine.ch/en/blog/from-wordpress-to-hugo/ #website #fulltextsearch #feature #swisshosting #nine

Oh look, another "groundbreaking" #blog post about #DuckDB from a self-proclaimed data wizard. 🙄 Apparently, the limitations of basic text queries are just too much for our hero, who bravely delves into the wild world of Full-Text Search. 🌟 Spoiler alert: it's as thrilling as watching paint dry on a data frame. 🥱
https://peterdohertys.website/blog-posts/full-text-search-w-duckdb.html #DataWizard #FullTextSearch #DataFrame #HackerNews #ngated

Oh look, another "groundbreaking" #blog post about #DuckDB from a self-proclaimed data wizard. 🙄 Apparently, the limitations of basic text queries are just too much for our hero, who bravely delves into the wild world of Full-Text Search. 🌟 Spoiler alert: it's as thrilling as watching paint dry on a data frame. 🥱
https://peterdohertys.website/blog-posts/full-text-search-w-duckdb.html #DataWizard #FullTextSearch #DataFrame #HackerNews #ngated

Full-Text Search with DuckDB

https://peterdohertys.website/blog-posts/full-text-search-w-duckdb.html

#HackerNews #FullTextSearch #DuckDB #Database #Technology #DataAnalysis #SearchEngine

Full-Text Search with DuckDB

https://peterdohertys.website/blog-posts/full-text-search-w-duckdb.html

#HackerNews #FullTextSearch #DuckDB #Database #Technology #DataAnalysis #SearchEngine

I had a chat with the #Fold3 gentleman (they were wearing fantastic shirts!). American Rev War is their first to use AI. They are happy with the results and will be moving on to other wars.

I find Fold3 incredibly useful but extremely "Clunky" to use. This should be a great improvement to be able to read some records better.

#RootsTech #Genealogy #FamilyHistory #AI #FullTextSearch #AmericanRevolution

I had a chat with the #Fold3 gentleman (they were wearing fantastic shirts!). American Rev War is their first to use AI. They are happy with the results and will be moving on to other wars.

I find Fold3 incredibly useful but extremely "Clunky" to use. This should be a great improvement to be able to read some records better.

#RootsTech #Genealogy #FamilyHistory #AI #FullTextSearch #AmericanRevolution

@geneapleau I too have had some great success with the full text search. Particularly with #wills, finding those executors and beneficiaries you might never have otherwise located. #familysearch #fulltextsearch #genealogy @genealogy

@geneapleau I too have had some great success with the full text search. Particularly with #wills, finding those executors and beneficiaries you might never have otherwise located. #familysearch #fulltextsearch #genealogy @genealogy

💡 For the first time I just submitted a proposal for a new (small) feature in #django ; it's related to #fulltextsearch in #postgresql , especially for non-English websites/webapps . You can use Github reactions to let me know what you think about it 😉

https://github.com/django/new-features/issues/107

💡 For the first time I just submitted a proposal for a new (small) feature in #django ; it's related to #fulltextsearch in #postgresql , especially for non-English websites/webapps . You can use Github reactions to let me know what you think about it 😉

https://github.com/django/new-features/issues/107

Die neuste Version der #Nextcloud App zur #Volltextsuche (31.0.1 bzw. 32.0.1) ist leider nicht mehr kompatibel zu #Elasticsearch 8.
Der folgende Artikel zeigt daher, wie ihr ein Upgrade auf #Elasticsearch9 durchführen könnt, damit die Volltextsuche in eurer Nextcloud wieder einwandfrei funktioniert: https://decatec.de/home-server/nextcloud-volltextsuche-upgrade-auf-elasticsearch-9/

#HomServer #Search #Cloud #Fulltextsearch

Die neuste Version der #Nextcloud App zur #Volltextsuche (31.0.1 bzw. 32.0.1) ist leider nicht mehr kompatibel zu #Elasticsearch 8.
Der folgende Artikel zeigt daher, wie ihr ein Upgrade auf #Elasticsearch9 durchführen könnt, damit die Volltextsuche in eurer Nextcloud wieder einwandfrei funktioniert: https://decatec.de/home-server/nextcloud-volltextsuche-upgrade-auf-elasticsearch-9/

#HomServer #Search #Cloud #Fulltextsearch

Just shipped a new hands-on tutorial:
Native Full-Text Search in Quarkus with PostgreSQL — no Elasticsearch, no extra infra.

We build:
• tsvector + tsquery search
• Ranking with ts_rank
• Snippets with ts_headline
• A clean Qute front end

If you want fast, simple, reliable search for docs or blogs, Postgres already has everything you need.

Read it here:
https://www.the-main-thread.com/p/quarkus-postgresql-full-text-search-tutorial

#java #quarkus #postgres #fulltextsearch

Just shipped a new hands-on tutorial:
Native Full-Text Search in Quarkus with PostgreSQL — no Elasticsearch, no extra infra.

We build:
• tsvector + tsquery search
• Ranking with ts_rank
• Snippets with ts_headline
• A clean Qute front end

If you want fast, simple, reliable search for docs or blogs, Postgres already has everything you need.

Read it here:
https://www.the-main-thread.com/p/quarkus-postgresql-full-text-search-tutorial

#java #quarkus #postgres #fulltextsearch

Cải thiện tốc độ tìm kiếm email lên đến 71% với PostgreSQL Full-Text Search
Tìm kiếm email nhanh hơn 59-71% với tsvector và GIN index
Cải thiện hiệu suất tìm kiếm với trọng số và xếp hạng liên quan
#PostgreSQL #FullTextSearch #GINIndex #tsvector #TìmKiếmEmail #HiệuSuất #CảiThiện #Database #Optimization #Vietnamese #English #Tech #Technology #Programming #Developer #Software #EmailSearch #SearchEngine #DatabaseDesign

https://dev.to/_877737de2d34ff8c6265/how-we-achieved-up-to-71-faster-email-sear

I wrote a full text search engine in Go

https://github.com/wizenheimer/blaze

#HackerNews #I #wrote #a #full #text #search #engine #in #Go #GoLang #FullTextSearch #OpenSource #SoftwareDevelopment #HackerNews

I wrote a full text search engine in Go

https://github.com/wizenheimer/blaze

#HackerNews #I #wrote #a #full #text #search #engine #in #Go #GoLang #FullTextSearch #OpenSource #SoftwareDevelopment #HackerNews

The #FullTextSearch may be a bit broken now, I'm going to switch that over to our new instance as well so it may take a little bit for it to re-index all of the content that should be in the search engine.

The #FullTextSearch may be a bit broken now, I'm going to switch that over to our new instance as well so it may take a little bit for it to re-index all of the content that should be in the search engine.

New feature.

I've added #fulltextsearch to beyng.com using #PageFind. I'm excited because Google/Bing/Yandex only index 10% of the ~40,000 pages–philosophy texts don't contain the keywords advertisers pay for. Now it's possible to search them all.

Features I'm hoping for in future releases of PageFind are open-results-in-new-tab and include og:image in results.

It takes half-a-day to FTP upload the ~180 megabytes of index files, so I won't be updating indexes often.

https://www.beyng.com/hb/pagefind.html

1. advanced search operators prototype. status: not quite ready for prime time.
   • has a bunch of goofy operators nobody but me will ever use, such as is:article
   • still missing some classics like lang:, domain:, before:, and after:, and some oddballs like is:bot (would require extra join) and sort: (would break ID-based paging)
   • needs docs, although i know where Past Vyr basically already wrote them: https://github.com/VyrCossont/mastodon/pull/8 😇
2. indexed full text search prototype. status: heretical.
   • only works on PostgreSQL: SQLite's full-text search is much fussier and requires using a "virtual table" and frankly i can't be bothered, at least tonight
   • direct port of https://github.com/VyrCossont/mastodon/pull/3 and has the same limitations: HTML isn't stripped, and media alt text and poll options aren't indexed
   • fixing that would start by adding a tsvector column that concatenates (with record separators? as an array?) the contents of filterableFields for a status, updates it every time the status or its attachments are edited, and GIN-indexes that column
   • ignores the whole issue of matching posts to language tags and language tags to PG text search configurations by assuming that everything is English
   • still massively faster than unindexed ILIKE that vanilla GTS uses

edit: fixed a backwards flag in has:media and related operators

#GoToSocial #GTS #FullText #FullTextSearch

ok, here you go, updated GTS search patches for 0.18.0rc1. notice how they're on my repo? these are completely unofficial. do not bug anyone but me about them.

1. improved hashtag search. status: upstreamable, mostly.
   • doesn't require # prefix to search hashtags
   • searches for matches anywhere in a hashtag: Mac now matches VintageMac as well as MacOS
   • includes hashtags when not specifically searching for accounts or statuses, like most Mastodon-compatibles
   • doesn't change existing tag sorting. popularity and/or recency might be more useful
2. offset paging for searches. status: not upstreamable yet.
   • more compatible: many clients can't do ID paging
   • allows paging hashtag search results: Mastodon API has no concept of IDs for hashtags, so ID paging can't work for those anyway
   • possible performance issues: see comments on why main doesn't have it already. personally, i haven't noticed and i run this instance on a tiny VPS
3. remove search restrictions. status: heretical.
   • searches any post on your instance (except other accounts' private/direct posts, and accounts that have you blocked)
   • includes public, unlisted, your own private and DM posts, and private and DM posts that are replies to you
   • expanded search is default: revert to standard GTS behavior by adding scope:classic or in:library operator to search query
   • definite performance issues: this means searching more posts! GTS does not use either PG full-text indexes/operators or SQLite full-text virtual tables, and this patch doesn't change that.
   • doesn't include alt text of media attachments, or polls, because main doesn't

i may add more patches to this list in the medium future as i add more functionality to my own instance, for example, date range operators (before:date, after:date), post property operators (has:image,has:poll, has:cw, is:sensitive, visibility:public), threading operators (to:[email protected], is:reply, -is:reply), sort operators (sort:oldest, sort:newest, sort:favs) and maybe PG full-text indexing if i have a really good day (i really don't wanna figure out SQLite's weird shit! someone else do it!)

randos don't debate me about Fedi search. my clients can't set per-post interaction controls yet so i'll just block you.

#GoToSocial #GTS #FullText #FullTextSearch

It looks like the #Universeodon relay has gone down due to an issue on the host it's running on. Apologies other #MastoAdmin 's out there. It's rebooting now and I need to do some digging to see why the host restarted in the first place. Apologies for any disruption.

It's also taken down our #FullTextSearch on #MastodonAppUK and some other core services on MastodonAppUK so I'm looking into this all now.

@sylv_a Once people come to Mastodon, how do we encourage them to opt into #FullTextSearch? One reason why I still occasionally use X since moving to PMG in fourth quarter 2022 is that X still lets me #search for what others are saying about a particular topic, even if I fail to #GuessTheHashtag.

@sylv_a Once people come to Mastodon, how do we encourage them to opt into #FullTextSearch? One reason why I still occasionally use X since moving to PMG in fourth quarter 2022 is that X still lets me #search for what others are saying about a particular topic, even if I fail to #GuessTheHashtag.

I just came across this interesting project - an #ElasticSearch #OpenSearch replacement written in #rust. RUST! This has got to be an order of magnitude more memory efficient than it's #java counterparts.

If so, this could be a game changer for small instances. Opensearch is by FAR the worst - greediest - and most finicky process in my rack.

Is anyone on the #fediverse using this already? I'd love to hear your thoughts!

https://github.com/quickwit-oss/quickwit

#FullTextSearch #NoMoreJavaOnServers

I just came across this interesting project - an #ElasticSearch #OpenSearch replacement written in #rust. RUST! This has got to be an order of magnitude more memory efficient than it's #java counterparts.

If so, this could be a game changer for small instances. Opensearch is by FAR the worst - greediest - and most finicky process in my rack.

Is anyone on the #fediverse using this already? I'd love to hear your thoughts!

https://github.com/quickwit-oss/quickwit

#FullTextSearch #NoMoreJavaOnServers

@[email protected] Worse is that a lot of this is self-inflicted (#fulltextsearch becoming "opt-in" in #Mastodon which is ignored in other #fediverse software like #Misskey which indexes everyone's posts anyway), and the #ActivityPub spec has specifically dealt with this issue which is to recursively forward every remote post an instance receive to every other remote instances they know of. This doesn't appear to be done for as:Public in the cc target. ​:sagume_think:​

@[email protected] Worse is that a lot of this is self-inflicted (#fulltextsearch becoming "opt-in" in #Mastodon which is ignored in other #fediverse software like #Misskey which indexes everyone's posts anyway), and the #ActivityPub spec has specifically dealt with this issue which is to recursively forward every remote post an instance receive to every other remote instances they know of. This doesn't appear to be done for as:Public in the cc target. ​:sagume_think:​

Что ищет он в краю далёком? Как найти смысл жизни с PostgreSQL

Эта статья родилась из пары лекций, которые я прочитал студентам в рамках курса, посвященного вопросам машинного обучения. Почему именно PostgreSQL? Почему векторы? За последние два года тема языковых моделей стала невероятно популярной, и вместе с этим появилось множество инструментов, доступных даже начинающему инженеру, стремящемуся познакомиться с миром текстового анализа. Доступность этих технологий открывает безграничные возможности для их применения в самых разных областях: от систем управления знаниями до «копилотов», помогающих более тщательно анализировать анамнез пациентов, или информационных киосков, позволяющих собрать идеальную корзину товаров для пикника. Вряд ли данная работа может похвастаться полнотой или глубиной, однако, я надеюсь, что она предоставит те самые “хорошие” точки входа, которые позволят, погружаясь в детали, открыть для себя множество новых интересных и полезных тем для исследований и инженерных проектов. Откроем скрытые смыслы

https://habr.com/ru/articles/855712/

#postgresql #postgres #pgvector #vectorization #fulltextsearch #fulltext_search #hnsw #python #java #Knowledge_Management_Systems

Быстрее пули: как найти счастье с PostgreSQL

В этой статье мы расскажем о том, как эффективно реализовать полнотекстовый поиск с помощью PostgreSQL. Узнайте, как улучшить скорость и точность поиска по текстовым данным, используя такие инструменты, как tsvector , tsquery и индексы GIN , и как эти возможности могут значительно повысить производительность вашего приложения.

https://habr.com/ru/articles/853124/

#fulltextsearch #полнотекстовый_поиск #postgresql #gin #базы_данных #базы_данных #индекс #индексация

Is there such a thing as using too many #hashtags in #Mastodon or in the #fediverse in general? I guess if you are doing it on every word that might be considered #spam or could violate your instance's rules. Perhaps I am worried over #netiquette too much, but as long as #FullTextSearch is not the default in Mastodon, I will tend toward using more hashtags than less.

Unless you #selfhost your own #mastodon instance, or you're a #mastoAdmin, you have likely taken #fullTextSearch for granted.

Behind search for masto is a resource-hungry #java application called #elasticSearch. Sure you might be able to get Masto to run on a #raspberrypi4, but you won't be running elastic search with it.

Having recently moved my instance to a much more powerful system, I now run ES, and WOW what a difference being able to search post makes! Soooo many times I wanted to reference a toot that had scrolled by but had no way of finding it. Now I do!

Even though I've put some serious resource limits on ES, it's bar none the most resource intensive service running (out of around 25).

Please consider sending a few bucks to the #MastodonAdmin of your instance. Better yet, if you can afford it, sign up for a monthly donation. They need it.

This only applies in #Mastodon. #Misskey and its forks like #Sharkey have #fulltextsearch too and they will ignore your "#privacy setting" for every post it has ever seen from you. And IIRC Misskey did implement full-text search way earlier than Mastodon has started debating about it. ​:sagume_think:​

Why did this debate of full-text search arose again? ​:seija_coffee:​

#fediversemeta

RE: https://social.growyourown.services/users/FediTips/statuses/111954686984517469

@[email protected] #Misskey for example also has #fulltextsearch already for every #fediverse post that it saw, so unless their #Mastodon instance decides to switch to allowlist federation and block every Misskey and other fediverse instances that ignores the proprietary "consent setting" for full-text search, their expectation of "#privacy" (if you can even call it that considering #ActivityPub's design) has been broken long ago.

Mastodon really is an outlier. ​:seija_coffee:​

@[email protected] @[email protected]

@Brendanjones
Ja, noice. As long as it only runs on that single #instance... once #Bluesky gets federated, today's admiring Feeds depth is blown to bits. #Fulltextsearch won't ever work in a federated environment. Because no individual instance can have its search engine under the hood crawl the output from thousands or millions of other instances.
Hashtagging important words is a compromise. Did you put your browser into Advanced mode and organize your GUI with columns for followed hashtags? I did and am enjoying that very much.
You could lend your request in #1 more reach by adding #MastoDev or maybe #MastoAdmin for example, and be sure to receive replies from accounts you never heard of, even without anyone boosting it.

Oh my god you can actually do #fulltextsearch here in #Misskey?? I thought the feature is just not available yet, I didn't know you have to activate it by giving the "note search" permission to a #role you have ​:koishtare:​

THAT WOULD'VE SAVED ME A LOT OF TIME LOOKING BACK THROUGH MY OWN NOTES ​:parsee_angy:​

(and i don't need to #hashtag just to be able to look it up in the future)

#fediversemeta