home.social

#user_agent — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #user_agent, aggregated by home.social.

  1. → Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
    blog.cloudflare.com/perplexity

    “We observed that #Perplexity [an AI-powered answer engine] uses not only their declared #user_agent, but also a generic browser intended to #impersonate Google Chrome on macOS when their declared crawler was blocked.”

    “This activity was observed across tens of thousands of domains and millions of requests per day.”

    #AI #evade #stealth #website #browser #crawler #blocked

  2. → Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
    blog.cloudflare.com/perplexity

    “We observed that #Perplexity [an AI-powered answer engine] uses not only their declared #user_agent, but also a generic browser intended to #impersonate Google Chrome on macOS when their declared crawler was blocked.”

    “This activity was observed across tens of thousands of domains and millions of requests per day.”

    #AI #evade #stealth #website #browser #crawler #blocked

  3. → Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
    blog.cloudflare.com/perplexity

    “We observed that #Perplexity [an AI-powered answer engine] uses not only their declared #user_agent, but also a generic browser intended to #impersonate Google Chrome on macOS when their declared crawler was blocked.”

    “This activity was observed across tens of thousands of domains and millions of requests per day.”

    #AI #evade #stealth #website #browser #crawler #blocked

  4. → Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
    blog.cloudflare.com/perplexity

    “We observed that #Perplexity [an AI-powered answer engine] uses not only their declared #user_agent, but also a generic browser intended to #impersonate Google Chrome on macOS when their declared crawler was blocked.”

    “This activity was observed across tens of thousands of domains and millions of requests per day.”

    #AI #evade #stealth #website #browser #crawler #blocked

  5. → Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
    blog.cloudflare.com/perplexity

    “We observed that #Perplexity [an AI-powered answer engine] uses not only their declared #user_agent, but also a generic browser intended to #impersonate Google Chrome on macOS when their declared crawler was blocked.”

    “This activity was observed across tens of thousands of domains and millions of requests per day.”

    #AI #evade #stealth #website #browser #crawler #blocked

  6. @timtilberg ooooh I now see the issue with `this` and multiple header/cookie sets.

    Currently I have `#user_agent=` write it's value directly into `#headers`. However, this pattern doesn't exactly work neatly with cookies, if you override `#headers=` after setting `#cookie=`.

  7. @timtilberg ooooh I now see the issue with `this` and multiple header/cookie sets.

    Currently I have `#user_agent=` write it's value directly into `#headers`. However, this pattern doesn't exactly work neatly with cookies, if you override `#headers=` after setting `#cookie=`.

  8. @timtilberg ooooh I now see the issue with `this` and multiple header/cookie sets.

    Currently I have `#user_agent=` write it's value directly into `#headers`. However, this pattern doesn't exactly work neatly with cookies, if you override `#headers=` after setting `#cookie=`.

  9. @timtilberg ooooh I now see the issue with `this` and multiple header/cookie sets.

    Currently I have `#user_agent=` write it's value directly into `#headers`. However, this pattern doesn't exactly work neatly with cookies, if you override `#headers=` after setting `#cookie=`.

  10. @timtilberg ooooh I now see the issue with `this` and multiple header/cookie sets.

    Currently I have `#user_agent=` write it's value directly into `#headers`. However, this pattern doesn't exactly work neatly with cookies, if you override `#headers=` after setting `#cookie=`.