#scrapy — Public Fediverse posts

Released scrapy-contrib-bigexporter 1.0.0 (https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters) - additional export formats for the webscraping framework Scrapy.

Migrated parquet export from fastparquet to pyarrow as fastparquet is deprecated (https://docs.dask.org/en/stable/changelog.html#fastparquet-engine-deprecated)

Migrated orc export from pyorc to pyarrow to reduce the number of dependencies

#scrapy #crawling #python #parquet #orc #pyarrow #webcrawling #scraping

scrapy-contrib-bigexporter 0.6.1 released: https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters

Added: You can customize Iceberg table location

#scrapy #webscraping #bigdata #iceberg #apacheiceberg #opensource #python

scrapy-contrib-bigexporter 0.6.0 released: https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters

New: Export your webscraped items in Scrapy to Apache Iceberg tables with simple configuration

#scrapy #webscraping #bigdata #iceberg #apacheiceberg #opensource #python

Achei que esse ano não ia rolar uma palestra para o Caipyra. Mas no fim aprovaram uma das minhas propostas! Para quem trabalha com raspagem de dados e com Scrapy, bora aprender uns tópicos um pouquinho mais avançados agora em junho, lá em Ribeirão Preto?
https://2025.caipyra.python.org.br/

#Caipyra2025 #Python #Scrapy

Настраиваем паука для сбора данных: как работает фреймворк Scrapy

В Точке мы обучаем наших AI-ассистентов, а для этого нужно много данных. В статье расскажу, как быстро собрать информацию практически с любого сайта при помощи фреймворка Scrapy.

https://habr.com/ru/companies/tochka/articles/885184/

#scrapy #python #сбор_данных

Create your personal web crawler with #scrapy and #raspberrypi. Complete tutorial to scrape your favourite websites with a simple #python script #linux #diy https://peppe8o.com/use-raspberry-pi-as-your-personal-web-crawler-with-python-and-scrapy/

Как парсить данные с Python

Парсинг — это автоматический поиск различных паттернов (на основе заранее определенных конструкций) из текстовых источников данных для извлечения специфической информации. Не смотря на то, что парсинг — широкое понятие, чаще всего под этим термином подразумевают процесс сбора и анализа данных с удаленных веб-ресурсов.

https://habr.com/ru/companies/timeweb/articles/877596/

#timeweb_статьи #html #python #парсинг #ubuntu #xml #вебсайт #JSON #javascript #scrapy

Would this be useful?
What would you use it for?

Examples:

- Page testing using different proxies
- Page testing using different devices (still can't figure why it's not properly working though)
- Ads tracking (yours and/or competitors)
- Tracking featured content
- other?

https://bit.ly/486KQRa

#scrapy #Python #playwright

2/2

@rolanddreger If the wget way is not enough (if e.g. you need to modify the contents): Some years ago I created something for the same use case with #python and #scrapy (https://scrapy.org/) – let me know if you need some input there.

Using a proxy while crawling

This is another feature of using the meta parameter while crawling with #advertools.

It's as simple as providing a proxy URL.

There is also a link to using rotating proxies if you're interested

https://bit.ly/3SXh8b8

#crawling #scraping #scrapy #proxy

Happy to share a new release of #advertools v0.16

This release adds a new parameter "meta" to the crawl function.

Options to use it:

🔵 Set arbitrary metadata about the crawl
🔵 Set custom request headers per URL
🔵 Limited support for crawling some JavaScript websites

Details and example code:

https://bit.ly/3SXh8b8

#SEO #crawling #scraping #python #DataScience #advertools #scrapy

You might be familiar with what I'm terming the "Token Wars" - in which #LLM and #GenAI companies seek to ingest text, image, audio and video content to create their #ML models. Tokens are the basic unit of data input into these models - meaning that #scraping of web content is widespread.

In retaliation, many sites - such as Reddit, Inc. and Stack Overflow - are entering into content sharing deals with companies like OpenAI, or making their sites subscription only.

Another solution that has emerged recently is content blocking based on user agent. In web programming, the client requesting a web page identifies themself - usually as a browser or a bot.

User agents can be blocked by a website's robots.txt file - but only if the user agent respects the robots.txt protocol. Many web scrapers do not. Taking this a step further, network providers like Cloudflare are now offering solutions which block known token scraper bots at a a network level.

I've been playing with one of these solutions called #DarkVisitors for a couple weeks after learning it about it on The Sizzle and was **amazed** at how much traffic to my websites were bots, crawlers and content scrapers.

https://darkvisitors.com

(No backhanders here, it's just a very insightful tool)

#TokenWars #tokenization #scraping #bots #scrapy #WebScraping

I'm deconstructing a pair of jeans that I've been using part of cus they were old. I'm planning on keeping the legs intact as I'm going create another jean wall art with them.

This is one of three that I have done. How does it look?

#sew #sewing #upcycle #upcycling #upcycled #craft #handmade #crafty #crafts #scrapy #clothing #clothes #jeans #art

As a volunteer #FLOSS community manager, it bugs me that 10 years in, it's still impossible to filter #GitLab issues by created (& closed) date ranges, to get a rough idea of how many tickets have been created in the time period, and substract them from the total of tickets were closed based on various target milestones.

https://gitlab.com/gitlab-org/gitlab/-/issues/17481
https://gitlab.com/gitlab-org/gitlab/-/issues/17758
https://gitlab.com/gitlab-org/gitlab/-/issues/121728

Y'know, *basic* statistics. For talk slides.
Anyone got a #scraping script? Maybe with #scrapy?

Literally just made a pin version.

#sew #sewing #upcycle #upcycling #upcycled #craft #handmade #crafty #crafts #scrapy

Made some more peace symbols using jean scraps. How'd they look? Just need to see them onto something.

I did wonder if I should actually sew safety pins on the backs and have them as removable pin badges instead of patches. However I may need to make smaller ones to do that.

I thought about making them more uniform and circular but I like the contrast of the lighter circle on bold darker material.

Good?

#sew #sewing #upcycle #upcycling #upcycled #craft #handmade #crafty #crafts #scrapy

If you're into #scraping and #Python, you should probably know that there's a new cool kid in town. Just a few days ago Apify has released a new OSS framework called #Crawlee.

It's asyncio, type hints, it can seamlessly switch between basic and browser (Playwright) requests. Auto retries, proxy rotation. I'm very much #Scrapy person, but these promises get me really intrigued! https://github.com/apify/crawlee-python

Tell them what you think - talk to Saurav at their #EuroPython booth or file GitHub issues 🛠️

If you're into #scraping and #Python, you should probably know that there's a new cool kid in town. Just a few days ago Apify has released a new OSS framework called #Crawlee.

It's asyncio, type hints, it can seamlessly switch between basic and browser (Playwright) requests. Auto retries, proxy rotation. I'm very much #Scrapy person, but these promises get me really intrigued! https://github.com/apify/crawlee-python

Tell them what you think - talk to Saurav at their #EuroPython booth or file GitHub issues 🛠️

If you're into #scraping and #Python, you should probably know that there's a new cool kid in town. Just a few days ago Apify has released a new OSS framework called #Crawlee.

It's asyncio, type hints, it can seamlessly switch between basic and browser (Playwright) requests. Auto retries, proxy rotation. I'm very much #Scrapy person, but these promises get me really intrigued! https://github.com/apify/crawlee-python

Tell them what you think - talk to Saurav at their #EuroPython booth or file GitHub issues 🛠️

If you're into #scraping and #Python, you should probably know that there's a new cool kid in town. Just a few days ago Apify has released a new OSS framework called #Crawlee.

It's asyncio, type hints, it can seamlessly switch between basic and browser (Playwright) requests. Auto retries, proxy rotation. I'm very much #Scrapy person, but these promises get me really intrigued! https://github.com/apify/crawlee-python

Tell them what you think - talk to Saurav at their #EuroPython booth or file GitHub issues 🛠️

If you're into #scraping and #Python, you should probably know that there's a new cool kid in town. Just a few days ago Apify has released a new OSS framework called #Crawlee.

It's asyncio, type hints, it can seamlessly switch between basic and browser (Playwright) requests. Auto retries, proxy rotation. I'm very much #Scrapy person, but these promises get me really intrigued! https://github.com/apify/crawlee-python

Tell them what you think - talk to Saurav at their #EuroPython booth or file GitHub issues 🛠️

"Gathering data from the web using Python" workshop by Renne Rocha at #PyConUS
Free 3-hour tutorial!
Video: https://www.youtube.com/watch?v=sD30nvc1ff0
GitHub repo: https://github.com/rennerocha/pyconus2024-tutorial

Rocha maintains the Querido Diário open-source platform https://queridodiario.ok.org.br/ that scrapes official documents of varoius cities in Brazil to make them more easy to access.

#python #ddj #WebScraping #scrapy @python

【Scrapyクローラログ】クローリングにおけるログ出力の考え方【前編】
https://qiita.com/realmadridmarcelo/items/b4a8a183682efcf68d13?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items

#qiita #スクレイピング #ログ #Scrapy #クローラー

async scrapy selenium grid https://github.com/dozymoe/scrapy-selenium-grid

document, pip module, tests will follow

#python #scrapy

Ne Weile mit #Scrapy (Webscraping #Framework in #Python) rumgespielt und festgestellt dass es "alles" kann was ich in die Richtung jemals tun wollen kann.

…aber eine denkbar schlechte Idee für Software ist, die ich irgendwo als Paket auf nen beliebigen Rechner aufspielen und ausführen will.

Also doch BeautifulSoup oder so…

Day 17 of #100DaysOfCode:

Update custom crawlers with new ones (mostly suggested by Lee Driggers 🙏 )

🔵 Add a a list of 1k user-agents (list can/should be expanded)
🔵 Parse user agents for custom filtering
🔵 Random user-agent crawler
🔵 Android crawler (crawl with a randomized UA of one of the Android options)

https://bit.ly/3TIZtDx

#DataScience #crawling #scraping #scrapy #SEO #Python #data

1/2

Day 15 of #100DaysOfCode:

Created a bunch of custom #advertools crawlers, with one line of code each.

You can set your own defaults (e.g. follow_links=True by default?)

Examples:

🔵 Exploratory crawler: spider mode, on. Stop after 2k URLs.
🔵 Rude crawler: Don't obey robots rules.
🔵 Polite crawler: Obey robots (default), crawl very slowly, with long periods between crawled pages.

#DataScience #crawling #scraping #scrapy #SEO #Python #data

1/2

Упорядочиваем закладки

У меня много статей в закладках. Многие из них я добавил, чтобы прочитать позже. Это статья не про то, почему так произошло и как с этим бороться, а про то, как выбрать статью для удаления чтения. Давайте найдем самые лучшие статьи. Критериями могут быть, например, рейтинг, просмотры и т.д. И красиво оформим в виде HTML-файла. Пример закладок @alizar : 800+ статей . Упорядочить!

https://habr.com/ru/articles/800475/

#python #scrapy #закладки #прокрастинация #gtd #скрапинг

In the latest release, auto-throttling* is enabled by default. The intervals between requests are dynamically adjusted to ensure you are not overwhelming servers.

Check it out here:
https://bit.ly/49kHBp4

#SEO #TechSEO #DataScience #Python #digitalanalytics

*magic provided by #scrapy
2/2

Python, MSA, Kafka

Всем привет! Сегодня микросервисная архитектура, что называется "на хайпе". Я перечитал достаточно много статей по данной тематике, но обнаружил, что среди всего прочего, не так много публикаций, объясняющих данную концепцию на конкретном примере (может, плохо искал). Сегодня я бы хотел пополнить ряды авторов и написать свою первую публикацию, не судите строго!

https://habr.com/ru/articles/793936/

#Python #msa #microservices #kafka #fastapi #scrapy #aiogram

Oficina: Extraindo dados da Intenet usando Python

Laboratório Hacker de Campinas, sábado, 11 maio (09:00)

https://eventos.lhc.net.br/event/oficina-extraindo-dados-da-intenet-usando-python

Oficina: Extraindo dados da Intenet usando Python

Laboratório Hacker de Campinas, sábado, 11 maio (09:00)

https://eventos.lhc.net.br/event/oficina-extraindo-dados-da-intenet-usando-python

Oficina: Extraindo dados da Intenet usando Python

Laboratório Hacker de Campinas, sábado, 11 maio (09:00)

https://eventos.lhc.net.br/event/oficina-extraindo-dados-da-intenet-usando-python

Oficina: Extraindo dados da Intenet usando Python

Laboratório Hacker de Campinas, sábado, 11 maio (09:00)

https://eventos.lhc.net.br/event/oficina-extraindo-dados-da-intenet-usando-python

Oficina: Extraindo dados da Intenet usando Python

Laboratório Hacker de Campinas, sábado, 11 maio (09:00)

https://eventos.lhc.net.br/event/oficina-extraindo-dados-da-intenet-usando-python

What are your favorite / the best #WebCrawlers for broad / #WebScale #crawling?

I've built a list but am looking for anything I missed: https://github.com/davidshq/awesome-search-engines/blob/main/WebCrawlers.md

Main options I've found include #Apache #Nutch, #StormCrawler, #Scrapy, #Norconex, #PulsarR, #Heritrix, and #sparkler

#question #search #SearchEngines

What are your favorite / the best #WebCrawlers for broad / #WebScale #crawling?

I've built a list but am looking for anything I missed: https://github.com/davidshq/awesome-search-engines/blob/main/WebCrawlers.md

Main options I've found include #Apache #Nutch, #StormCrawler, #Scrapy, #Norconex, #PulsarR, #Heritrix, and #sparkler

#question #search #SearchEngines

What are your favorite / the best #WebCrawlers for broad / #WebScale #crawling?

I've built a list but am looking for anything I missed: https://github.com/davidshq/awesome-search-engines/blob/main/WebCrawlers.md

Main options I've found include #Apache #Nutch, #StormCrawler, #Scrapy, #Norconex, #PulsarR, #Heritrix, and #sparkler

#question #search #SearchEngines

What are your favorite / the best #WebCrawlers for broad / #WebScale #crawling?

I've built a list but am looking for anything I missed: https://github.com/davidshq/awesome-search-engines/blob/main/WebCrawlers.md

Main options I've found include #Apache #Nutch, #StormCrawler, #Scrapy, #Norconex, #PulsarR, #Heritrix, and #sparkler

#question #search #SearchEngines

What are your favorite / the best #WebCrawlers for broad / #WebScale #crawling?

I've built a list but am looking for anything I missed: https://github.com/davidshq/awesome-search-engines/blob/main/WebCrawlers.md

Main options I've found include #Apache #Nutch, #StormCrawler, #Scrapy, #Norconex, #PulsarR, #Heritrix, and #sparkler

#question #search #SearchEngines

Després de dinar, treballem amb #Scrapy amb https://github.com/JimenaEB presidenta de #PythonES #PyBCNDay #PythonBarcelona #PyLadiesBarcelona #PyDayBCN

Després de dinar, treballem amb #Scrapy amb https://github.com/JimenaEB presidenta de #PythonES #PyBCNDay #PythonBarcelona #PyLadiesBarcelona #PyDayBCN

#Python #Frameworks #Libraries #numpy #tensorflow #theano #pandas #pytorch #keras #matplotlib #scipy #seaborn #django #flask #bottle #cherrypy #pyramid #web2py #turboGears #cubic #dash #falcon #pyunit #behave #splinter #robot #pytest #opencv #mahotas #pgmagick #simpletk $scikit #arcade #pyglet #pyopengl #pygame #panda3d #lxml #requests #selenium #scrapy #code #developing #programming #coding

#Python #Frameworks #Libraries #numpy #tensorflow #theano #pandas #pytorch #keras #matplotlib #scipy #seaborn #django #flask #bottle #cherrypy #pyramid #web2py #turboGears #cubic #dash #falcon #pyunit #behave #splinter #robot #pytest #opencv #mahotas #pgmagick #simpletk $scikit #arcade #pyglet #pyopengl #pygame #panda3d #lxml #requests #selenium #scrapy #code #developing #programming #coding

#Python #Frameworks #Libraries #numpy #tensorflow #theano #pandas #pytorch #keras #matplotlib #scipy #seaborn #django #flask #bottle #cherrypy #pyramid #web2py #turboGears #cubic #dash #falcon #pyunit #behave #splinter #robot #pytest #opencv #mahotas #pgmagick #simpletk $scikit #arcade #pyglet #pyopengl #pygame #panda3d #lxml #requests #selenium #scrapy #code #developing #programming #coding

#Python #Frameworks #Libraries #numpy #tensorflow #theano #pandas #pytorch #keras #matplotlib #scipy #seaborn #django #flask #bottle #cherrypy #pyramid #web2py #turboGears #cubic #dash #falcon #pyunit #behave #splinter #robot #pytest #opencv #mahotas #pgmagick #simpletk $scikit #arcade #pyglet #pyopengl #pygame #panda3d #lxml #requests #selenium #scrapy #code #developing #programming #coding

#Python #Frameworks #Libraries #numpy #tensorflow #theano #pandas #pytorch #keras #matplotlib #scipy #seaborn #django #flask #bottle #cherrypy #pyramid #web2py #turboGears #cubic #dash #falcon #pyunit #behave #splinter #robot #pytest #opencv #mahotas #pgmagick #simpletk $scikit #arcade #pyglet #pyopengl #pygame #panda3d #lxml #requests #selenium #scrapy #code #developing #programming #coding