#scrapy — Public Fediverse posts

Released scrapy-contrib-bigexporter 1.0.0 (https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters) - additional export formats for the webscraping framework Scrapy.

Migrated parquet export from fastparquet to pyarrow as fastparquet is deprecated (https://docs.dask.org/en/stable/changelog.html#fastparquet-engine-deprecated)

Migrated orc export from pyorc to pyarrow to reduce the number of dependencies

#scrapy #crawling #python #parquet #orc #pyarrow #webcrawling #scraping

In the latest release, auto-throttling* is enabled by default. The intervals between requests are dynamically adjusted to ensure you are not overwhelming servers.

Check it out here:
https://bit.ly/49kHBp4

#SEO #TechSEO #DataScience #Python #digitalanalytics

*magic provided by #scrapy
2/2

#Python #Frameworks #Libraries #numpy #tensorflow #theano #pandas #pytorch #keras #matplotlib #scipy #seaborn #django #flask #bottle #cherrypy #pyramid #web2py #turboGears #cubic #dash #falcon #pyunit #behave #splinter #robot #pytest #opencv #mahotas #pgmagick #simpletk $scikit #arcade #pyglet #pyopengl #pygame #panda3d #lxml #requests #selenium #scrapy #code #developing #programming #coding