#advertools — Public Fediverse posts

Using a proxy while crawling

This is another feature of using the meta parameter while crawling with #advertools.

It's as simple as providing a proxy URL.

There is also a link to using rotating proxies if you're interested

https://bit.ly/3SXh8b8

#crawling #scraping #scrapy #proxy

Happy to share a new release of #advertools v0.16

This release adds a new parameter "meta" to the crawl function.

Options to use it:

🔵 Set arbitrary metadata about the crawl
🔵 Set custom request headers per URL
🔵 Limited support for crawling some JavaScript websites

Details and example code:

https://bit.ly/3SXh8b8

#SEO #crawling #scraping #python #DataScience #advertools #scrapy

Day 75 of #100DaysOfCode

User-agent parser app refactor and update

🔵 Upload a list of user-agent strings
🔵 Get them split into their components (OS, family, device, brand, version...)
🔵 Download parsed UA's to a CSV file
🔵 Interactively visualize the UA's on multiple levels using any of the components

https://bit.ly/3EEPRkv

#advertools #SEO #DataVisualization #DataScience #logfiles

Day 15 of #100DaysOfCode:

Created a bunch of custom #advertools crawlers, with one line of code each.

You can set your own defaults (e.g. follow_links=True by default?)

Examples:

🔵 Exploratory crawler: spider mode, on. Stop after 2k URLs.
🔵 Rude crawler: Don't obey robots rules.
🔵 Polite crawler: Obey robots (default), crawl very slowly, with long periods between crawled pages.

#DataScience #crawling #scraping #scrapy #SEO #Python #data

1/2

Day 14 of #100DaysOfCode:

Created a tutorial on analyzing millions of URLs:

🔵 2.4M URLs from a web server log file
🔵 Splitting into their components creates a 5.7GB (giga) DataFrame
🔵 Using the new output_file parameter saves the same data in a 67MB (mega) file
🔵 Read only the columns you want, while filtering for a subset of rows
🔵 Enjoy!

Notebook and video:

https://bit.ly/49socSd

#DataScience #Python #logfile #URL #SEO #advertools

Day 8 of #100DaysOfCode:

Added the option to specify custom date formats for log files:

🔵 advertools.logs_to_df will attempt to convert datetime columns to a datetime type according to default formats
🔵 Supply your own date format if your logs have a different one (or if you decide to change it)
🔵 Date format will be using the strftime format spec
🔵 Coming to adv v.0.15.0

#advertools #logfile #DataScience #SEO #Python

Day 1 of #100DaysOfCode:

Added the ability to supply request headers while fetching sitemaps with #advertools
(available in the next release)

This can help in changing the User-agent for example. It can also be used to only fetch sitemaps that haven't changed, using the If-None-Match header. You can keep a fresh set of sitemaps, check continuously, and only download updated ones.

You can use any other header of course.

#DataScience #SEO #XML #Sitemap #Python

Finding internal broken links is much easier than external ones with the crawlytics links function.

After crawling a website:

🔵 Get the link summary table with crawlytics[.]links
🔵 Filter the error pages from the crawl table by any status code you want e.g. >=400, != 200, etc.
🔵 Merge the two tables
🔵 Done

Here is a notebook if you want to test it out:

https://bit.ly/43dKhCB

#advertools #DataScience #SEO #Python #DigitalAnalytics #DigitalMarketing

External link analysis with the #advertools crawlytics module

🔵 Use the links() function to map all links on website (URL, anchor text, nofollow, internal/external)
🔵 Count the most linked-to domains
🔵 Crawl external links and get status codes
🔵 Locate broken external links on the website using their location and anchor text
🔵 Enjoy

Get a copy of the HTML report (includes link to code repo):
https://bit.ly/48OowL5

#DataScience #SEO #Crawling #Python #DigitalAnalytics #DigitalMarketing

Data Science with Python for SEO Course: This Monday!
Get the full details and join here:

https://bit.ly/dsseo-course

If you have any questions let me know, and if you think others might benefit, please let them know.

#DataScience #SEO #Python #DigitalMarketing #DigitalAnalytics #advertools #pandas #plotly #DataVisualization

Internal links: How interlinked are the different sections of a website?

🔵 Using adv[.]crawlytics[.]links we get a mapping of all links (source -> destination)
🔵 Using adv[.]url_to_df we get each component of those links (scheme, domain, path, etc)
🔵 Count the combinations of the first directories to get the number of links from/to each section of the website

What do you think?

#DataScience #digitalanalytics #Python #advertools #SEO

#GSC analysis report template - 1st version

Discussed in #advertools office hours tomorrow

Here's a copy of the current report. Would love any recommendations, issues, suggestions...

https://bit.ly/48BApVI

#GoogleSearchConsole #DigitalAnalytics #SEO #DataScience #Python #advertools Report created using @Posit 's Quarto

Google Search Console Animated Monthly Clicks Chart

Download it here: https://bit.ly/41BUc2X
Code & sample data: http://bit.ly/4050W8h

Coming soon...

#DataScience #DataVisualization #Python #advertools #adviz #SEO #SEM #DigitalMarketing #DigitalAnalytics

The split of topics that The New York Times covered in 2022.

Interactive HTML chart & code:
https://bit.ly/3zSxbNh

You can check other years and see how/if their publishing has changed.

I removed the dates from URLs in this case (YYYY/MM/DD) to get a better overview. Note that you can include links* in the chart:
Links*: more than one
Links*: using a URL shortener like bit[.]ly
Links*: containing UTM codes

#DataScience #DataVisualization #Python #treemap #advertools #adviz #SEO