#advertools — Public Fediverse posts

What's your favorite Python SEO crawler?

According to The Google, there seems to be one dominant option.

python3 -m pip install advertools

If you dig deeper, you'll find two other crawlers:
- One for status codes and all response headers
- One for downloading images from a list of URLs

#advertools
#SEO
#DataScience
#Python

Using Python for doing SEO, versus using Python to develop software/tools for SEO.

The first activity is called SEO.

The second one is called software development.

Important difference.

You can use Python for crawling (try one of the #advertools crawlers), analyzing log files (also advertools), XML sitemaps (yes, yes, advertools), running bulk robots.txt tests, weighted n-grams, and much more. These are SEO tasks. Running them in bulk, using a programming language and its powerful

1/4

Using a proxy while crawling

This is another feature of using the meta parameter while crawling with #advertools.

It's as simple as providing a proxy URL.

There is also a link to using rotating proxies if you're interested

https://bit.ly/3SXh8b8

#crawling #scraping #scrapy #proxy

Happy to share a new release of #advertools v0.16

This release adds a new parameter "meta" to the crawl function.

Options to use it:

🔵 Set arbitrary metadata about the crawl
🔵 Set custom request headers per URL
🔵 Limited support for crawling some JavaScript websites

Details and example code:

https://bit.ly/3SXh8b8

#SEO #crawling #scraping #python #DataScience #advertools #scrapy

Evergreen crawling: tracking updated content

One of the things you can do is focus on a certain set of pages, and check key changes.

In this example, you'll see when and which sponsors were added to the sponsors page on BrightonSEO's website. This is based on parsed data from the page's JSON-LD.

#advertools #crawling #scraping #SEO

Evergreen crawling with XML sitemaps (status check)

This is how things will look.
All URLs in the sitemap get crawled the first time.

Then a relatively tiny number of URLs is crawled (new URLs in the sitemap, & the URLs with a changed lastmod).

URLs are saved to the same crawl file to immediately compare what changed, which URLs are frequently updated, which ones were recently introduced, & when.

https://bit.ly/4fnhiSO

#advertools #DataScience #Python #crawling #scraping #SEO

Evergreen crawling with XML sitemaps (updated to run in bulk)

The script now takes a list of tuples of XML sitemap URL, and website name, and runs the through them all, creating dynamic file names, for example

{name}_sitemap.csv
{name}_crawl.jl
{name}_errors.txt

Now you just need to add a URL and a name to add a new website to the process, instead of creating new specific files from scratch.

https://bit.ly/4fnhiSO

#advertools #crawling #sitemaps #scraping #SEO #DataScience #Python

Evergreen crawling using XML sitemaps

Crawl once then only crawl new &/or modified URLs

1. Download a sitemap
2. Crawl its URLs
3. Save it to a CSV file
4. After a month/week/day download the same sitemap again
5. Find URLs to crawl
A. New URLs not found in the last_sitemap
B. URLs that exist in both, but with a different lastmod
6. Crawl the new URLs
7. Save the current_sitemap and overwrite the last_sitemap
8. repeat

https://bit.ly/4fnhiSO

#advertools #SEO #crawling #scraping #Python

I'm liking the new default body text selector in #advertools

Many xpath/css selectors would be needed to extract the content on these pages, from many other templates on the same website?
It automatically excludes header, footer, and nav elements.

This is key in extracting the main text of (sub)category pages which might not have the same template across the website.

I'd love to know if you try it and/or have suggestions or issues.

#crawling #scraping #SEO #DataScience #Python

XML sitemaps: How to set custom request headers while fetching

Three examples are demonstrated:

🔵 Setting a custom User-agent
🔵 Fetching only if the sitemap's ETag was changed
🔵 Fetching only if the sitemap was modified since the Last-Modified response header

Let me know if you have other headers that might be interesting to use.

https://bit.ly/3WbCAd6

#advertools #SEO #DataScience #Python

Comparing crawls

A guide on how to compare any element (text or numeric) across two crawls

🔵 Select an element to compare (h1, size, etc)
🔵 Get URLs where the element has changed
🔵 Calculate the similarity ratio between changed elements
🔵 Get the matching blocks (sub-strings)
🔵 Filter URLs where a numeric value has changed by X%

Full details and code here:

https://bit.ly/3SeWVx7

#crawling #scraping #Python #analytics #DataScience #advertools #SEO

#advertools v0.15 is out! New features:

🔵 Supply any custom request headers when fetching XML sitemaps (User-agent, If-None-Match, If-Modified-Since, etc…, contributed by Joe Joiner. Thanks!

🔵 Compare crawls (adv.crawlytics.compare). Supply two crawl DataFrames, as well as a column name. You get the values that are different, and if numeric, by how much, absolute and percentage.

1/3

Testing new release v0.15.0

#advertools

A guide on how to audit and analyze internal links on a website:

🔵 Crawl the site
🔵 Map URLs to their links
🔵 Create a directed graph to represent the links
🔵 Score nodes (URLs) with various metrics (in/out/links, degree centrality, pagerank)
🔵 Cluster, summarize, and visualize URLs by internal pagerank

https://bit.ly/3xYT8ge

#advertools #scikitlearn #networkx #links

Updated guide on auditing external links:

🔵 Count domains most linked to
🔵 Interactive status code chart
🔵 Find and locate broken links
🔵 Full redirect report

https://bit.ly/4cQke86

#SEO #DataScience #advertools #scrawping #crawling #Python

Just wrote an explanation of the advertools crawl file.

If you use it for crawling, & would like to know in more detail how it is structured, what kinds of columns are available, & how they relate to one another.

Some columns are independent, containing a single value.
Some columns contain multiple values of the same tag on the same page.
Some columns form part of a whole.
It's all one jsonlines (.jl) file.

https://bit.ly/3Lbkq65

#SEO #DataScience #advertools #scraping #crawling #Python

Day 100 of #100DaysOfCode
ZERO! 🎊🎉🥳

Create a crawl + LLM content evaluation app

🔵 Enter up to 10 URLs
🔵 The app crawls them, extracts title, body text and evaluates
🔵 Evaluation is done by asking #Google's helpful content guidelines questions, answered by #ChatGPT
🔵 Export/copy URL, title, body text & evaluations

Now that the 100 days are over, and the addiction is developed, I have no guarantees!

https://www.youtube.com/watch?v=GOFJXRALDw4

#SEO #DataScience #Python #crawling #scraping #advertools #AI

Day 99 of #100DaysOfCode
ONE!

Make a minor #advertools release using the new body text selector:

🔵 To be customizable later
🔵 A step toward cleaner automated content evaluation with LLMs
🔵 More useful than I thought
🔵 Pages that have many different selectors
🔵 Websites with many different templates

Check the example screenshots. Otherwise possible but tedious.

🔵 Bonus: use it with any crawler

pip install advertools>=0.14.3

#SEO #DataScience #Python #crawling #scraping #advertools

Day 98 of #100DaysOfCode
TWO!

Create a function that generates an XPath expression for extracting the main body text from a webpage:

🔵 Ignore tags that don't typically contain text: nav, footer, img, video, etc
🔵 Include tags that typically do: b, h1, h2, li, etc
🔵 All within <body>
🔵 Ability to customize: maybe an iframe has content that you consider part of the main content?

https://bit.ly/4cfwizT

#SEO #DataScience #Python #crawling #scraping #advertools

Happy to announce a new cohort for my course: Data Science with Python for SEO
🔵 For absolute beginners
🔵 Cohort starts July 15, 2024
🔵 Make a leap in your data skills
🔵 Run, automate, and scale many SEO tasks with Python like crawling, analyzing XML sitemaps, text/keyword analysis
🔵 In depth intro to data manipulation and visualization skills
🔵 Get started with #advertools #pandas and #plotly

https://bit.ly/dsseo-course
1/

Day 97 of #100DaysOfCode
THREE!

Create team descriptions for all NBA teams

🔵 Use the bulk prompting approach
🔵 Use Google's guidelines as part of the prompt
🔵 Tried to get the description in plain text without formatting, but ended up using its formatting
🔵 New content category created
🔵 Still not properly linked, to do next

Here's the code that generated the prompts if you're interested

https://bit.ly/3W0pUXS

#SEO #DataScience #Python #GenerativeAI #LLM #advertools #OpenAI #ChatGPT

Day 96 of #100DaysOfCode
FOUR!

Extend the content evaluation script:

🔵 Use crawling as an essential part of the process to obtain content
🔵 Optionally set XPath/CSS selector to extract main content
🔵 Otherwise use the default BODY_TEXT selector
🔵 Run the evaluation process concurrently to make it much faster

https://bit.ly/45F5Mh1

#SEO #DataScience #Python #GenerativeAI #LLM #advertools #OpenAI #ChatGPT

Day 95 of #100DaysOfCode
FIVE!

Create a new app: Evaluate content with LLMs

🔵 Paste the title and text of an article
🔵 Get content guidelines evaluated/answered by ChatGPT
🔵 Export to a spreadsheet

https://bit.ly/4ct3TpG

#SEO #DataScience #Python #GenerativeAI #LLM #advertools #OpenAI #ChatGPT #dash

UX question: I can make this work in bulk but how would you upload twenty articles?
Zip file, spreadsheet, other?
What's the template for that?
What about titles?

Day 94 of #100DaysOfCode
SIX!

Evaluate content in bulk with LLMs

🔵 Get a set of structured criteria/questions that evaluate to true/false (Google's helpful content guidelines)
🔵 Ask the LLM to evaluate the article accordingly
🔵 Rinse, repeat
🔵 Produce a summary report, dig deeper

https://bit.ly/4eKxWvg

#SEO #DataScience #Python #GenerativeAI #LLM #advertools #OpenAI #ChatGPT

Got the idea from a video by Daniel Foley Carter

Day 92 of #100DaysOfCode
EIGHT!

Create a slides version of the GSC audit/analysis template

https://bit.ly/3RxZWrS

#SEO #GSC #analytics #advertools #DataScience #Python #plotly #quarto

Day 91 of #100DaysOfCode
NINE!

Update the sitemap dashboard with a few key summary values:

https://bit.ly/4cnxcdh

#SEO #SEM #analytics #advertools #DataScience #Python #plotly #quarto

Day 89 of #100DaysOfCode

Create 1st draft of an XML sitemap dashboard

🔵 Compare multiple sitemaps side-by-side (industry, competitors, etc.)
🔵 See who made changes when, and how much
🔵 Assuming lastmods are correct

- OpenAI revamped the whole site this Apr, changed 221 URLs in Jun 9 - Jun 15
- brightonSEO started changing everything since last Sep, 51% of pages were updated by Sep 25

https://bit.ly/4cnxcdh

#SEO #SEM #analytics #advertools #DataScience #Python #plotly #quarto

Day 88 of #100DaysOfCode

Automating Python scripts on a server

Blog post created from a video

pydub: Splitting audio file
whisper: Speech to text model
GPT4o: Formatting the article. This is the most interesting part (see images), because the talk has natural language, file names, Python and bash code, and it was able to properly format everything, and I mainly copied/pasted.

https://bit.ly/4bYymwj

#SEO #SEM #analytics #advertools #DataScience #Python #datamanipulation

Day 87 of #100DaysOfCode

Create a script to invert a mapping:

FROM:

URL-A: keyword1, keyword2, keyword3
URL-B: keyword2, keyword4

TO:

keyword1: URL-A, URL-B
keyword2: URL-B, URL-C

etc...

https://bit.ly/4cEqGPN

#SEO #SEM #analytics #advertools #DataScience #Python #datamanipulation

Day 86 of #100DaysOfCode

Created and deployed a blog, with a few initial posts.

Because, they say, it's good to have one...

https://bit.ly/4enkaOH

#SEO #SEM #analytics #advertools #DataScience #Python #quarto #plotly

Day 82 of #100DaysOfCode

Released an update to the ecdf function

🔵 Add fuller docstring and parameter explanations
🔵 Choose a better marker (open circle) to cater for overlaps with larger datasets
🔵 Fix hover label formatting and text
🔵 Remove sparklines from histogram and clearer Y axis title

https://bit.ly/3VieUmQ

#SEO #SEM #DataScience #DataVisualization #advertools #adviz #Python

Day 80 of #100DaysOfCode
🟦🟦🟦🟦🟦🟦🟦🟦⬜⬜ 80%

New charts to visualize distributions (ecdf)

🔵 Histogram with an ecdf to show distribution and the cumulative distribution
🔵 Observations are hoverable
🔵 Get insights on each observation and where it fits in the sample (how many are less, greater, etc)
🔵 Split by URL segment or keyword category with one simple option

https://bit.ly/3VieUmQ

pip install --upgrade adviz

#SEO #SEM #DataScience #DataVisualization #Python #Plotly #adviz #advertools

Day 75 of #100DaysOfCode

User-agent parser app refactor and update

🔵 Upload a list of user-agent strings
🔵 Get them split into their components (OS, family, device, brand, version...)
🔵 Download parsed UA's to a CSV file
🔵 Interactively visualize the UA's on multiple levels using any of the components

https://bit.ly/3EEPRkv

#advertools #SEO #DataVisualization #DataScience #logfiles

Day 72 of #100DaysOfCode

Created the ADV_AD_FROM_STRING() function and a simple workflow for creating text ads in bulk:

🔵 Split a string of text into slots of at most 30, 30, 30, 90, 90 characters each (to be made more flexible later)
🔵 Obtain URLs in bulk
🔵 Create ads in bulk, taking care not to split in the wro,ng place and not crea,te wrong text spl,its

Vidoe demo:
https://bit.ly/3yIhBGK

Google sheet:
https://bit.ly/3VnsBSL

#advertools #textads #googleads #SEM #PPC

Day 72 of #100DaysOfCode

Created the ADV_AD_FROM_STRING() function and a simple workflow for creating text ads in bulk:

🔵 Split a string of text into slots of at most 30, 30, 30, 90, 90 characters each (to be made more flexible later)
🔵 Obtain URLs in bulk
🔵 Create ads in bulk, taking care not to split in the wro,ng place and not crea,te wrong text spl,its

Vidoe demo:
https://bit.ly/3yIhBGK

Google sheet:
https://bit.ly/3VnsBSL

#advertools #textads #googleads #SEM #PPC

Day 67 of #100DaysOfCode

Google Sheets tutorial: sitemaps, URL structure, and list-mode scraping with #advertools

You can do programming with sheets if you want to.

The recent introduction of named functions is making this easier (no app script).
You will like it if you're into functional programming, lambda, map, reduce...

Make a copy of the sheet to get access to all functions discussed

Enjoy!

https://bit.ly/3Ki2bvj

#SEO #Googlesheets #scraping #crawling #sitemaps

Day 22 of #100DaysOfCode

Updated and migrated the Google SERP Dashboard + heatmap:

🔵 Hover to show ranking domains for each position
🔵 Use Dash Ag Grid for displaying and filtering raw data on the page (you can export of course)
🔵 Use all available parameters of the Google search API

https://bit.ly/3UgVAHC

#DataScience #SEO #SERP #Google #advertools #Python #adviz #plotly #aggrid #frontend #webdev

1/2

Day 15 of #100DaysOfCode:

Created a bunch of custom #advertools crawlers, with one line of code each.

You can set your own defaults (e.g. follow_links=True by default?)

Examples:

🔵 Exploratory crawler: spider mode, on. Stop after 2k URLs.
🔵 Rude crawler: Don't obey robots rules.
🔵 Polite crawler: Obey robots (default), crawl very slowly, with long periods between crawled pages.

#DataScience #crawling #scraping #scrapy #SEO #Python #data

1/2

Day 14 of #100DaysOfCode:

Created a tutorial on analyzing millions of URLs:

🔵 2.4M URLs from a web server log file
🔵 Splitting into their components creates a 5.7GB (giga) DataFrame
🔵 Using the new output_file parameter saves the same data in a 67MB (mega) file
🔵 Read only the columns you want, while filtering for a subset of rows
🔵 Enjoy!

Notebook and video:

https://bit.ly/49socSd

#DataScience #Python #logfile #URL #SEO #advertools

Day 8 of #100DaysOfCode:

Added the option to specify custom date formats for log files:

🔵 advertools.logs_to_df will attempt to convert datetime columns to a datetime type according to default formats
🔵 Supply your own date format if your logs have a different one (or if you decide to change it)
🔵 Date format will be using the strftime format spec
🔵 Coming to adv v.0.15.0

#advertools #logfile #DataScience #SEO #Python

Day 1 of #100DaysOfCode:

Added the ability to supply request headers while fetching sitemaps with #advertools
(available in the next release)

This can help in changing the User-agent for example. It can also be used to only fetch sitemaps that haven't changed, using the If-None-Match header. You can keep a fresh set of sitemaps, check continuously, and only download updated ones.

You can use any other header of course.

#DataScience #SEO #XML #Sitemap #Python

SERP Heatmap

Would it improve the heatmap if horizontal lines were added to show the average position for each domain?

Is red better or a more neutral gray color better?

How else can this be improved?

#DataVisualization #SEO #advertools #adviz #Python #Plotly #SERP

Finding internal broken links is much easier than external ones with the crawlytics links function.

After crawling a website:

🔵 Get the link summary table with crawlytics[.]links
🔵 Filter the error pages from the crawl table by any status code you want e.g. >=400, != 200, etc.
🔵 Merge the two tables
🔵 Done

Here is a notebook if you want to test it out:

https://bit.ly/43dKhCB

#advertools #DataScience #SEO #Python #DigitalAnalytics #DigitalMarketing

External link analysis with the #advertools crawlytics module

🔵 Use the links() function to map all links on website (URL, anchor text, nofollow, internal/external)
🔵 Count the most linked-to domains
🔵 Crawl external links and get status codes
🔵 Locate broken external links on the website using their location and anchor text
🔵 Enjoy

Get a copy of the HTML report (includes link to code repo):
https://bit.ly/48OowL5

#DataScience #SEO #Crawling #Python #DigitalAnalytics #DigitalMarketing

Data Science with Python for SEO Course: This Monday!
Get the full details and join here:

https://bit.ly/dsseo-course

If you have any questions let me know, and if you think others might benefit, please let them know.

#DataScience #SEO #Python #DigitalMarketing #DigitalAnalytics #advertools #pandas #plotly #DataVisualization

Internal links: How interlinked are the different sections of a website?

🔵 Using adv[.]crawlytics[.]links we get a mapping of all links (source -> destination)
🔵 Using adv[.]url_to_df we get each component of those links (scheme, domain, path, etc)
🔵 Count the combinations of the first directories to get the number of links from/to each section of the website

What do you think?

#DataScience #digitalanalytics #Python #advertools #SEO

#GSC analysis report template - 1st version

Discussed in #advertools office hours tomorrow

Here's a copy of the current report. Would love any recommendations, issues, suggestions...

https://bit.ly/48BApVI

#GoogleSearchConsole #DigitalAnalytics #SEO #DataScience #Python #advertools Report created using @Posit 's Quarto

Google Search Console Animated Monthly Clicks Chart

Download it here: https://bit.ly/41BUc2X
Code & sample data: http://bit.ly/4050W8h

Coming soon...

#DataScience #DataVisualization #Python #advertools #adviz #SEO #SEM #DigitalMarketing #DigitalAnalytics

The split of topics that The New York Times covered in 2022.

Interactive HTML chart & code:
https://bit.ly/3zSxbNh

You can check other years and see how/if their publishing has changed.

I removed the dates from URLs in this case (YYYY/MM/DD) to get a better overview. Note that you can include links* in the chart:
Links*: more than one
Links*: using a URL shortener like bit[.]ly
Links*: containing UTM codes

#DataScience #DataVisualization #Python #treemap #advertools #adviz #SEO