#webstats — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #webstats, aggregated by home.social.
-
England Vs Argentina last night was *busy*! Easily the busiest ever live event on the BBC (I can't give you streaming stats, sorry).
On my logging pipeline (subset of BBC logs):
- Peak: 2.2M row/s (105GB/s)
- Total: 20.1B rows (16.4TB), 2.7M log files
- 0 write fails
- 0 duplicated rows
- Peak: 1035 Cloud Run containers @ 2x CPU each ∴ 2070 CPUBest bit is that we did precisely nothing, it all just worked.
-
England Vs Argentina last night was *busy*! Easily the busiest ever live event on the BBC (I can't give you streaming stats, sorry).
On my logging pipeline (subset of BBC logs):
- Peak: 2.2M row/s (105GB/s)
- Total: 20.1B rows (16.4TB), 2.7M log files
- 0 write fails
- 0 duplicated rows
- Peak: 1035 Cloud Run containers @ 2x CPU each ∴ 2070 CPUBest bit is that we did precisely nothing, it all just worked.
-
Inspired by @onlinegoddess's recent #blog post about (respectful) #webstats collection, here's my own 30-year journey through collecting a little, then a lot, then less, and now basically no #analytics data about my website visitors -
-
Inspired by @onlinegoddess's recent #blog post about (respectful) #webstats collection, here's my own 30-year journey through collecting a little, then a lot, then less, and now basically no #analytics data about my website visitors -
-
Super busy night of streaming for the BBC yesterday - but that wasn't *quite* as busy as Tuesday (ITV broadcast the England match yesterday).
I can't give the streaming numbers but we hit just under 1 million log lines per second on my logging system - which only handles a subset of the media streaming logs.
I wonder how busy the first England match we broadcast will be?
-
Super busy night of streaming for the BBC yesterday - but that wasn't *quite* as busy as Tuesday (ITV broadcast the England match yesterday).
I can't give the streaming numbers but we hit just under 1 million log lines per second on my logging system - which only handles a subset of the media streaming logs.
I wonder how busy the first England match we broadcast will be?
-
Turns out a load of people in the UK set their alarm for 07:00 - closely followed by 07:17 then 08:11 (which is a good deal higher than 08:00).
These are the RPS stats for the BBC service which (re)validates authentication sessions.
You can also clearly see a spike around 09:30 when, presumably, people have looked through email and check the news. Then there's the spike for the 13:00 news/lunch break.
We're all creatures of habit, it's the same *every* day.
-
Turns out a load of people in the UK set their alarm for 07:00 - closely followed by 07:17 then 08:11 (which is a good deal higher than 08:00).
These are the RPS stats for the BBC service which (re)validates authentication sessions.
You can also clearly see a spike around 09:30 when, presumably, people have looked through email and check the news. Then there's the spike for the 13:00 news/lunch break.
We're all creatures of habit, it's the same *every* day.
-
83% of requests (globally) to our apex homepages (bbc.co.uk/ and bbc.com/) are from "Python Requests".
That's 84 million requests per day.
In the words of Bricktop...
-
83% of requests (globally) to our apex homepages (bbc.co.uk/ and bbc.com/) are from "Python Requests".
That's 84 million requests per day.
In the words of Bricktop...
-
We monitor how much traffic we get from each country per day to check errors/blocking etc. (as I've previously mentioned).
Big drop-off recently from Angola & as usual it's an ISP (TV Cabo) changing their "is the internet working" config.
That prompted me to (again) look at how much we're serving to Fortigate "is the internet working" checks per month, roughly:
- 109 million web pages
- 10 Terabytes of egressWhat a waste.
Make `head` requests rather than `get`, please!
-
We monitor how much traffic we get from each country per day to check errors/blocking etc. (as I've previously mentioned).
Big drop-off recently from Angola & as usual it's an ISP (TV Cabo) changing their "is the internet working" config.
That prompted me to (again) look at how much we're serving to Fortigate "is the internet working" checks per month, roughly:
- 109 million web pages
- 10 Terabytes of egressWhat a waste.
Make `head` requests rather than `get`, please!
-
Wow, as reported elsewhere, traffic from Iran to www.bbc.co.uk & www.bbc.com has dropped to virtually nothing as of yesterday.
Only 3 ASNs have made any requests at all today:
- iran information technology company pjsc
- netiface limited
- m247 europe srl (unsure if this is *really* in Iran or not)We'd normally see about 200k requests/day from Iran and today we've had just over 600.
-
Wow, as reported elsewhere, traffic from Iran to www.bbc.co.uk & www.bbc.com has dropped to virtually nothing as of yesterday.
Only 3 ASNs have made any requests at all today:
- iran information technology company pjsc
- netiface limited
- m247 europe srl (unsure if this is *really* in Iran or not)We'd normally see about 200k requests/day from Iran and today we've had just over 600.
-
Gemini Protocol Deployment Statistics
#HackerNews #GeminiProtocol #Deployment #Statistics #GeminiProxy #WebStats #InternetTrends #TechNews
-
Gemini Protocol Deployment Statistics
#HackerNews #GeminiProtocol #Deployment #Statistics #GeminiProxy #WebStats #InternetTrends #TechNews
-
~51,000 T-Mobile USA IP addresses make a total of 13 million requests to https://bbc.com/ using `python-requests`.
Being on the internet is fun.
-
~51,000 T-Mobile USA IP addresses make a total of 13 million requests to https://bbc.com/ using `python-requests`.
Being on the internet is fun.
-
3% of requests to www.bbc.co.uk & www.bbc.com have a `user-agent` of OKHttp.
Part of that 3% is a single IP in Turkey which is making nearly 3 million request per day using OKHttp on it's own.
Being on the internet is fun.
-
3% of requests to www.bbc.co.uk & www.bbc.com have a `user-agent` of OKHttp.
Part of that 3% is a single IP in Turkey which is making nearly 3 million request per day using OKHttp on it's own.
Being on the internet is fun.
-
Page views± on the Celebrity Traitors Live Page as a global heatmap.
We had requests from 136 countries all-in.
Didn't watch any of it myself but folks seemed to enjoy it - was definitely a big chunk of traffic.
± from "likely humans" - not obvious bots
-
Page views± on the Celebrity Traitors Live Page as a global heatmap.
We had requests from 136 countries all-in.
Didn't watch any of it myself but folks seemed to enjoy it - was definitely a big chunk of traffic.
± from "likely humans" - not obvious bots
-
-
-
Hmm chatgpt ser ud til at holde fobindelsen til min hjemmeside i live meeeeget længe. Til sammenligning er besøg fra copilot og perplexity normal <1sek.
#webstats #AIscrapers #weird #danskertrut -
Hmm chatgpt ser ud til at holde fobindelsen til min hjemmeside i live meeeeget længe. Til sammenligning er besøg fra copilot og perplexity normal <1sek.
#webstats #AIscrapers #weird #danskertrut -
What is this? 2021?
-
What is this? 2021?
-
The BigQuery plugin for Grafana now (since v3) supports querying via the BigQuery Storage API.
We updated our plugin yesterday so I've started enabling the Storage API (it's a toggle in the query editor).
So far, I am seeing the Storage API being *significantly* faster, my estimate would be that it's at least 2-3x faster. Should also be cheaper, AFAICT.
https://grafana.com/grafana/plugins/grafana-bigquery-datasource/?tab=changelog
-
The BigQuery plugin for Grafana now (since v3) supports querying via the BigQuery Storage API.
We updated our plugin yesterday so I've started enabling the Storage API (it's a toggle in the query editor).
So far, I am seeing the Storage API being *significantly* faster, my estimate would be that it's at least 2-3x faster. Should also be cheaper, AFAICT.
https://grafana.com/grafana/plugins/grafana-bigquery-datasource/?tab=changelog
-
Unsure if it's in other SQL ecosystems but I *love* the wildcard syntax BigQuery has - makes it (intentionally) super easy with our data taxonomy.
If you have a load of tables inside a dataset, you can query across all of those tables with e.g.:
```
select col1, col2 from `some_dataset.*` where ...
```
We have a dataset per "service" (CDN, load balancer etc.) so if you want to query across everything on that services, it's nice and simple. -
Unsure if it's in other SQL ecosystems but I *love* the wildcard syntax BigQuery has - makes it (intentionally) super easy with our data taxonomy.
If you have a load of tables inside a dataset, you can query across all of those tables with e.g.:
```
select col1, col2 from `some_dataset.*` where ...
```
We have a dataset per "service" (CDN, load balancer etc.) so if you want to query across everything on that services, it's nice and simple. -
Essex County Council makes nearly 1 million requests per day to the ancient and creaky news.bbc.co.uk (which doesn't even work over HTTPS).
Edit: With an empty user agentWhat's the betting this is *yet again* "is the internet working" from some crufty old firewall?
-
Essex County Council makes nearly 1 million requests per day to the ancient and creaky news.bbc.co.uk (which doesn't even work over HTTPS).
Edit: With an empty user agentWhat's the betting this is *yet again* "is the internet working" from some crufty old firewall?
-
I've been looking at the logs for our zone apexes today (bbc.co.uk & bbc.com - which are simply redirects to their www equivs).
Good to see this is getting used for so many valid cases...oh, wait. FML
-
I've been looking at the logs for our zone apexes today (bbc.co.uk & bbc.com - which are simply redirects to their www equivs).
Good to see this is getting used for so many valid cases...oh, wait. FML
-
I've been working on an automated triager for the frequent volumetric DDOS we see against www.bbc.com & www.bbc.co.uk.
The idea is to use our edge access logs (stored in BigQuery) to isolate & describe the attack traffic then recommend any additional mitigations/filters etc. It also gives us a database of DDOS metrics/sources we can reference.
Obviously I had to add the obligatory pew-pew map.
-
I've been working on an automated triager for the frequent volumetric DDOS we see against www.bbc.com & www.bbc.co.uk.
The idea is to use our edge access logs (stored in BigQuery) to isolate & describe the attack traffic then recommend any additional mitigations/filters etc. It also gives us a database of DDOS metrics/sources we can reference.
Obviously I had to add the obligatory pew-pew map.
-
The % of user agents which are known bots or have "bot" in their UA string yesterday (a normal Monday) were:
www.bbc.co.uk: 3.0%
www.bbc.com: 8.7%That's more than 33 million web pages in total.
-
The % of user agents which are known bots or have "bot" in their UA string yesterday (a normal Monday) were:
www.bbc.co.uk: 3.0%
www.bbc.com: 8.7%That's more than 33 million web pages in total.
-
What are people using to serve a static website created with hugo? My preference is for a docker image, and for the server to include some statistics on hits.
-
We alert on total daily requests per country to www.bbc.co.uk & www.bbc.com. This is done is by comparing daily total requests per day by country (from GeoIP) versus a % of the 30 day median value.
Recently, we've had a number of African countries which've fired alerts & after triaging, they all had a single-digit number of BBC World Service articles get super popular for a few days which skewed the median. No idea why. Any ideas?
Example: https://www.bbc.com/storyworks/specials/rewriting-cancer/ -
We alert on total daily requests per country to www.bbc.co.uk & www.bbc.com. This is done is by comparing daily total requests per day by country (from GeoIP) versus a % of the 30 day median value.
Recently, we've had a number of African countries which've fired alerts & after triaging, they all had a single-digit number of BBC World Service articles get super popular for a few days which skewed the median. No idea why. Any ideas?
Example: https://www.bbc.com/storyworks/specials/rewriting-cancer/ -
Just raised the 1000th Pull Request on the logging pipeline I run (which ingests our CDN and client telemetry/error logs).
Feels like quite a milestone and since the repo has been active for just over 3 years old, that's over 300 Pull Requests per year which averages out to more than one Pull Request every working day.
-
Just raised the 1000th Pull Request on the logging pipeline I run (which ingests our CDN and client telemetry/error logs).
Feels like quite a milestone and since the repo has been active for just over 3 years old, that's over 300 Pull Requests per year which averages out to more than one Pull Request every working day.
-
Seems a lot of people are interested in the US election...here's our "last 30 days of total requests per ASN" for "other" (total for all ASNs outside the top 100).
Requests yesterday were more than 2x the mean value over the previous 28 days.
#BBC #USElection #WebStats -
Seems a lot of people are interested in the US election...here's our "last 30 days of total requests per ASN" for "other" (total for all ASNs outside the top 100).
Requests yesterday were more than 2x the mean value over the previous 28 days.
#BBC #USElection #WebStats -
Man Utd have sacked their manager, Erik Ten Haag.
The "Breaking News" article doubled our request rate on www.bbc.co.uk.
-
Man Utd have sacked their manager, Erik Ten Haag.
The "Breaking News" article doubled our request rate on www.bbc.co.uk.
-
We monitor traffic to www.bbc.co.uk & www.bbc.com per country & got alerts that daily requests from Angola have dropped off loads recently.
Looking at the Angola traffic split by network AS. AS36907 traffic looks suspicious! Spidey sense triggered...the "before" traffic was *way* too consistent.
Digging in to the logs, looks like they removed their Fortigates on 6th Sept. which'd been sending 343k req/day for www.bbc.co.uk/ , every single day!
Gotta love being on the internet!
-
We monitor traffic to www.bbc.co.uk & www.bbc.com per country & got alerts that daily requests from Angola have dropped off loads recently.
Looking at the Angola traffic split by network AS. AS36907 traffic looks suspicious! Spidey sense triggered...the "before" traffic was *way* too consistent.
Digging in to the logs, looks like they removed their Fortigates on 6th Sept. which'd been sending 343k req/day for www.bbc.co.uk/ , every single day!
Gotta love being on the internet!
-
Interesting.
I get traffic reports from both Google and Bing on our website's performance. I also get raw traffic data from Cloudfare.
Google keeps saying my traffic is down each month. Bing right now is reporting HIGHER numbers than Google. Google keeps pushing adsense on us as something we should "increase use of".
I wonder if us opting out of Google's ad trackers, and adsense, in February this year with our site redesign is the cause. ;) (we still have the most basic analytics stuff, via Yoast SEO, running).
Meanwhile, Cloudfare reports real numbers. We had 440K uniques in July. Google says we had 135K. (6 mo ago they said we had around 210K). Bing 144K. Our site logs? (after bots, etc weeded out): 442K.
What's really interesting is that Bing reports we have higher traffic than Google does. This is the first month that's happened.
DON'T TRUST GOOGLE's ANALYTICS.
#google #webanalytics #googleanalytics #webstats #webtraffic
-
Interesting.
I get traffic reports from both Google and Bing on our website's performance. I also get raw traffic data from Cloudfare.
Google keeps saying my traffic is down each month. Bing right now is reporting HIGHER numbers than Google. Google keeps pushing adsense on us as something we should "increase use of".
I wonder if us opting out of Google's ad trackers, and adsense, in February this year with our site redesign is the cause. ;) (we still have the most basic analytics stuff, via Yoast SEO, running).
Meanwhile, Cloudfare reports real numbers. We had 440K uniques in July. Google says we had 135K. (6 mo ago they said we had around 210K). Bing 144K. Our site logs? (after bots, etc weeded out): 442K.
What's really interesting is that Bing reports we have higher traffic than Google does. This is the first month that's happened.
DON'T TRUST GOOGLE's ANALYTICS.
#google #webanalytics #googleanalytics #webstats #webtraffic
-
#Canadian #media - well, the #legacy ones at least - are up in arms over the fact that #Meta / #Facebook is planning to block #news from appearing in #feeds in #Canada.
You managed to get a #law passed that means they would have to #pay you for the privilege of #linking to you, didn't you?
Now they get to choose whether to continue linking to you, or not.
You can't have it both ways with "You must link to us, but you must pay us to link".
I think they're in for a rude #webstats awakening.
-
#Canadian #media - well, the #legacy ones at least - are up in arms over the fact that #Meta / #Facebook is planning to block #news from appearing in #feeds in #Canada.
You managed to get a #law passed that means they would have to #pay you for the privilege of #linking to you, didn't you?
Now they get to choose whether to continue linking to you, or not.
You can't have it both ways with "You must link to us, but you must pay us to link".
I think they're in for a rude #webstats awakening.
