#cardinality — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #cardinality, aggregated by home.social.
-
The cardinality wall: The hidden data bottleneck for LEO constellations
https://atlas.whatip.xyz/post.php?slug=the-cardinality-wall-the-hidden-data-bottleneck-for-leo-constellations
<p>The satellite industry is entering an era defined by scale
#constellations #cardinality #bottleneck #satellite -
Hi @mariusor,
json-ld has this whim to be completely lost about #cardinality. -
Hi @mariusor,
json-ld has this whim to be completely lost about #cardinality. -
#KnowledgeSharing post:
Yesterday I talked about #metrics #cardinality and churn, and how they can affect #Prometheus server performance.
Today I want to talk about unbounded cardinality, which is when there is no fixed maximum value for the cardinality of a time series. Unbounded cardinality is not only detrimental to Prometheus server performance, but it also introduces a lot of metrics noise, leading to degraded #observability.
A common example of unbounded cardinality is mapping an HTTP server's request path to a label. At a glance this may seem like a good idea, for example to identify which requests are taking the longest to serve. However, this only works if the received requests represent valid traffic. A client that is scraping the site or a malicious actor performing reconnaissance will throw the dictionary at the server, and a small site serving a dozen or so legitimate pages will produce millions of idle time series from all these requests that were seen only once.
My recommendation for HTTP request metrics is to only produce aggregate metrics. For example, instead of producing metrics for every request path, instead produce metrics for the routes configured on the server. This does not mean that observability granularity needs to be sacrificed: Metrics can be complemented with #traces for each HTTP request, and aggressive sampling can be applied to the traces so that only the "interesting" ones are kept - the exceptions, requests taking too long, with too big of a payload, etc.
-
#KnowledgeSharing session:
When dealing with low performance problems in #Prometheus servers, the two most common causes are high cardinality and high churn rate.
#cardinality represents how variable a metric is, as a combination of the variability of its labels. For example, a metric with five labels where each can have ten different values has a cardinality of 10**5. This means that in the worst case, your Prometheus server will be storing 100k data points per scrape for this time series alone. At this scale, every little optimisation counts. Reducing the variability of a single label by half reduces the cardinality of the time series by half, saving you 50k data points per scrape. Similarly, reducing the variability of each label by just one point makes the time series cardinality 9**5, which comes to just over 59k.
#ChurnRate is the rate in which old time series are replaced with new ones. For example, a workload that runs as a cron job and includes the PID as a label will churn every time the job executes, because it will be a new process. High churn rate tends to kill Prometheus server performance because it invalidates the in-memory caches of the server, leading to higher write costs.
One thing to point out is that churn rate and cardinality are not necessarily coupled. Yes, higher cardinality will lead to higher churn rate, but a metric that cycles labels between numbers 1 and 100 can still cause churn, and a series with high cardinality labels (such as VM names when running thousands of VMs) does not imply high churn if the labels stay consistent. However, time series that significantly impact Prometheus server performance do frequently show high cardinality and high churn rate at the same time.
-
#KnowledgeSharing session:
When dealing with low performance problems in #Prometheus servers, the two most common causes are high cardinality and high churn rate.
#cardinality represents how variable a metric is, as a combination of the variability of its labels. For example, a metric with five labels where each can have ten different values has a cardinality of 10**5. This means that in the worst case, your Prometheus server will be storing 100k data points per scrape for this time series alone. At this scale, every little optimisation counts. Reducing the variability of a single label by half reduces the cardinality of the time series by half, saving you 50k data points per scrape. Similarly, reducing the variability of each label by just one point makes the time series cardinality 9**5, which comes to just over 59k.
#ChurnRate is the rate in which old time series are replaced with new ones. For example, a workload that runs as a cron job and includes the PID as a label will churn every time the job executes, because it will be a new process. High churn rate tends to kill Prometheus server performance because it invalidates the in-memory caches of the server, leading to higher write costs.
One thing to point out is that churn rate and cardinality are not necessarily coupled. Yes, higher cardinality will lead to higher churn rate, but a metric that cycles labels between numbers 1 and 100 can still cause churn, and a series with high cardinality labels (such as VM names when running thousands of VMs) does not imply high churn if the labels stay consistent. However, time series that significantly impact Prometheus server performance do frequently show high cardinality and high churn rate at the same time.
-
In #Mathematics, there are different types of #Infinity, primarily categorized by their size or #Cardinality.
https://knowledgezone.co.in/posts/Infinity-in-Mathematics-67863206a7e475eb5e30404a
-
In #Mathematics, there are different types of #Infinity, primarily categorized by their size or #Cardinality.
https://knowledgezone.co.in/posts/Infinity-in-Mathematics-67863206a7e475eb5e30404a
-
Evolution of the notation for the empty set:
0 - George Boole 🇬🇧 (1847)
𝑂 - Georg Cantor 🇩🇪 (1880)
{} - Gottlob Frege 🇩🇪 (1882)
Λ - Giuseppe Peano 🇮🇹 (1889)
∅ - Andre Weil 🇫🇷 (1939)"Much later, my own part in these discussions earned me the respect of my daughter Nicolette, when she learned the symbol Ø for the empty set at school and I told her that I had been personally responsible for its adoption." - Andre Weil
-
Evolution of the notation for the empty set:
0 - George Boole 🇬🇧 (1847)
𝑂 - Georg Cantor 🇩🇪 (1880)
{} - Gottlob Frege 🇩🇪 (1882)
Λ - Giuseppe Peano 🇮🇹 (1889)
∅ - Andre Weil 🇫🇷 (1939)"Much later, my own part in these discussions earned me the respect of my daughter Nicolette, when she learned the symbol Ø for the empty set at school and I told her that I had been personally responsible for its adoption." - Andre Weil
-
Just released: Discrete Math Cheat Sheet by AviMathPerson
Download it free at http://www.cheatography.com/avimathperson/cheat-sheets/discrete-math/?utm_source=mastodon
Here's their description of it: Notes from a course I took in discrete math coving a variety of topics including set theory, relations, cardinality, etc.
@cheatsheets #CheatSheet #CheatSheets #math #functions #discrete #sets #relations #combinatorics #cardinality
-
from "Was Cantor Surprised?" by Fernando Q. Gouvêa
-
from "Was Cantor Surprised?" by Fernando Q. Gouvêa
-
How to make high cardinality work in time series databases: Part 1
https://last9.io/blog/how-to-make-high-cardinality-work-in-time-series-databases-part-1/
-
How to make high cardinality work in time series databases: Part 1
https://last9.io/blog/how-to-make-high-cardinality-work-in-time-series-databases-part-1/
-
Something a little different today on the channel: HyperLogLog!
It's one of my favorite algorithms, used to estimate cardinality of a set. Typically used in environments with very large datasets (spread across many servers in a cluster) where a true, accurate distinct count would be very expensive.
HLL uses a simple observation about coin flipping probabilities, and extends that to cardinality estimation. Really clever algo, and provides a very fast and compact datastructure with reasonably small errors (<2% across billions of unique elements, typically in just a few kb of memory).
https://www.youtube.com/watch?v=lJYufx0bfpw
#programming #algorithm #hyperloglog #cardinality #datastructures
-
Something a little different today on the channel: HyperLogLog!
It's one of my favorite algorithms, used to estimate cardinality of a set. Typically used in environments with very large datasets (spread across many servers in a cluster) where a true, accurate distinct count would be very expensive.
HLL uses a simple observation about coin flipping probabilities, and extends that to cardinality estimation. Really clever algo, and provides a very fast and compact datastructure with reasonably small errors (<2% across billions of unique elements, typically in just a few kb of memory).
https://www.youtube.com/watch?v=lJYufx0bfpw
#programming #algorithm #hyperloglog #cardinality #datastructures
-
Want to know how I significantly reduced resource usage on my Prometheus setup? :prometheus_logo:
Check it out in my latest article 👇
https://medium.com/@dotdc/prometheus-performance-and-cardinality-in-practice-74d5d9cd6230
-
Want to know how I significantly reduced resource usage on my Prometheus setup? :prometheus_logo:
Check it out in my latest article 👇
https://medium.com/@dotdc/prometheus-performance-and-cardinality-in-practice-74d5d9cd6230