#monitorama24 — Public Fediverse posts

Gathering data from lots of different sources, providing it to the model then processing the output through post processing steps? That sounds like an asynchronous workflow, USE TRACING ~ @cartermp lauding the benefits of tracing for deploying and iterating AI applications #monitorama #monitorama24 #observability

“Disintegration is a feature not a bug, of asynchronous systems, until you introduce telemetry and monitoring” ~ Johannes Tax from @grafana #monitorama #monitorama24 #observability

Johannes Tax at @grafana describing the pain of distributed tracing in asynchronous systems “Disintegrated telemetry: The pains of monitoring asynchronous workflows” #monitorama #monitorama24 #observability

“Name your metrics, alerts and dashboards with the language you would use if you were having a conversation about the system at lunch, not the cryptic defaults or uuid hostnames” ~ @danslimmon #monitorama #monitorama24 #observability

We’ve all heard the definition for observability, but, who is one? What is involved in determining? ~ @danslimmon presenting “No observability without theory” You need a theory of the system to build valid inferences
#monitorama #monitorama24 #observability

Julia Thoreson at Bloomberg sharing “Incident Management: Lessons from Emergency Services” breaking down how the lessons learned in emergency services can apply to incident management in technical systems #monitorama #monitorama24 #incidentmanagement

Pete Fritchman’s Takeaways on managing internal services effectively:
Internal Services impact customers
Leverage your observability tools
Talk to your internal customers
*APPLY SRE PRINCIPLES*

#Monitorama #monitorama24 #observability #SRE

“New hires are super value able in your internal customer interviews, they actually expect things to work and aren’t bitter yet” ~ Pete Fritchman #Monitorama #monitorama24 #observability #sre

“Treat your internal tooling outages like the most critical production outages, because they’ll always hit when you’re trying to recover from a critical production outage” ~ Pete Fritchman #monitorama #monitorama24 #observability

“The shoemaker’s children have no shoes - why SRE teams must help themselves” Pete Fritchman making the case for investing in watching the watchmen, and techniques for accomplishing it. #monitorama #monitorama24 #observability #sre

Hashmaps to counts work great for small sets, but what happens when you need to count sets larger than memory? You need HyperLogLog or Disjunctive Normal Form (CVM) ~ @phredmoyer #monitorama #monitorama24 #observability

“Use counters to count things” @phredmoyer providing some examples where counting things by processing petabytes of log or trace data is prohibitively expensive and justify spending a bit more on dealing with higher cardinality metrics.
#monitorama #monitorama24 #observability

Baggage is bad for your relationships, good for your service graphs. @kalyanaj makes the case for an arbitrary key value metadata store (baggage) to propagate through your services to enable controllability and observability use cases. #monitorama #monitorama24 #observability

“Distributed Context Propagation: How you can use it to Improve Observability, Test in Production, and more...” @kalyanaj explaining the importance of context in interpreting observability data #monitorama #monitorama24

“Every team has a different answer for discovering what the dependencies of their services are, some say firewall rules, some look at network flows, tracing gives us a uniform answer to this” ~ Sudeep Kumar #Monitorama #monitorama24 #observability #tracing

“We have so many microservices, people are always looking for an excuse to create more, and no one knows which ones they’re already dependent on” ~ Sudeep Kumar from Salesforce with “Tracing Service Dependencies at Salesforce”

#Monitorama #monitorama24 #observability #tracing

“Low cardinality in Prometheus and low cardinality in Clickhouse are vastly different things” - @colind in his talk “Experiments in Backing Prometheus with Clickhouse” #Monitorama #Monitorama24 #observability

“If we’re being honest, we all, generally, have a visceral reaction to people trying to get us to adopt new tools” ~ Noa Levi describing how a forced migration and the associated conversations between the engineering and observability teams facilitated adoption without adoption as a stated goal #Monitorama #monitorama24 #observability

Noa Levi presenting “How we tricked engineers into utilizing distributed tracing” on her experience getting tracing adopted at Strava

#Monitorama #Monitorama24 #Observability

“a little bit of work in reducing latency at the beginning of the data pipeline can provide orders of magnitude lower cost at the query end of the pipeline” ~ @djosephsen
#Monitorama #monitorama24 #observability

”We need to have a word about this ‘watershed’ metaphor for data” - @djosephsen presenting “Pugs, Poe's and pipelines; An engineering perspective on big-data streams for operational telemetry”
#Monitorama #Monitorama24 #data

Thai Wood breaking down the observability lessons from an aerospace story, a time-honored tradition here at #Monitorama “The complexity of success and failure: the story of the Gimli Glider”

#Monitorama24 #Observability

Ferris Ellis “Is Your Kernel Being Honest? Understanding & measuring low level bottlenecks” breaking down fundamental performance diagnosis in the Linux OS

#Monitorama #Monitorama24 #Observability

Dave McAllister presenting “The subtle art of misleading with Statistics”
“If you can’t find the bias in your sampling, look harder”

#Monitorama #Monitorama24 #Statistics

David Gildeh at Netflix with “The Observability Data Lake - 1 year on” incepting an update to his talk from last year

#Monitorama #Monitorama24 #Observability

@austinlparker introduces the ARS(e) theory, or CAP thereom for mere mortals, you don’t even get one:

#Monitorama #monitorama24 #otel

“It’s all just structured data, but it’s boring without aggregation” - @austinlparker on the core values of the #OTEL project, and why the wait for consensus on interoperable observability is worth the wait.

#monitorama #monitorama24 #OpenTelemetry

“The Hater’s Guide to Open Telemetry” with @austinlparker is up next, we made it three talks into #monitorama24 before the @xkcd Competing Standards comic made an appearance.

#Monitorama #otel

”All of the observability infrastructure in the world is just noise unless someone spends time developing an understanding of your system” ~ David Caudill There’s likely a 10:1 investment return when spending that time up front versus in the middle of the night when you’re troubleshooting an outage.

#monitorama24 #monitorama #constructionism

“Many times addressing DEBUG log noise is just a matter of education” - Alex Hidalgo “There are well defined POSIX standards for the various log levels (WARN, ERROR, DEBUG), many junior team members just aren’t aware of them”

#monitorama24 #monitorama #AlexHidalgo

First talk today is “Logs are good, actually” by Alex Hidalgo, “not all systems look like easily traced, high query per second systems”

#Monitorama24 #monitorama #AlexHidalgo