#numericalaccuracy — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #numericalaccuracy, aggregated by home.social.
-
Alright, future engineers!
**Round-off Error:** Error from storing exact numbers with limited precision in computers.
Ex: 1/3 stored as 0.33333.
Pro-Tip: Use double precision & re-order ops to minimize accumulation in long calculations!
#NumericalAccuracy #FloatingPoint #STEM #StudyNotes