#mgcvchat — Public Fediverse posts

Let's try this again outside of the heat wave and weekend:

I'm looking for help with #rstats, specifically larger #bam / #gam in #mgcv with NCV.

It's running, but it has been about 60h by now, minimal CPU usage, and steadily climbing RAM usage.

Any pointers how to speed things up are welcome!

The #bam in question:

```
gam_roi_treatment_TW <- bam(
cumulative_time_in_roi ~
s(time_point_s, bs = "tp", k = 10) +
s(time_point_s, treatment, bs = "sz", k = 10) +
s(time_point_s, sex, bs = "sz", k = 10) +
s(time_point_s, strain, bs = "sz", k = 10) +
s(time_point_s, treatment, sex, bs = "sz", k = 10) +
s(time_point_s, treatment, strain, bs = "sz", k = 10) +
s(time_point_s, sex, strain, bs = "sz", k = 10) +
s(time_point_s, treatment, sex, strain, bs = "sz", k = 10) +
s(age_baseline, bs = "tp", k = 10) +
s(time_point_s, ID, bs = "fs", k = 10),
data = roi_no_talad,
family = tw(),
select = FALSE,
method = "NCV",
nei = roi_notalad_nei,
control = gam.control(ncv.threads = N_THREADS)
)
```

#mgcvchat

https://fediscience.org/@volephd/116821580992514861

made some updates last week to my GAM blog: adaptive smoothing, now with plots of the smoothing parameter function

https://calgary.converged.yt/articles/adaptive_smoothing.html#bonus-plotting-the-smoothing-parameter-over-the-covariate

big thanks to Philip Dixon who asked an interesting question!

#rstats #mgcvchat

📈 Yes you can do that in mgcv update

big thanks to Zachary Susswein for spotting that my code was out of date in my neighbourhood cross-validation examples: https://calgary.converged.yt/articles/ncv.html https://calgary.converged.yt/articles/ncv_timeseries.html

They are now up-to-date, as is the helper package mgcvUtils: https://github.com/dill/mgcvUtils

#mgcvchat #mgcv

new (out for a while but sitting in my browser from before Christmas) paper in Biometrika from Benjamin Säfken, Thomas Kneib and Simon Wood on smoothing parameter degrees of freedom

Green OA @ Edinburgh https://www.pure.ed.ac.uk/ws/portalfiles/portal/475921820/asae052.pdf

#mgcvchat #mgcv

#mgcv mini-lifehack:

(assuming you have multithreading enabled) you can get a rough idea of what's happening when fitting a big model by looking at your CPU usage. If only 1 core is being used, the model is still "building" (assembling of design/penalty matrices), once you switch to all cores, then you're actually fitting the model. Sometimes that first model construction phase can take a long time (with a very big model), so it'll probably take a very very long time to fit. So buckle-up.

#mgcvchat

oh, hey, I reviewed this! gratia is an excellent tool for mgcv users! Thanks @gavinsimpson!

https://fosstodon.org/@joss/113691283592026369

#mgcvchat

Checkout my soundcloud https://arxiv.org/abs/1902.01330 #mgcvchat

my mgcv Wrapped 2024

top 5 basis functions:

1. thin-plate regression splines
2. B-splines
3. soap film smoother
4. cubic cyclic splines
5. random effects (psych!)

#mgcvchat

spending some more time thinking about neighbourhood cross-validation in #mgcv (see original post here: https://calgary.converged.yt/articles/ncv.html), but for time series.

Pretty nice to be able to get back to a yearly trend here without needing to specify an autoregressive structure. We just need to specify a cross-validation scheme and the autocorrelation is "dealt with" during fitting.

Full post on this soon. #mgcvchat #rstats

I've been writing-up some bits on un/under-documented parts of mgcv. Here's a bit of chat about the new "neighbourhood cross-validation" method that was uploaded to arXiv a wee while ago: https://calgary.converged.yt/articles/ncv.html

More to come on this, including some details on how to setup neighbourhoods in practice.

(Please @ me with errors/typos etc)

#mgcvchat

Preprint from Simon Wood on the new cross-validation smoothness estimation in #mgcv: https://arxiv.org/abs/2404.16490. It's a neat performant + data-efficient way to estimate GAMs based on complex CV splits (like spatial/temporal/phylo ones).

See ?NCV in latest {mgcv} for examples (https://cran.r-universe.dev/mgcv/doc/manual.html#NCV)

I might write a helper to convert {rsample}/{spatialsample} objects into mgcv's funny CV indexing structure.

#rstats #ml #tidymodels #mgcvchat @MikeMahoney218 @gavinsimpson @ericJpedersen @millerdl

Treat today at Edinburgh Uni stats seminar: Emiko Dupont (Bath) talking about her new work on spatial confounding (https://arxiv.org/abs/2309.16861) #statschat #statistics #mgcvchat

I did a new thing with the talk I gave at Lancaster yesterday and make a page of notes to accompany the slides (to make the slides less busy), probably doesn't make much sense if you don't have me yakking on in the background, but in case it's useful #mgcvchat

https://converged.yt/talks/lancaster-gam-2023/

a bit of fiddling this afternoon for a colleague got me to fitting penalized ordinal covariates in mgcv via a 1D Markov random field. No idea if this is a good idea! (Or if someone else has already done this...)

https://gist.github.com/dill/96763b6f57ac98229e62c02bac141236

#mgcvchat

I am once again asking you to read the documentation on specifying cyclic splines in mgcv before just writing bs="cc" and walking away

#mgcvchat