Generalized additive mixed model