Laugardalslaug is a great option for families looking for an authentic Icelandic experience. Icelander's go to the local pools instead of the pricier spas. This pool in particular is quite large and has a number of fun things for kids to enjoy.
Don't forget to shower naked before taking a dip—it's just an Icelandic thing for hygiene!