This is probably the most visited hot spring area in Iceland, but don't worry - it's quite large. Reykjadalur ("Steam Valley") is located just 45 minutes south-east of Reykjavik and has a warm river running right through it. From late spring to early fall the water is usually around 38-40°C / 100-104°F and it takes most people around 1 and 2 hours to get there (one way), depending on your speed (it's around 4km / 2.5 miles, one way).
There's no admission fee or permits - you only need to pay for parking if you park at the main parking spot.
QUICK NOTES:
What it is: A beautiful valley with a geothermal river running through it - great for bathing.
Best season: Summer (late spring and early autumn are still great and even winter, it's just a bit harder getting there during winter and your time frame is shorter).
Best time of day: Any but to beat the crowds I would probably do it pretty early or late in the evening, depending on the season.
Nice to know: It's free of charge and with no real facilities (except for wooden boardwalks and some really basic changing partitions). So if you need to do a number two - please do it before the hike!