The hot springs drew the Romans to the area. They built a town called Aquincum in what is now the north of the city. There's a museum and it's quite impressive. A lot of the ruins have been excavated.
The museum is quite small, but has some really interesting stuff. When they excavated the site, they found the remains of a water-powered organ that was played at public events. These were mentioned in ancient documents, but this was the first example that's been found.
Down the road from the museum is the ruin of the former arena.