Just realised you said a Sunday or a Monday: Sunday, no question,
Also your route using the M4 is not a good idea, assuming you go via Oxford then this is 130 miles, vs 80 for the direct route. Check the AA Route Planner at http://www.theaa.com/route-planner/index.jsp
If you go on a Monday, no matter what time you leave you will have working traffic to deal with, which will increase the time * 1.5 and the stress * 100. Leave at 07:30 on Sunday the roads will be empty, you should be here by 10:00.
Some city centre car parks have time limits - I suggest you use the Avon Street Car Park, which is £12.50 for the whole day. Head straight for the Roman Baths, before the queues build up.
Are you really unable to spend the night in Bath ? you may be in the car for > 5 hours in a single day. I have looked at the trains, not an option, sadly.