We got a taxi from Darling Harbour to the airport last month and the taxi driver asked which route did we want to go. I couldn't understand much of what he said but he asked if we wanted to go the quickest way by the freeway (I think). I guess he asked because this might have included tolls? We said yes and it seemed fairly quick - only about 20 mins - and cost $37 which was cheaper than expected - I had thought $50 as mentioned above. This was 11am mid-week.
On a previous visit there a few years ago (again from DH to the airport) it took about 30/40 mins at lunchtime and seemed to take a lot longer getting out of the city, i.e. not quickly on to the freeway.
This will make more sense to the Aus contributors but thought it might help if your taxi driver asks which way you want to go.