What do I do about an unreliable contractor?

The drywaller showed up on time – amazing! He didn’t keep his measurements from when he bid the job so he took the measurements again, inspected the job site and pronounced it good to go with minor shimming. He drove off to pick up the supplies and came back with half of the drywall, which he unloaded and stored on our front porch. He said he’d be back the next day with the remaining dry wall and would stay and get going on hanging it. Meanwhile I cleared the path from the front porch up to the room…the disorganization begins. My second least favorite part of living on a construction site is moving things out of the way, which then interferes with living. My least favorite thing is the mess that construction makes, especially the fine white dust from sanding drywall. I need to plan things to do for the next week to live peacefully through this phase of the project!! Fortunately the kitchen will not be impacted so meals can proceed as normal.
One week later...
It is now a week later from the first day the drywaller showed up and the drywall is still not hung! This kind of unreliability is why we are DIYers! He shows up each work day with some story of important personal things he has to do and then puts in a couple of hours work. The project schedule is affected as we missed the "Drywall finished" milestone, which is a prerequisite for the rest of the tasks on this project! Meanwhile my schedule is seriously affected as I am taking off from work and juggling appointments to accommodate him and his “alleged” schedule. I had hired him before and he does excellent work but I forgot how unreliable he is with his promised timeline. So I’m stuck with a conundrum – If I voice my anger and frustration I risk igniting his temper and I’m in the house alone with him and his helper; if I continue to be nice, he most likely will continue to take advantage of the situation and blow me off. Hmm…what’s a girl to do?