2004 Crown Vic blower motor won't work

I have a 2004 CVPI I have had about a year now and the blower motor works here and there.. It is coming into winter again and need it to work.. Pretty much what it does is it likes to not come on when you need it the most but leave it alone it will come on in a few minutes.. I have checked both the fuse and relay if anything wasn't making good contact and everything seems ok in the fusebox.. I pulled out another relay and stuck it in the blower motor spot and still didn't work.. Any info in the right direction would be great.. Do I need to be following wires,bad ground,low-medium-high switch or the motor itself or something I would never think of looking for? The motor spins fine after it comes on and will stay on all day after it gets past not wanting to come on.. I could probably figure it out in time and just started troubleshooting it today and really do not want it to turn into an all winter troubleshoot.. Thanks