I just had this same problem on my '05 6.0, and it finally got to the point where it wouldn't start at all. I've still got the dieselcare esp warranty on it, so I had it hauled into the dealer. The problem was an oil leak at the STC fitting on the Hi-Pressure Oil Pump. Up until this repair, I had the same problem you're having for close to a year. I even took the truck in to them in the winter time complaining of it, but the truck didn't act up when they had it. The truck has started fine ever since.
I'm not sure how similar my 6.0 is to your 6.4, but thought I'd share anyway since this problem frustrated me too.