I've had very stressful music making experiences, but most are very satisfying. If a track is stressing me out, I might pause to write a new angry/stressed out one to cathart a bit, which always helps. I try to avoid deadlines and curb expectations where possible.
I always have a phase where I despise what I'm working on, after a honeymoon period. Perspective is everything. As you said, what stands in the way of enjoyment comes down to stress and fear of failure, so the best advice is to 1) track down the source of the fear (and this can even be explored via music, which is the natural language of emotion in my opinion), and 2) have fun: maybe try something new/different/weird, whether in music or in your life in general.
