It may be the moisture content that's making it happen.
I smoke a lot of SG flakes and we all know how they come.

Usually when I get home from work, I'll lay out a flake. Then about 3 hours later I'll rub it out and smoke it. It may just not be enough time.

I'll also watch out for breathing into the stem like Tobias mentioned. That likely happens often as well.

I'll take @Shemp Jizzle advice and try to dry it out until it looks too dry. I don't know why I have such a fear of that. For some reason I think if I dry it too much the flavor will be gone.