Loosing 20 minutes a day is certainly not normal for that movement. Plus or minus 30 seconds a day is probably within specs, and most are probably more accurate than that.
I don't think it's been magnetized - that usually causes the watch to run fast on the order of one to three hours/day.
I would suspect that there is some very slight friction somewhere, or perhaps the hands are touching (perhaps the minute hand just contacts the seconds hand?) Wearing it causes it to over come that friction. You could try setting it done in a different orientation to see if that makes a difference, but that's just a short term work around.
If the watch really is keeping much different time when wearing it vs. setting it down, I think it's time to get it serviced. Should be one of the simpler movements to have serviced and then you're good for another three to five years.