That looks like years of scuffing with dirty hands/gloves. The only way you'll get that out is with a cut polish. Possibly even sanding. But by the looks of it, it needs a repaint anyway.

Posted by: @nicktumi

Scotty, love your channel. Its helped me fix up my 2006 Honda Ridgeline with 215k miles. I love this truck. But my paint looking horrible. I have dirt/oil on the door from previous owner and am trying to figure out how to get it out. I have not seen any fixit videos online that have this issue. Maybe I could send you photos.

You might try an automotive grade clay bar. Professionals use them to rid any residue before they wax cars. 

Agree with @justin-shepherd about the clay bar.  That would be the gentlest option.

If that doesn't work, a little rubbing compound with a buffer should do it; follow it with polishing compound and a good wax.