2005 scion XB. 175,000 miles. Automatic. No codes came up when scanned. 

I've been having some issues with my car's acceleration recently.  At first I thought it just needed new spark plug and ignition coils so I replaced those.  Shortly after my car's starter died and I replaced that.  Now I have an issue where I start my car, drive down the street and then either stop at a stop sign or a streetlight.  When I go to accelerate, all of a sudden my car decides to sluggishly accelerate, I am pushing down on the gas pedal but it has a delay in giving any power.  It also seems like what power I do get is pretty low so instead of leaving the light or stop sign at a decent speed, I sluggishly pull out.  Potentialy creating a dangerous scenario for myself or others.

The kicker is, after 10 minutes or so my car runs perfectly fine as if it never had a problem in the first place.  Also the idle the entire time is steady with no power fluctuations.  I am thinking of replacing the fuel pump and filter because I suspect that may be the issue however I don't see and videos describing my problem.

Thanks.