Stop, Take Step Back, And Remember What You Already Know