Swarzak, who missed 20 games with a mild right rotator cuff strain, gave up two runs on four hits and a walk while recording just two outs in Tuesday's 4-1 loss to the A's.
He fared better in his first appearance since coming off the DL on Friday, when he tossed a scoreless inning against the Mariners.
"He just needs to get some more pitching in," Gardenhire said. "He just needs to get the ball down. He was up with everything."