Ipswich won the Championship fixture against Barnsley, but it was revealed that McCarthy had already agreed to step down before the game. It was originally expected that he would leave Portman Road when his contract expired at the end of the season.
"I spoke with Marcus [Evans] at the weekend and we agreed that it was in the best interests for me, the players and the club that I move on now," McCarthy said.
57-year-old Klug will take charge for the four remaining games of the season.
The retired midfielder has twice previously served as interim boss in 2009 after the sacking of Jim Magilton. He had served as assistant manager to the Northern Irishman and later took up his role with the club's academy.
Klug's first game in charge will see Ipswich take on Nottingham Forest on Saturday.