Despite a second-half siege on the hosts' goal, Sunderland could not grab an equaliser following Gardner's first-half strike which halved the deficit and they have now recorded just two wins in their last 22 Barclays Premier League fixtures.
"We're frustrated but we have a tough game coming up against Chelsea now and we need to take hold of games from the start, not just from the start of the second half," he said.
"We've just got to take our chances. It's taking us a while to get into games. We need to stand up and be counted and take our chances. We can't keep doing this after games, saying 'it'll come, it'll come'. We need to make it come."
Norwich had stormed ahead with goals from Sebastien Bassong and Anthony Pilkington and managed to keep the Black Cats at bay to ruin Martin O'Neill's one year anniversary as Sunderland boss.
Gardner came close in the second half, hitting the post with a free-kick and seeing another effort cleared off the line, but revealed the players need to do more to turn things around with the club languishing in 17th place in table.
Former England Under-21 international Gardner does think Sunderland have started to show their quality in patches during games of late but is hoping they can start games at a quicker pace in the coming weeks.
"We started off slowly again against Norwich and found ourselves two goals down," he said. "Not many teams play Norwich off the park at home - which is what we did in the second half. It's there, we can do it but we need to do it from start to finish."
Whilst Sunderland continue to struggle for form and consistency Norwich have quietly gone about their business and Chris Hughton's side are now unbeaten in their last eight league matches.
Since losing 4-1 at Chelsea on October 6 the Canaries have picked up points against the likes of Everton and Southampton and recorded memorable victories over Arsenal and leaders Manchester United at Carrow Road.
Source: PA