The Frenchman and fellow substitute Aaron Ramsay scored in the final seven minutes as Arsenal twice came from behind, the Welshman levelling two minutes before Giroud headed the decider.
Giroud had scored 12 Premier League goals in just 11 starts last season, but Lacazette was selected for Arsenal's opener and netted inside two minutes.
Jamie Vardy then capitalised on bad defending to twice put the Foxes ahead either side of Danny Welbeck's equaliser before Arsene Wenger sent for the cavalry.
Giroud then sent the previously despondent Arsenal fans wild when he headed the 85th minute victor from a corner, Leicester keeper Kasper Schmeichel only managing to turn the ball onto the underside of the bar before it bounced over the line.
And speaking after the game, the 30-year-old confirmed his intention to stay and fight for his place at Emirates Stadium.
"Giroud? He's fantastic", Wenger said of the France global, who usually started on the bench last season and had been linked with a move away from The Emirates.
"You know, I must say I opened the door for him at some stage because I know that it's important for him to play". He's a fantastic player, but as well he has a big weight in this squad.
"No. It's a determined no", he said.
"You win 4-3, you're more happy than concerned, you know", he added.
"We had some avoidable goals which we conceded and overall we have to work on that", he said referring to poor marking in the by his team's defence.
"You come away from home, score three goals and are almost there and then concede two.bitterly disappointed", he said. I then wanted Wilfred and Daniel Amartey to get up close to their midfield players and for Iheanacho to have the space in the last ten minutes to get on the back of Granit Xhaka. "We were a threat and played football and contributed to a thoroughly entertaining game".