SWAT teams have been deployed to the area.
Gloria Parker, an eye witness, told the broadcaster that she and other people began running for their lives when they heard 10 to 15 bangs that they initially thought were firecrackers.
There is no indication a suspect has been detained.
Earlier Sunday, Santa Clara Valley Medical Center spokeswoman Joy Alexiou said the hospital has received two victims from the shooting and expects three more.
"I remember seeing a whole stampede of people running, jumping fences", he said, according to the network.
Festival officials did not immediately respond to request for comment.
The annual festival was on its final day.
"What's going on?" a woman can be heard asking on one video. San Jose Mayor Sam Liccardo, a Democrat, tweeted that the city's police and fire department were sent to Gilroy to help with what he called a "horrific shooting".
Few other details were immediately available of the incident on Sunday afternoon (around 0100 GMT on Monday). "And there were people on the ground, crying", she told the newspaper.
"I saw him for approximately one second, he was firing with intent to hit people", he said.
California Sen. Kamala Harris said on Twitter that her office is "closely monitoring the situation".
Meanwhile, President Donald Trump sent a tweet saying: "Law Enforcement is at the scene of shootings in Gilroy, California".
Thousands of people and more than 100 officers had spent Friday and Saturday at the annual "Old Timers' day" celebration.