"In other words, it may not be that the whole Taliban comes over in one fell swoop", he said, according to Reuters.
Afghan President Ashraf Ghani last month offered talks without preconditions to the Taliban insurgents, who have previously said they would talk only with Washington about a possible peace agreement.
"There is interest that we've picked up from the Taliban side", Mattis told reporters shortly before landing in Kabul, saying the signs date back several months.
He said there were "elements of the Taliban clearly interested in talking with the Afghan government", while admitting that not all the rebels would be prepared to do so.
Taliban fighters control large parts of the country and thousands of Afghan soldiers and civilians are being killed every year, with a divided Kabul government unable to take effective action. The Associated Press reported that security concerns were so high that reporters traveling with Mattis were not allowed to publish stories until his party had moved from the Kabul airport to the US -led military coalition's headquarters.
The United States past year stepped up its military assistance to Afghanistan, notably through a sharp increase in air strikes, with the aim of breaking a stalemate with the insurgents and forcing them to the negotiating table. Mattis told reporters that he thinks victory is still possibly defined as a political settlement with the Taliban.
Last week, senior US intelligence officials gave Congress a decidedly mix forecast for Afghanistan.
Mattis has said that the US goal is to enable Afghan forces to weaken the Taliban to the point where the Afghans can manage their own security.
The Taliban urged Washington last month to begin talks to end nearly 17 years of war in Afghanistan, which suggests they want to explore dialogue.