The deportees could have been descendants of members of the Windrush generation, named after one of the first ships that brought Caribbean migrants to the United Kingdom in 1948, after the Second World War.
"We have found 63 cases where individuals could have entered the United Kingdom before 1973 ... who have been removed or deported", he said.
Mr Javid denied there was a "systemic" problem in the Home Office, but accepted that in Windrush cases people had faced "too large a burden" in proving long-term residency. Some have been detained and may have been wrongly deported.
Although Mr Javid stressed the figure was provisional, his admission gave an indication of the scale of the exercise facing the Home Office.
He said: "I've asked officials to be absolutely certain and thorough and check over every record and make sure".
Mr Javid told the Home Affairs Select Committee that the exercise has so far found 63 people who may have arrived in the United Kingdom from the Caribbean before 1973 and have been removed or deported.
Only last week immigration minister Caroline Nokes said she was still not aware of a single wrongful deportation of a Windrush citizen, despite Home Office chiefs admitting they knew of such cases.
The Labour MP David Lammy said: "This is the worst human rights and home affairs crisis in my time in politics". The UK's opposition claims the crisis was the outcome of a "hostile environment" created by the Home Office when it was headed by Theresa May from 2010 to 2016.
Sadiq Khan responded to the news on Twitter. "Govt [sic] must urgently provide answers on exactly how many have been affected, and what action is being taken to right this wrong".
The Windrush migrants arrived between the late 1940s and 1973, mainly from the Caribbean, but some have been threatened with deportation in recent years.