Helicopters and boats are being used to rescue people still trapped by flooding in areas like Okayama Prefecture, where Kurashiki is located.
Some residents of Mabi had shrugged off the warnings, however, given the area's history of floods.
It's typhoon season in Japan, but this rainfall has been especially heavy - with as much as 10 centimetres falling every hour at its peak.
Officials in Kurashiki, Okayama Prefecture, reported Tuesday that many first aid emergency supply resources were left piled up near a bridge in a flooded region, consequently becoming a hindrance for Self-Defense Force personnel.
Many major highways and rail lines are still closed and tens of thousands of homes in Hiroshima are still without electricity or clean water.
Akira Tanimoto says his apartment narrowly survived the floods and mudslide at his residential complex over the weekend, and even if he wants to go back there with his wife and two pet birds, he can't because there is no water, power or food available.
Japanese government spokesman Yoshihide Suga said the whereabouts of 92 people is unknown, mostly in the southern area of Hiroshima prefecture.
In the message, Francis expressed his solidarity with all those affected, offered his encouragement to rescue crews and said he was praying for the dead and injured "and the consolation of all those who grieve".
Here are some of the factors that have contributed to the worst rain-related disaster in Japan in over two decades. Some 30,000 people remained in shelters on Sunday, while the evacuation orders and advisories were issued for almost 6 million people, according to the Fire and Disaster Management Agency.
In Hiroshima, Okayama and Ehime prefectures, the Japan Disaster Medical Assistance Team (DMAT), which comprises doctors, nurses and other staff members dispatched from outside those prefectures, began its activities, visiting evacuation centers to see if there were any evacuees complaining about health problems.
"It will be over 35 Celsius (95 degrees Fahrenheit) in some areas". The Japan Times reports rescuers on Tuesday were looking for people stranded in homes and in flooded areas that had not yet drained.
Japanese authorities issued evacuation orders to around five million people during the worst of the rains, but the orders are not mandatory, and many ignored them.
Hiroshima prefecture has been severely affected by rains.
More rain is expected over the next few days.
The Government has set aside 70 billion yen ($845 million) in infrastructure funds to respond to disasters, with 350 billion yen ($4.22 billion) in reserve, Mr Aso said, adding that an extra budget would be considered if needed.
Japan issues weather warnings early, but its dense population means that nearly every bit of usable land, including some flood plains, is built on in the mostly mountainous country, leaving it prone to disasters.