The new Polish government has vowed a "more humanitarian" border policy.
He said border guards took her to hospital in Hajnówka, close to the border, when they found her.
On Saturday activists said she had left hospital and was in an apartment with her baby.
Mr Duszczyk denied that the government of Donald Tusk was continuing pushbacks and said its top priority was "zero deaths at the border".
"When the Eritrean mother and baby were taken in by soldiers at the border and sent to hospital, she asked for international protection, and we've started the procedure under EU law," he said
"We do our best to help teenagers and women, and since January this year, we've accepted 350 migrants asking for international protection," he added.
In January more than 100 NGOs and 500 activists and cultural figures urged Mr Tusk to end pushbacks and said his government was condoning human rights violations if it did not do so.
However activist Iwo Los, also from Grupa Granica, said the practice was continuing.
His organisation said border guards had pushed back more than 1,700 people since December. Some 25 of these people were still missing and five were known to have died, it alleged.
Mr Duszczyk said the Polish government was facing an "organised, instrumentalised migration war" being carried out by Belarus.
Poland alleges that since 2021 Belarus has been encouraging people from the Middle East and Africa to travel to Belarus and then cross the border illegally to Poland.
In 2021 the European Union accused Belarus's authoritarian leader President Alexander Lukashenko of facilitating the influx in retaliation against sanctions."We're trying to combine security with humanitarianism while saving our border in line with EU law," Mr Duszczyk said.
