They allege they were also made to take tests for HIV and hepatitis following their arrest in February 2015 on suspicion of homosexual activity.
Homosexual acts are illegal in Kenya, punishable by up to 14 years in prison.
The High Court in Mombasa has given government lawyers a week to respond.
In a statement, campaign group Human Rights Watch (HRW), which has called for a global ban on the practice, said:
"Under international law, forced anal examinations are a form of cruel, inhuman, and degrading treatment that may amount to torture."
Anal examinations prove nothing, and they accomplish nothing, other than humiliating and demeaning people who are considered moral 'outcasts,'" HRW researcher Neela Ghoshal said.
UN experts have also described anal examinations to determine sexuality as "medically worthless".
Last year, Kenya's Deputy President William Ruto said there was no room for gay people in the country.
Most religious groups in Kenya and other African countries are strongly opposed to homosexuality, saying it is un-African.
Several people have been prosecuted in the last few years under Kenya's penal code.