The landslide, which followed heavy rains, struck Bukalasi, situated on the slopes of Mount Elgon, on Thursday afternoon.
"Most of the people were caught at the market, the landslide pushed huge boulders into a river which burst its banks and the water swept away the people," said Commissioner for Disaster Preparedness and Management Martin Owor.
Relief teams were combing the area to search and rescue survivors, Owor said.
"There are people who were displaced and they need shelter, food and all other support and we're moving that relief to the area," he said.
The rainy season in that part of Uganda, about 250 km (155 miles) from the capital Kampala and close to the Kenyan border, runs from September to December.
An avalanche in the same area in 2010 killed at least 80 people.
Many of the East African country's mountainous regions, also including areas in the southwest and west near the borders with Rwanda and the Democratic Republic of Congo, often experience landslides at this time of the year if rains are unusually heavy.