When conversing with large language models, it is common for users to ask an ambiguous open-domain question that could lead to multiple answers, especially when exploring new topics. For example, "Who won the US Open?" can result in different athletes according to the referenced events and years. We propose AmbigChat, an automatic approach that hierarchically disambiguates a factual question and guides users to navigate answers via UI widgets in a multi-turn conversational interface. Using the ambiguity taxonomy we generated from an analysis of 5,000 queries, AmbigChat identifies ambiguous facets of a question and constructs a disambiguation tree, where each level corresponds to a facet. Users can traverse the tree to explore answers via interactive disambiguation widgets and expand the conversation by referencing tree nodes through drag and drop. We iterated our interaction design with six design professionals and tested the effectiveness of the disambiguation tree generation algorithm on a variety of factual queries. Our evaluation with 16 participants shows that AmbigChat not only helps the participants find answers more easily and efficiently, but also facilitates structured explorations of the topic space.