The proposal, introduced in April, would have allowed criminal suspects to be extradited to mainland China.
The bill was suspended in June when Ms Lam called it “dead”, but she stopped short of withdrawing it.
Full withdrawal is one of five key demands of protesters, who are also calling for full democratic rights.
In a televised address on Wednesday, Ms Lam also announced other measures that appeared to be designed to soothe unrest.
She said two senior officials would join an existing inquiry into police conduct during the protests. An independent investigation into alleged police brutality against protesters is another of the activists’ key demands.
On Monday, Ms Lam was heard on leaked audio tapes blaming herself for igniting Hong Kong’s political crisis, and saying it was unforgiveable of her to have caused such huge havoc.
The extradition bill quickly drew criticism after being unveiled in April. Opponents said it would undermine Hong Kong’s legal freedoms and might be used to intimidate or silence critics of Beijing.
Hong Kong is now in its 14th successive week of demonstrations, and saw fresh violence between police and activists last weekend.
What did Carrie Lam say?
In the recorded message Ms Lam said the protests had “shocked and saddened the Hong Kong people” and the violence was “pushing Hong Kong towards a highly dangerous situation”.
“No matter what discontentment the people have towards the government or the society, violence is not the way to resolve problems,” she said.
“Currently, stopping the violence is the top priority, maintaining the law and rebuilding the rules of society. The government will sternly tackle violence and illegal action.”
Ms Lam said she and other senior officials would visit communities in Hong Kong and talk to people directly about their concerns.
What reaction has there been?
Pro-Beijing lawmaker Regina Ip told BBC Chinese the latest move was “a positive step forward”.
“It won’t pacify everybody but hopefully it would clear remaining doubt in the minds of some of the peaceful protesters,” she said.
“The protests have been driven by a multitude of reasons including deep-rooted anger about Hong Kong’s widening wealth gap, housing conditions and political system. I am glad that the chief executive said that she would go to the district to enter direct dialogue with the people from all walks of life.”
Pro-democracy politician Wu Chi-Wai, however, dismissed Carrie Lam’s compromises as “fake”.
“We must stop police brutality. Otherwise the protests will keep going,” he said.
Ahead of Ms Lam’s announcement, leading pro-democracy activist Joshua Wong said the withdrawal of the bill would be “too little too late”.
In a series of tweets he said all the protesters’ demands had to be met.
Demonstrators are also demanding an amnesty for those arrested, greater political reforms and for officials to stop describing the protests as riots.
Hong Kong was handed back to China in 1997 after more than 150 years of British rule. It has remained semi-autonomous under a “one country, two systems” principle but some fear China is seeking greater control.
Will concessions be enough to stop protests?
Analysis by Stephen McDonell, BBC News, Hong Kong
Three months ago, hundreds of thousands of people took to the streets of Hong Kong. However, their concerns regarding a bill allowing for extradition to mainland Chinese courts controlled by the Communist Party were dismissed by Carrie Lam.
Fast-forward to now and we are seeing weekly, increasingly violent street clashes between radicalised activists and riot police. Petrol bombs, rubber bullets and tear gas are regular features on the streets of this Asian financial hub.
By her own reckoning, Hong Kong’s chief executive has created “unforgivable havoc” by bungling the response to broad public opposition to her proposal.
Finally, she is officially withdrawing the bill.
There had been widespread speculation that Carrie Lam did not have the authority to adhere to this or any of the protesters’ demands because Beijing has really been calling the shots.
It is possible that she has been given the green light to pull the bill to try to show that Hong Kong’s autonomous decision-making is still intact.
But the longer it has taken for the extradition bill debate to be resolved, the wider the demands from activists have become. Many demonstrators now say they will not stop holding rallies without a genuinely independent inquiry into the Hong Kong police force and universal suffrage in this region.
What is the bill about?
Opponents said allowing criminal suspects to be extradited to mainland China would undermine the city’s judicial independence and risked exposing Hong Kongers to unfair trials and violent treatment.
They also argued the bill would give China greater influence over Hong Kong and could be used to target activists and journalists.
Opposition was widespread from the start with groups from all sections of society voicing their criticism.
Hundreds of petitions started by university and secondary school alumni, overseas students and church groups also appeared online.
Lawyers, prosecutors, law students and academics joined the protests and called on the government to shelve the proposal.
Several countries also expressed concern. In response, China’s foreign ministry accused them of interference in China’s internal affairs.