Third seed Halep was the highest-ranked player in action on Wednesday but came unstuck against Mertens despite taking the opener and leading by a break in each of the subsequent sets.
Mertens emerged a 4-6 7-5 7-5 winner after two hours and 34 minutes, defeating Halep for the first time on clay and only the second time on any surface.
There were opportunities for both players throughout and Halep struck first, losing her break lead in the first but recovering on this occasion.
The Romanian, a two-time winner who has also twice lost in the final in Madrid, twice cancelled out a deficit in the second but could not then capitalise on a lead of her own.
Mertens battled back and broke to love to level the match, then maintained her composure to see out a back-and-forth decider once Halep had again forged ahead.
“She played really well, and she definitely deserved to win because she was stronger in the end,” Halep said.
“I cannot say I played bad, but I did some mistakes, important mistakes. Sometimes it’s just a little bit and the match is going away.”
Sabalenka is up next for Mertens, although she had a much more straightforward path through the third round.
Jessica Pegula held her up for only 52 minutes in a dominant 6-1 6-2 success.
In the same half of the draw, Anastasia Pavlyuchenkova upset another seed, following up her straight-sets victory over Karolina Pliskova by beating Jennifer Brady in three. She plays Karolina Muchova next.
At L’Open 35 de Saint-Melo, second seed Rebecca Peterson came from behind to see off Xiyu Wang, while Tamara Zidansek and Arantxa Rus also produced comebacks.
Nina Stojanovic survived three sets, but Christina McHale crashed out to Viktoriya Tomova.
Get breaking news alerts.
You have already subscribed