Tallon Griekspoor ground down Jan-Lennard Struff 6-7 (4/7), 7-5, 6-4 in the second singles rubber to eliminate the three-time winners in the semi-finals.
Botic van de Zandschulp beat Daniel Altmaier 6-4, 6-7 (12/14), 6-3 in a hard-fought opening match to set Netherlands on their way.
Jannik Sinner's Italy, the defending champions, face Australia in the second semi-final on Saturday.
The Dutch knocked out Rafael Nadal's Spain in the last eight on Tuesday on the way to their first final in 104 years of competing.
"What we're doing is more unique than, let's say, Italy, with the No 1 player (Sinner) in the team," said Haarhuis, who was in the team last time they made the semi-finals 23 years ago.
"The key is the team effort, the belief in ourselves.
"It was our goal to not lose in the quarter-finals for the third straight year, we believed we could do it."
Germany, who last triumphed in 1993, were hoping to reach a sixth final.
"Today the Netherlands took their one or two chances more than us, they deserved the win," said Germany captain Michael Kohlmann.
Van de Zandschulp kicked off the tie by claiming a scrappy victory with his 10th match point in two hours 44 minutes.
"It could have been easier but I did it the hard way," said the Dutchman who sent 22-time Grand Slam title winner Nadal into retirement on Tuesday with a famous win.
"At some point, I didn't know what to do any more on the match points."
Neither player blinked in a first set with few thrills, until the Dutchman, ranked 80th, nosed ahead for a 5-4 lead which he served out.
Van de Zandschulp broke in the fifth game of the second set and consolidated for a 4-2 lead, taking full control of the match.
The Dutchman forced four break points in the next game, but the world number 88 managed to escape with three aces to avoid a double break.
Moving into a higher gear the German was able to get back on serve at 4-4.
The Dutch player brought up five match points in the tie-break but could not hold his nerve and Altmaier took his fourth set point with a smash to force a third set.
Van de Zandschulp broke in the second game, but Altmaier immediately responded.
The Dutchman produced another break of serve to open up a 5-3 lead and, after wasting four more match points, finally wrapped up the win with his 10th as Altmaier could not return a powerful serve.
'Serving unreal'
Big servers Struff and Griekspoor could not force a break in the first set, or even a break point, and a tie-break quickly became inevitable.
The German imposed himself with two mini-breaks and took his third set point.
Griekspoor battled hard to hold in the eighth game of the second set, the first in the match to go to deuce, and the Dutchman saved two break points which proved a pivotal moment, changing the momentum in the clash in his favour.
The world number 40 broke to snatch a 6-5 lead against Struff, ranked three places lower, and polished off the set with an ace blasted down the middle.
Struff smashed into the net to gift a break in the first game of the third set and the Dutchman produced five holds to triumph, barely allowing his opponent a sniff on his serve.
"I had a little chance in the second set but he didn't give me many opportunities," lamented Struff.
Griekspoor ended the tie with an ace, his 25th of the night, and fell to his knees in celebration as orange-clad Dutch fans celebrated with glee.
"I felt like the level of the match was unbelievable, both of us were serving unreal," said Griekspoor.
"Big respect to him but very happy to make the final."
(For more sports news and updates, follow Ahram Online Sports on Twitter at @AO_Sports and on Facebook at AhramOnlineSports.)
Short link: