WASHINGTON ― Rep. Reid Ribble (R-Wis.) was one of the first members of the Republican Party to say he would never vote for Donald Trump, calling the candidate a toddler and denouncing his treatment of women more than a year ago.

Now that the Trump campaign is going down in flames amid multiple sexual harassment allegations, Ribble isn’t saying “I told you so,” and he takes no pleasure in the quandary his GOP colleagues find themselves in.

“There’s no joy in that for me at all,” he told “So That Happened,” the HuffPost Politics podcast.

House Speaker Paul Ryan (R-Wis.), for instance, is under constant criticism for condemning Trump’s outrageous statements but not withdrawing his official endorsement. Ribble said Ryan’s position is an earnest effort to appeal to a GOP conference that’s as diverse as Republican voters across the country, not a craven attempt to have it both ways.

“A Republican member of Congress that might be representing a very red district in Alabama and Texas, that message that they’re getting from voters is likely to be very different than a member of Congress representing a Republican district in New England, upstate New York or even Wisconsin,” he said.

Easy for Ribble to say ― he’s retiring after this term.

Ribble’s take may be a bit more generous than that of some of his GOP colleagues. House Republicans have told HuffPost that Ryan misplayed the Trump endorsement, contrasting the speaker’s style of calling out offenses to Senate Majority Leader Mitch McConnell’s approach of largely ignoring Trump.

“The difference between Paul Ryan and Mitch McConnell is that Paul Ryan’s running for president, and Mitch McConnell’s not,” one member said, referring to a potential 2020 race.

Some Republicans have grown frustrated with Ryan’s one-foot-in, one-foot-out endorsement. They contend that he’s making it more difficult in close districts by adding legitimacy to the question of whether Republicans should unendorse Trump. At least, that’s what Ryan heard on a roughly hour-long conference call on Monday. After he and other GOP leaders laid out their plan of leaving Trump to his own devices, member after member spoke about why that might not be such a great idea.

Rep. Dana Rohrabacher (R-Calif.) called the leadership team “cowards,” and said he’d rather lose his seat than see a Hillary Clinton presidency. Rep. Trent Franks (R-Ariz.) dismissed Trump’s remarks as “locker room talk.” And while noting that he rarely speaks out against GOP leadership, Rep. Kevin Cramer (R-N.D.) said he considered this idea to be a bad one.

Members said their impression was that GOP Conference Chairwoman Cathy McMorris Rodgers (R-Wash.), who pointed out that it was the conference’s most dialed-in call of all time, shut down the conversation early to prevent a pile-on. (Only Rep. Charlie Dent (R-Pa.) spoke in support of leadership’s plan.)

But Ribble thinks Ryan rightly sees that Republicans need to understand the varying needs of the conference.

“What Paul Ryan was expressing was, ‘Listen, you guys pay very close attention to the lead of your constituents, because you know that better than anybody else, and feel free to do what you feel is necessary to win your election,’” Ribble said.

Still, some conservatives seem poised to use Ryan’s distancing on Trump as a reason to oppose him during a speaker election in January. Rep. Jim Bridenstine (R-Okla.), for instance, tweeted Wednesday that, “if Paul Ryan isn’t for Trump, then I’m not for Paul Ryan.”

Either way, members and top aides seem to think there will be a fight for the direction of the GOP once the election is over.

“Come Nov. 9, there will be a fight for who we are, what we stand for,” one aide told HuffPost.

And while there doesn’t seem to be a real plan for how Republicans have that fight, it will probably begin with Ryan and Trump duking it out.

“We will lose voters over it,” one conservative member told HuffPost this week. “But does it have to happen? Maybe.”

Another member added: “This is the fight we have to have.”