I can't tell you exactly because of so many variables, so I will just offer my experiences.
I ran my truck on 36 14.50 16.5 super swamper tsl radials forever. It most likely will sound stupid, but the truck on those specific tires, no others, was extremely nimble and always got an excellent hook off the line. By the way my truck has a 6 inch superlift with the 4 leaf pack before the easy ride or whatever they have now. So I don't have any body roll at all. Its a regular cab longbed so not really any weight. But at 35 psi those tires would run the slalom WAY faster than my stock 250 on stock street tires. Not that I did that but The red truck was VERY stable and consistent. And it did make a noticable difference with stock programming when I went to the swampers. But not really anything that bad. Just noticable. Since I am much stronger than I was stock now, I actually had the tires off not to long ago for something, I can't remember, and put the stockers back on there for the fun of it, and although it did seem faster I guess, It was not anything like I thought it would be. So I don't worry about it. I went to 35 14.5 16.5 swamper ssr radials and they were no faster, but the sidewall roll was substantially higher. I am now running 35 12.5 16.5 BFG mud terrains and they are not even in the league. I couldn't believe it, but they handle like crap no matter what pressure I try, they induce horrable power-hop, and they're not that much quieter. But they last. And that's what I need right now. I'll probably go back to the swampers next time. The tsl radials. I also didn't notice a worthwhile difference when going to the BFG's being a lighter tire.
The answer would be yes it will slow you down. But not nearly as much as you might think. Besides, it's that much more demeaning when you toast the brand new sportscars in the full-size diesel and you're riding on big rubber.
You're just not gonna get the same times at the track. But it woln't be as bad as you think.