Having built my 461 with 1.65 rockers (and having a problem with them last summer on the Power Tour), I'll share my opinion --- that's all it is, opinion. YMMV and all that. 1.65's (factory on the Ram Air IV's) do two things: they add a tiny bit more lift and duration without having to change the cam, and they also put more stress on all the other valve train components. When I rebuild mine (hopefully soon) I'm going to go back to 1.5 rockers and instead have a cam ground that gives me the lift and duration I want with them. Result: same lift/duration specs but with less stress on all the parts. However, don't think that you can just replace rockers with 1.65's as a low cost way to get a little more out of your present cam. That's because the different geometry (the pushrod seat is closer to the rocker stud) also moves the pushrod lower in the head where it will almost always rub --- so to do it right you have to pull the heads and grind out the pushrod passages anyway to fix the rubbing problem. At that point, you've already done the majority of the work you'd have to do to swap cams. For me, why add the stress to the valve train when I can get the exact same results by using 1.5's and a different cam profile?
You said you're already not happy with the idle quality and fuel mileage. Going with a "bigger" cam (or swapping to 1.65's) is only going to make that worse, not better. As has been said before many times, Pontiacs are not chevys. You can't take the Chinese menu approach (1 from column a, 2 from column b, throw a handful of darts at a Jeg's catalog, etc.) in building one and get good results. You've got to consider everything about the car from bumper to bumper, including what you want it to do (race only, street, highway cruise) and how you're planning to drive it (highway, city, track), including that always a pain question of that pesky budget, and come up with a plan for the whole car. This includes rear gear ratio, transmission, torque converter (if an automatic), braking ("big" cams tend to kill vacuum and thus power brakes), fuel availability (compression ratio vs. octane) and more --- then come up with a plan that fits both your needs and your budget.
It can be done, but you have to think about it. Also the realities of the marketplace will mean that it will be more expensive than building "just another bowtie". Personally though, I find it much more satisfying