Sounds like a carb problem, not a distributor problem. First, make sure your choke is wide open and that the choke linkage is not set on the high idle side. You should see a screw adjustment for this as well as the fast idle screw - 2 different screws & adjustments if you hav this on your carb. Make sure the cam/arm is not hanging up or not dropping down like it should to kick down the choke and lower your idle speed. Here is a You Tube video:
Sometimes the throttle cable can be too tight and keep the primaries on the carb from fully closing. Quick check would be to undo the cable and fire it back up. If it idles down, then adjust the cable to give it a little more slack.
Wrong carb gasket on the base allowing either the carb to not fully close or allow a vacuum leak. There are several gaskets that fit and you have to be sure you have the correct one. There could also be a vacuum leak somewhere in the intake system, ie the carb, a vacuum line, even a bad intake gasket - all can let more air in and raise idle speeds. I have never done this, but it is said you can use a small hand held propane torch to run it along these various areas to find any vacuum leaks. If the engine speeds up while running and doing this test, you found the vacuum leak.
Bigger cam will sometimes affect the metering rods within the carb. There is a spring found under the power valve - which is controlled by engine vacuum. Less vacuum will allow the spring/rods to be in the raised position allowing a richer mixture, where they would normally be pulled down into their lowered position for a factory cam/strong vacuum signal. They make weaker springs that can be swapped out to match the lower vacuum signal and drop the metering rods into position to run leaner at idle. The top of the carb has to be pulled off to access the spring.
Is the carb flooding? Too high a float setting could be pushing some extra fuel into the carb causing it to run richer/faster.
Retarding your timing should drop idle speed, but retarding it too much will make your engine run hot. Set your timing to factory specs and then do some testing with the carb as it does sound more like a carb/vacuum issue rather than timing at this point.