Apparently the part was originally offered to Cary Grant, who would only do one movie, and wouldn’t commit to a series. Patrick McGoohan turned it down as well before it was offered to Connery.
Ian Fleming favoured David Niven in the role.
Adam West was sought out to play Bond around the time Diamonds are Forever was made. He turned it down.
And if Pierce Brosnan didn’t come back for Die Another Day, Dougray Scott would have been donning the tuxedo.