I've checked and the Russell attribution may be fake news. These are his actual words from a 1927 lecture:
If everything must have a cause, then God must have a cause. If there can be anything without a cause, it may just as well be the world as God, so that there cannot be any validity in that argument. It is exactly of the same nature as the Hindu's view, that the world rested upon an elephant and the elephant rested upon a tortoise; and when they said, 'How about the tortoise?' the Indian said, 'Suppose we change the subject.'
If we can believe wikipedia:
The earliest known version of the story in its "turtle" form appeared in 1854, in a transcript of remarks by preacher Joseph Frederick Berg addressed to Joseph Barker:
My opponent's reasoning reminds me of the heathen, who, being asked on what the world stood, replied, "On a tortoise." But on what does the tortoise stand? "On another tortoise." With Mr. Barker, too, there are tortoises all the way down. (Vehement and vociferous applause.)
— "Second Evening: Remarks of Rev. Dr. Berg"
Whereas the "rocks" version (which I hadn't heard until you mentioned it) can be dated back even further:
In the form of "rocks all the way down", the story predates James to at least 1838, when it was printed in an unsigned anecdote in the New York Mirror about a schoolboy and an old woman living in the woods:
"The world, marm," said I, anxious to display my acquired knowledge, "is not exactly round, but resembles in shape a flattened orange; and it turns on its axis once in twenty-four hours."
"Well, I don't know anything about its axes," replied she, "but I know it don't turn round, for if it did we'd be all tumbled off; and as to its being round, any one can see it's a square piece of ground, standing on a rock!"
"Standing on a rock! but upon what does that stand?"
"Why, on another, to be sure!"
"But what supports the last?"
"Lud! child, how stupid you are! There's rocks all the way down!"