OK,so finally getting around to this, I'm slightly embarassed given that I was the one who suggested it.
The prerequisite trivia:
This is the only episode of TOS with no female speaking roles!
As @KandorLives noted, there are only two scenes aboard the Enterprise in the entire episode - one where Scotty talks to Kirk, and the other at the very end of the episode.
William Shatner's father died during the filming of the episode, and even though he got the news quite early in the day he insisted on working through until they wrapped that evening. While the producers and some of the cast knew what had happened, most of the people on-set had no idea and only discovered it when he left to get on a plane at the end of the day.
Shatner took comfort in Leonard Nimoy (one of the scenes that day was Spock's mind-meld with the Horta, and the melodrama made Shatner laugh) and cinematographer Jerry Finnerman, whose father had died on a movie set not long before. In order to not miss any additional filming time, when Shatner returned they moved the discussion of the silicon spheres and killings from the Enterprise briefing room as scripted to the office on the planet, in order to not have to spend time prepping another set.
Shatner claimed this was one of his favourite episodes to film in his Star Trek Memories book (partially because of his interactions with Nimoy the day his father died), but would also give The City on the Edge of Forever that accolade in a DVD special feature.
The closing scene was one of Leonard Nimoy's favourites, saying it defined Spock's relationships, self-image and characteristics.
In 1995, Arthur C. Clarke said this was the only TOS episode he remembered clearly, saying "It impressed me because it presented the idea, unusual in science fiction then and now, that something weird, and even dangerous, need not be malevolent. That is a lesson that many of today's politicians have yet to learn."
the Horta costume predates the story idea. Janos Prohaska, one of the costume designers made it and wore it into Gene Coon's office. Coon asked him what it was, and got the reply "I don't know, it can be whatever you want". Coon then decided to write an episode using the odd design as a focal point. Prohaska portrayed the Horta in the episode, and had designed and performed in a similar costume for an episode of the Outer Limits prior to Trek
And so, the episode.
This is one of Star Trek's quintessential moments. It's not the first instance of the "I'm a doctor, not a..." running gag, but it's the first time that exact phrase is used, and it's the iconic moment that everyone remembers from McCoy. It's also one of the best examples of Roddenberry's Infinite Diversity in Infinite Combinations ideology.
Like I said talking about Balance of Terror above, this episode looks at an issue that probably didn't have a specific name in the 60s, but does now - othering. The people of Janus VI assume that the Horta is an enemy from the beginning, and never waver from this until Kirk and Spock manage to calm them down. There's no attempts to communicate or even to understand what's happening. There's something bad out there and by God, they're gonna get it.
Of couse, the message of the story - that even the truly alien can be reasoned with it lessened a bit in the portrayal of the miners as semi-antagonists. Because Kirk, Spock and McCoy come down from above with their enlightened lifestyle in contrast to the "simple" miners and show them the error of their ways, there's quite a feel of colonialism to the tale. It doesn't detract too much overall - superior colonialism is basically Trek's stock in trade and the core ideals of the story are too powerful for this to be any more than a blip.