I know what the Experiment is and this is my theory.
Spoilers if you have not read "The Secret History of Twin Peaks" or are not caught up on season 3.
So, after reading most of the "Secret History of Twin Peaks" by co creator Mark Frost, I think I have figured out what the creature that vomited up BOB and killed the couple in the black cube room is.
It's the whore of Babylon.
So in the book, "The Secret History of Twin Peaks", the archivist notes that Jack Parsons, co founder of JPL, engaged in Thelemic rituals supposedly aimed at summoning the whore of Babylon (Historically, Jack Parsons was both a scientist and indeed into the occult). The book claims that he engaged in this ritual at White Sands in Nevada just before the Roswell incident. Douglas Milford notes in the book that he interviewed Parsons about this ritual and Parson claims that he saw that which mortals probably shouldn't see.
Episode 8 picks up on July 16th, 1945 in White Sands, New Mexico...
Within some occult beliefs, the splitting of the atom is considered blasphemous. They claim that it opens the door to corrosive and unhealthy elements that cling to the worst characteristics of man kind. Perhaps this is why Parsons was interested in rocketry and his research boosted so much of the atomic program of the US. Perhaps the US government, either purposefully, or accidentally, performed the ritual correctly this time by performing blasphemy themselves in splitting the atom, thus summoning the Whore of Babylon.
That's my theory on the show at least. Makes this scene make a heck of a lot more sense in my opinion.
The whore of Babylon is also referred to as "The Mother of Abominations". And we see this creature vomit up BOB and other nefarious creatures as well. I'm pretty positive I'm right.