In common usage, this is an imaginary substance that psychics and mediums excrete that forms itself into semi-physical things. The idea is that a medium who is talking to someone in the afterlife will somehow manifest ectoplasm, and this circulates in the air and forms the apparition of the dead person in the room.
“Ecto” is Greek for “outside,” and “plasm” means “that which has form.” Thus, ectoplasm is a substance that forms things in the “outside.”
This is fairly common device in fiction. Most people of my generation will remember it from Ghostbusters. Whenever ghosts disappeared, they left behind a green slime which was referred to as “ectoplasm.” (The modified ambulance they drove around in was named “Ecto-1.”)
Essentially, ectoplasm was invented as a term to bridge the gap between the psychic and physical worlds. I guess a physic or medium had to explain how things from “the other side” entered the physical world. Their explanation was that these things were formed from ectoplasm, which was created by the psychic.
The word was actually coined in the late 1800s by a paranormal researcher. Not surprisingly, no one has ever proven the existence of ectoplasm. When investigated, apparitions that appear during seances are invariably revealed to be stage props of some kind.
Curiously, ectoplasm does have a specific scientific meaning – it’s the outer part of a cell’s cytoplasm. However, this usage is overshadowed by the paranormal reference.