To Tommy: because the phenomenology of "ghosts" is not my area of research, I have not earned the right to expound anything like a responsible theory. I will, WITH A LOT OF HUMILITY, say a few things that I believe I've gained from my family's experiences and reading the best literature.
Maybe it would not be a waste of time to make a short list of the authors of that literature.
1). Fr. Herbert Thurston, a Catholic priest with an open-minded intellectual approach to the subject [a Jesuit I believe]. His book is Ghosts and Poltergeists;
2). Catherine Crowe, a 19th century lady with connections all over England and Germany, who collected many credible incidents and published them in The Nightside of Nature.
3). ARG Owen, author of one of the more sensible reviews, Can We Explain the Poltergeist?;
4). GNM Tyrell, who authored probably the best overview specifically on apparitions, entitled simply Apparitions;
5). "Moderns" like William Roll, Scott Rogo, and Loyd Auerbach are OK, but the modern thrust towards "explaining" poltergeist events by subconscious breaking-out of psychokinesis usually caused by some unhappy highly-stressed puberty-era teenager is in my estimation wrong.
To even begin to attempt a theory of "ghosts" one must decide what you're even trying to explain. There are places where there are only physical effects. There are places where there are only apparitions. There are places [much rarer in the researched literature] where there are both. All these things have been labelled "ghosts" by pop culture. In my opinion, all cases are not the same thing. Surprisingly to me, my brother's house DOES have both, but the physical events are much more common.
Whenever an experienced anomalies researcher goes into a subject, he assumes that he is not going to be confronted by one thing but by several different "explanations". Of course, baloney is rampant. You have to expect that about 90% [maybe really about 9-out-of-10, i.e. not just a catchphrase] claims are bogus. Most of these are not hoaxes, but the excited poor observation or knowledge base of folks wanting to be a part of something wonderful. The "leftover 10%" comprise the true mystery.
Physical anomalies [motions of objects or sounds, which are a type of motion of air] happen. There is WAY too much of that attested to for the general claim not to be true. But what/who is the causal agency? You don't have many categorical choices. You can go with a living human being, like in the William Roll distressed teenager concept, and buy into spectacular psychokinesis effects which don't manifest with that person any other time. I softly reject this as many "poltergeist houses" persist in their behaviors regardless of what people happen to be occupying them.
What's left? The "spirits of the dead", minor demons [who go by many names in cultures worldwide: the djinn of the middle east and India are a prominent example], and what the Old Irish used to cal "the Middle Angels". These, according to the old folklore were angelic-class spirits who, when given the famous test by God, refused either to choose failure like Lucifer, or communion like Michael, and essentially said: no way to either, we'll think about it. God then "solved" that dilemma by placing these spiritual/paranormally-based creatures in parallel realities to worlds like our own, but with restrictions on the limits of their interactions and interferences.
And there are the worldwide concepts of the "playful tricksters", such as Raven, Coyote, et al in Native American lore. There is a theory that djinn, minor demons, middle angels, tricksters et al are all the same sort of entities and are behind such incidents. As I say, I have not earned the right of an opinion on this. It is a powerful enough hypothesis to cover the whole range of the anomalies, but it is untestable and would remain a matter of faith.
I'm stopping with that. All of these mini-reviews I write are a bit uncomfortable as they cannot possibly deliver the richness of the subjects and are therefore rife with potential misinformation. As with all of them: read the serious people, multiple viewpoints. Don't jump too quickly to a conclusion. Gather your own pile of data; see if it all fits or falls apart into separate piles. Think.