Knights are commonly viewed as warriors in armor whose only duties were to find adventures, court beautiful damsels, and charge at each other with padded lances surrounded by brightly-clad people feasting on mutton and wine. While that picture isn't complete hogwash, it's far from an accurate image of the European warrior class dominant during the Early and High Middle Ages.
Most knights did indeed wear armor (either plate armor or chainmail), and most of them didn't have a whole lot going on a lot of the time, but that's about the end of the similarity between fiction and reality. Courtly love didn't make it on the scene until much later (about the 14th century), knights seldom had the wherewithal to simply rove the countryside in search of sport, and jousts and melees were vicious, often blood-soaked affairs that had little to do with the pageantry with which later romantic imagination endowed them.
The European knight was quite simply a fighting man. His interests and pursuits were confined to those which would make him a better fighting man, and those included wielding weapons, riding a horse, wrestling, sometimes sailing, and other similar activities. When he went to a joust or a melee it wasn't to show off in front of the ladies, it was to fight, often to the death.
When he wasn't training or actively fighting (either at a competition or at war in the service of his lord), the knight was feasting, hunting, reveling, or just lounging about; the best ones also spent time administering their estates. The chivalric ideal was more about manliness, strength and prowess than anything else, and most knights would have been humiliated were they to be found deficient in any of those things, so they pursued them often and heartily.
Probably the most misunderstood aspect of knighthood, however, is the idea of feudalism. In the feudal society, a knight basically contracted with a lord or king for land and rank in exchange for military service, some form of tax, and food. The food and tax came from the land given to the knight and its produce: grain, vegetables, and meat raised by the serfs, along with any saleable items they may have produced.
This land was called an estate, and in order to protect its inhabitants from marauders, it was necessary for some form of fortification to be erected, often in the center or at one end of the estate. For a long time, all castles were wooden, and featured a "mott and bailey" construction in which the central tower was built on top of a man-made hill, at the base of which a moat was dug and which in turn was surrounded by a stockade or wall.
It was only later, as vassals (those swearing fealty, or allegiance, to the nobles from whom they received their land) became more established, that stone castles were built. The defensive value of a stone castle as opposed to a wood one is obvious, but they took a lot longer to build, consumed far more resources, required the hiring of tradesmen or the diversion of serfs from other tasks, and were much harder to repair. Eventually, however, stone castles were more a sign of wealth and ostentation than an actual defensive measure.
Romantic portrayals of knights were popular near the end of the feudal period, largely in an attempt to put a better face on what was starting to be seen as a barbaric practice (knighthood itself, that is, not feudalism per se). Unfortunately, this is largely the image we think of when we imagine knights, castles, chivalry, and the like: unfortunate both because it's historically inaccurate, and because it puts a good face on a system that really was pretty harmful for everyone involved.
The practice of holding serfs is obviously an abrogation of human rights and freedom, but just as bad was the constant bloodthirstiness that characterized the typical knight. His whole existence revolved around combat and death, and he gloried in those things, far more than he ever gloried in sighs and surreptitious glances at his lady-love.
People will always be rewriting history, mostly to try to prove that past times were better than our own, or that there is some innate goodness in humans, or whatever reason they may have for altering the facts. In the case of knighthood, the tendency to wrap its nasty elements in a cloth of pageantry and glamour contributes to an unhealthy view of warfare, of human freedom, and of an age predicated on the absence of peace, even while the name of the Prince of Peace was on everybody's lips.