This is an interesting op-ed type piece on prediction: http://www.scientificamerican.com/article.cfm?id=financial-flimflam.
It is a bit predictable and doesn’t have too many new things to say but the following para caught my attention:
There was one significant factor in greater prediction success, however, and that was cognitive style: “foxes” who know a little about many things do better than “hedgehogs” who know a lot about one area of expertise. Low scorers, Tetlock wrote, were “thinkers who ‘know one big thing,’ aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who ‘do not get it,’ and express considerable confidence that they are already pretty proficient forecasters.” High scorers in the study were “thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible ‘ad hocery’ that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess.”
This bit I completely agree with. What we need is a computational framework for doing exactly what the author says above! Then maybe the problem wouldn’t be as hopeless as it is now, even though (I agree) we will never have the perfect crystal ball..