This is the thesis proposed by Amanda Mummert and colleagues: that Neolithic was not so good, or rather that it had at least very poor trade-offs. We tend to think having regular sources of food as something good but when that means eating mostly one or two vegetable foods all year long it may well be a problem.
Whatever the reason what Mummert discovered is that the health and size of Europeans suffered with the introduction of farming.
Source: Science Daily.
Ref. A. Mummert et al., Stature and robusticity during the agricultural transition: Evidence from the bioarchaeological record. Economics & Human Biology, 2011. Pay per view.
Incidentally a couple of days ago, I stumbled upon the Wikipedia page on life expectancy and I noticed that same pattern in the list of documented life expectancy variation over time: the record before the Middle Ages was in the Paleolithic period, when newborns could expect to live 33 years. Then, with Neolithic, it fell to a mere 20 years, recovering only very slowly.