A pesticide called heptachlor epoxide was found at high levels in milk in the early 1980s in Hawaii, according to the study authors.
Only an association
The pesticide was used in the pineapple industry, and can also be found in well water. Use of the pesticide was banned in the United States around the same time, the researchers noted.
The current study can't prove that the pesticide or milk consumption directly causes Parkinson's disease; it can only show an association, according to study author R. D. Abbott, of Shiga University of Medical Science in Otsu, Japan.
Read: Berries and tea protect men from Parkinson's disease
The study authors also pointed out that they don't know for sure if the milk consumed by these men had heptachlor epoxide in it or not.
One Parkinson's expert urged restraint.
"This study is not a wake-up call to stop drinking milk – only 12 people who drank about two glasses of milk a day showed significant loss of neurons," said James Beck, vice president of scientific affairs at the Parkinson's Disease Foundation.
"Nevertheless, its results also suggest that low levels of a pesticide, accumulated in milk, contributed to the loss of brain cells. So a practical question to ask is if it is time to consider strategic purchases of organic foods – it may be."
Read: Daily exercise may lower Parkinson's risk
The study included almost 450 Japanese-American men living in Hawaii. Their average age was 54. All of the men took part in an ageing study and were followed for more than 30 years. After their deaths, autopsies were conducted.
Compared to non-smokers who drank less than two cups of milk a day, non-smokers who drank more than two cups of milk a day had 40 percent fewer cells in a certain area of the brain, the study revealed.
Smokers have lower risk
The loss of cells in that particular area of the brain occurs in Parkinson's disease. Changes in that area of the brain can begin decades before people have any symptoms of the disease, the study authors said.
There was no link between milk consumption and brain cell loss in smokers, the study noted. Previous research has found that smokers have a lower risk of Parkinson's disease, the researchers said.
Read: Brisk walking may curb Parkinson's symptoms
The study also found residue of heptachlor epoxide in 90 percent of study participants who drank the most milk, compared to 63 percent of those who drank no milk.
The findings were published online in the journal Neurology.
"There are several possible explanations for the association, including chance," Dr Honglei Chen, from the U.S. National Institute of Environmental Health Sciences, wrote in an accompanying journal editorial.
"Also, milk consumption was measured only once at the start of the study, and we have to assume that this measurement represented participants' dietary habits over time," he said.
Constipation: an early sign of Parkinson's?
Gene therapy vs. Parkinson's
'Junk' clue boosts hopes for Parkinson's treatment