While men and women who ate lots of nuts, peanut butter, fish, yogurt and low-fat cheese tended to lose weight, other foods commonly seen as "unhealthy" - such as eggs, full-fat cheese and whole milk - did not seem to make a difference in weight.
On the other hand, sugary drinks and refined or starchy carbohydrates - including white bread, potatoes and white rice - had the opposite effect.
"The idea that the human body is just a bucket for calories is too simplistic. It's not just a matter of thinking about calories, or fat.
What's the quality of the foods we are eating? And how do we define quality?" said senior researcher Dr. Dariush Mozaffarian, of Tufts University and the Harvard School of Public Health, in Boston USA.
In general, the researchers reported, adults gained more weight as the "glycemic load" in their diets rose.
Glycemic load measures both the amount of carbohydrates in the diet, and the quality of those carbohydrates, said Mozaffarian.
A white-flour bagel, for instance, has a glycemic load (GL) of about 25 units, he noted; in contrast, a serving of quinoa - a whole grain - has a GL of around 13 units, and a serving of chickpeas has a GL of only 3.
Read: The truth about white and brown bread
In this study, every 50-unit increase in a person's daily glycemic load - the equivalent of two bagels - was tied to an extra pound gained over four years.
What's more, certain foods - like eggs and cheese- were connected to weight gain only if people also boosted their intake of refined or starchy carbs.
Red and processed meats, meanwhile, were also tied to weight gain. Again, though, some of the harm was reduced if a person's glycemic load was kept in check.
So, Mozaffarian said, eating that burger with a salad, rather than fries, could be a smarter move. Better yet, he added, eat it without the bun.
The findings, reported online in the American Journal of Clinical Nutrition, are based on 24 years of diet information from nearly 121,000 U.S. health professionals. At the outset, all were healthy and normal-weight, on average.
Over time, the study found, people's weight crept up - as it tends to with age - but the odds differed depending on the typical quality of their protein and carbs. That was the case even when the researchers accounted for other lifestyle factors, including overall calorie intake.
Read: Obesity gene may explain why some gain weight as they age
To Mozaffarian, that means counting calories is not enough to maintain a healthy weight in the long run.
Dietary fat was once demonized, Mozaffarian said, and that only led to people eating more refined carbs. "A lot of people still think you need to avoid fat to lose weight," he said.
Read: Victoria Secret models use the low-carb, high-fat diet to lose weight
Now, Mozaffarian worries that "count calories" is the new "low fat."
Putting calorie counts on menus, he said, could send consumers the wrong message: If that deli sandwich has a relatively low calorie count, people may assume it's a good choice - even if it's mainly processed meat and refined carbs.
The quality of protein, carbs and fat is vital
"This study really brings that to light," said Lauri Wright, an assistant professor of community and family health at the University of South Florida, in Tampa.
"But I don't want people to think calories don't matter," Wright stressed.
There are also no "magic bullet" foods that will melt off the pounds, she said. Nor can people avoid weight gain, and stay healthy, simply by avoiding a few "bad" foods.
Instead, Wright advised, choose healthy carbs, including vegetables, fruits and fiber-rich grains; proteins like fish, chicken and nuts; and "good" fats such as those in vegetable oils and fatty fish.
"You could have chicken breast on whole-grain bread, plus a salad, for lunch," she said. "For a snack, have almonds, or hummus and vegetables. Then for dinner, have salmon and vegetables."
But, she added, "calorie balance" - including the calories burned through exercise - is still important.
Go the Mediterranean route
Zoe Harcombe says counting calories will make you fat
Why we get fat, according to Gary Taubes
SOURCES: Dariush Mozaffarian, M.D., M.P.H., dean, Friedman School of Nutrition Science and Policy, Tufts University, Boston; Lauri Wright, Ph.D., R.D.N., assistant professor, community and family health, University of South Florida, Tampa; April 8, 2015, American Journal of Clinical, Nutrition, online