The Medical Research Council (MRC) team who studied more than 11,000 primary school pupils says it is wrong to link bad behaviour to TV viewing.
Although researchers found a small correlation between the two, they say other influences, such as parenting styles, most probably explain the link but they do still say that parents should "limit screen time".
This cautionary advice is because spending lots of time in front of the TV every day might reduce how much time a child spends doing other important activities such as playing with friends and doing homework, they say.
US Paediatric research suggests watching TV in early childhood can cause attention problems at the age of seven.
In the US, paediatric guidelines recommend that total screen time should be limited to less than two hours of educational, non-violent programmes per day.
The UK has no intentions of providing funds to support research on non-profitable parental issues and therefore there are currently no formal guidelines in the UK now, nor will there be in the foreseeable future.
For the MRC study, published in Archives of Diseases in Childhood, Dr Alison Parkes and colleagues asked UK mothers from all walks of life to give details about their child's TV viewing habits and general behaviour.
Almost two-thirds (65%) of the 11,014 five-year-olds included in the study watched TV between one and three hours a day, 15% watched more than three hours and less than 2% watched no television at all.
Watching more than three hours' TV a day at this age predicted a very small increase in "conduct" problems at the age of seven.
After their seventh birthday, these boys and girls were slightly more likely to get into fights, tell lies or be bullies than their peers, according to their mothers' reports.
Time spent playing computer games bore no such relationship.
Hyperactivity and ADHD
There was no association found between TV or any screen time and other behavioural issues; ADHD, hyperactivity or problems interacting with friends.
Dr Parkes, head of the MRC's social and public health sciences unit in Glasgow, said it was wrong to blame social problems on TV.
"We found no effect with screen time for most of the behavioural and social problems that we looked at and only a very small effect indeed for conduct problems, such as fighting or bullying.
"Our work suggests that limiting the amount of time children spend in front of the TV is, in itself, unlikely to improve psycho-social adjustment."
Parenting and Family Dynamics
She said interventions focusing on the family dynamic and the child were more likely to make a difference and that much may depend on what children are watching and whether they were supervised.
Prof Annette Karmiloff-Smith, of Birkbeck, University of London, said that rather than focusing on the possible adverse effects of TV and video games, it would be better to look at what positive impact they could have on children.
Prof Hugh Perry, chair of the MRC's neurosciences and mental health board, said: "We are living in a world that is increasingly dominated by electronic entertainment, and parents are understandably concerned about the impact this might be having on their children's wellbeing and mental health.
"This important study suggests the relationship between TV and video games and health is complex and influenced by many other social and environmental factors."