It said people's lives had been put on hold for years because of flawed decision-making and poor communication.
The Parliamentary and Health Service Ombudsman's report found "major failings" in eight drivers' cases.
The DVLA said the vast majority of cases it handles are dealt with swiftly and correctly.
Criticisms in the report are directed at the Drivers' Medical Group - the part of the DVLA which considers whether drivers with a medical condition are safe to drive.
The DMG makes between 600,000 and 750,000 licensing decisions every year and around 10% of those are complex cases dealt with by medical experts.
The report looked at eight complaints in detail which were received by the ombudsman between 2014 and 2015.
These all concerned people with complex medical conditions who were unfairly prevented from driving, sometimes for several years, the ombudsman's report said.
The report found "major failings", including evidence of flawed decisions, significant delays, poor communication and complaint handling in those cases.
And it said it was concerned that other people had also been treated unfairly and that the same mistakes could be repeated.
David, from the north-east of England, told the DVLA he had been diagnosed with bipolar disorder when he reapplied for a licence to drive lorries.
But he was refused, even though his psychiatrist and GP said he was safe to drive.
After providing more information on his condition and the medication he was taking, he also explained that his new job depended on him being reissued with a licence.
But the delay resulted in his employer having to let him go from his £20,000 a year job.
He went to his union for help and they encouraged him to get a solicitor and take it to court, which he did.
He also complained to the ombudsman, who found in his favour.
But the experience, which lasted more than two years, left him feeling angry and frustrated.
"Everyone is an individual but I didn't see anyone face to face. It was just a tick box exercise," he says.
The DVLA sent a letter apologising for their actions and he received £6,000 in compensation.
"But I shouldn't have been put in that position in the first place," David says.
The report recommends that the DVLA improves the way it communicates with people applying for a licence and with medical professionals, and sets up robust standards to assess people with medical conditions fairly.
In addition, it says financial compensation should be offered to those affected by failures, where appropriate.
Parliamentary and Health Service Ombudsman Julie Mellor said the DVLA's failings had had a real impact.
"People's lives have been put on hold for years because of severe delays and flawed decisions by the DVLA, leading people to lose their jobs, causing stress, worry and isolation."
She said the DVLA had produced a new guide for medical professionals and improved its complaint handling and communications - but there was still more to do.
"Further action is needed to make the assessments of fitness to drive more robust, to prevent others from suffering the same injustice in the future."
Oliver Morley, chief executive of the DVLA, said it was sorry for the way some customers' cases were handled.
"These eight very complex cases, however, date back to 2009 and since then the vast majority of the four million cases we've handled have been dealt with swiftly and correctly.
"We have already made a number of improvements including more effective ways of managing cases, taking on more staff and introducing a new online service where drivers can tell us about their medical conditions."
Nick Lyes, RAC public affairs spokesman, said it was vital for the DVLA to reduce unnecessary delays - but it also had a balancing act to perform.
"It is vitally important that those who need their vehicles and are safe to use them can do so in the quickest time possible, but there is also a duty on authorities to keep our roads and drivers as safe as possible."
The charity Scope said disabled people should be supported to live as independently as possible - and that included getting them safely behind the wheel.
Conditions that could affect your ability to drive safely include:
- Other neurological and mental health conditions
- Physical disabilities
- Visual impairments