Researchers from University College London say women need to be aware of other warning signs - such as nipple changes - so that they get help fast.
The researchers examined the symptoms of 2,300 women who had recently been diagnosed with breast cancer.
They found that women with non-lump symptoms were more likely to put off visiting their doctor.
The researchers are presenting their work at the National Cancer Research Institute conference in Liverpool.
Swellings in the armpit or an infection of the breast tissue should be checked out, they say. So too should nipple changes - rashes, bleeding or discharge - and any sores on the skin of the breasts.
Dr Karen Kennedy, director of the National Cancer Research Institute, said: "This research shows that, all too often, women are delaying going to their doctor with symptoms of breast cancer.
"This could be because people are simply unaware that breast cancer can present in many different ways, not just through the presence of a lump.
"With a disease like breast cancer, it's essential to be diagnosed as early as possible so that a treatment plan can be developed and started."
Breast cancer symptoms and signs
See your GP if you notice:
- A new lump or area of thickened tissue in either breast that was not there before
- A change in the size or shape of one or both breasts
- Bloodstained discharge from either of your nipples
- A lump or swelling in either of your armpits
- Dimpling on the skin of your breasts
- A rash on or around your nipple
- A change in the appearance of your nipple, such as becoming sunken into your breast
Source: NHS ChoicesIn England and Wales, about nine in every 10 women diagnosed with breast cancer will survive for at least five or more years.
And about six out of every 10 can expect to survive for more than 20 years.
The more time that passes since diagnosis, the less likely it is that cancer will come back.
Women in the UK aged between 50 and 70 are invited for breast screening every three years.