In 1947, he challenged Sugar Ray Robinson for the World Welterweight Title and planned to buy his mother a house with the winnings.
Doyle had suffered heavy knockouts in California prior to the fight and the boxing commission refused to allow him to participate in any more.
However, Robinson agreed to fight him and the results were disastrous. During the fight, Robinson had the advantage every round except the sixth. After staggering twice and then being hurt, Robinson responded with a single left hook that ended things. Doyle was struck in the head and his body went rigid.
Robinson described the event saying, “That punch knocked Jimmy rigid.... With heels resting against the canvas as if hinged, Doyle's body went down. It struck the floor with a thud, like a rigid mass falling. His head crashed against the padded canvas, and as the referee started the count. Doyle raised his head and rested on his elbows.... The referee counted to ten. Doyle was out."
He was quickly rushed to St. Vincent’s Charity Hospital but failed to regain consciousness and was pronounced dead several hours later. Criminal charges were threatened against Robinson for fighting him, but none ever materialized. Robinson felt so terrible for what he had done that he gave the earnings of his next four fights to Doyle’s mother, fulfilling Doyle’s last wish of buying her a house.