Al-Nawawi records in his book Riyad al-Salihin:
Abu Huraira reported: The Messenger of Allah, peace and blessings be upon him, said:
إِذَا أَصْبَحَ أَحَدُكُمْ يَوْمًا صَائِمًا فَلَا يَرْفُثْ وَلَا يَجْهَلْ فَإِنْ امْرُؤٌ شَاتَمَهُ أَوْ قَاتَلَهُ فَلْيَقُلْ إِنِّي صَائِمٌ إِنِّي صَائِمٌ
When one of you wakes up in the morning for fasting, then he should not use obscene language or behave foolishly. If anyone abuses him or fights with him, he should say twice: Indeed, I am fasting.
Source: Ṣaḥīḥ Muslim 1151
Abu Huraira reported: The Messenger of Allah, peace and blessings be upon him, said:
مَنْ لَمْ يَدَعْ قَوْلَ الزُّورِ وَالْعَمَلَ بِهِ فَلَيْسَ لِلَّهِ حَاجَةٌ فِي أَنْ يَدَعَ طَعَامَهُ وَشَرَابَهُ
Whoever does not leave evil words and deeds while fasting, Allah does not need him to leave food and drink.
Source: Ṣaḥīḥ al-Bukhārī 1804
In another narration, the Prophet said:
رُبَّ صَائِمٍ لَيْسَ لَهُ مِنْ صِيَامِهِ إِلَّا الْجُوعُ
One might fast and he gets nothing from his fast but hunger.
Source: Sunan Ibn Mājah 1690