Here is some evidence, that we not only have to pray 5 times a day, but that we have to do it on time. The following is from Al Kabaair -The Major Sins by Sh. Adh-dhahabi

The Fourth Major Sin - Not Praying

Allah most High says: Then there succeeded them a generation which neglected prayers and followed lusts. They will meet with destruction (Ghayy), excepting the one who repents and believes and acts righteously. (Quran 19:59-60)

Ibn abbas (RAA) explained that it does not mean that they completely abandoned prayers, but that they delayed them to the end of their prescribed times. Sa'id bin Musayyab, a great Imam following the generation of the Companions, elaborated on this, saying: "A person neglects if he delays the noon prayer up to the afternoon prayer time, afternoon prayer up to almost sunset prayer time, sunset prayer up to evening prayer time, and evening prayer up to morning prayer time, while delaying morning prayer until just before sunrise.  If anyone dies without repentance while continuing in this manner, Allah has promised Ghayy for him, and that is a valley in Hell which is very deep and has foul food."

Allah Most High says: Woe to the worshippers who are unmindful of their prayer. (Quran 107:4-5)

Saad bin Abi Waqqas said: "I asked Allah's Messenger (SAW) about the meaning of 'unmindful,' and he replied: 'It is delaying it (the prayer) up to the end of its prescribed time.'" (Bazar)

While Allah calls such people , they are lazy and perform the prayer very late. He has promised them  (Wayl), which signifies a great punishment.

Some say, "Wayl is a valley in Hell, exceedingly hot, and that is the place for those who delay or miss the prayer, excepting the ones who repent and correct their habits."

Allah Most High says: O you who believe, let neither your wealth nor your children distract you from the remembrance of Allah. And those who do so, they will be the losers (Quran 63:9)

The commentators say: "The 'remembrance of Allah' in this verse means the five daily prayers. If anyone is so busy in buying and selling, or with his daily work of earning a livelihood, or with his children, that he cannot perform prayers on time, he will be among the losers."

The Prophet (SAW) said in this regard: "The first thing which will be judged among a person's deeds on the Day of Resurrection is the prayer. If that is in good order, he will pass the test and prosper, and if that is defective, he will fail the test and be a loser." (Tirmidhi)

Informing us about the inhabitants of Hell, Allah Most High says: (The people in Hell will be asked): "What caused you to enter Hell-fire?" They will say: "We were not among those who prayed, nor did we feed the needy. We used to engage in vain discourse with the vain-talkers, and we used to deny the Day of Judgment, until there came to us that which is certain (i.e.. Allah's judgment)." Then no intercession by an intercessors will benefit them. (Quran 74:42-48)

Allah's Messenger (SAW) is reported to have said: "If anyone abandons the afternoon prayer, his deeds are of no avail." (Bukhari).

And he said: "If anyone abandons prayer deliberately he has no claim upon Allah" (Sunan).

Abdullah bin amr bin al-As (RAA) said that the Prophet (SAW) mentioned prayer one day saying, "If anyone keeps to it, it will be light, evidence, and salvation for him on the day of Resurrection; but if anyone does not keep to it, it will not be for him light, evidence, or salvation; and on the Day of Resurrection, he will be associated with Qaran, Pharoah, Haman, and Ubayy bin Khalaf (a great enemy of Islam from among the Quraish)." (Ahmad, Darimi and Baihaqi)

Mu'adh bin Jabal (RAA) reported Allah's Messenger (SAW) as saying: "If anyone abandons prayer intentionally he has no claim to Allah Most High." (Ahmad)

Umar ibn al-Khattab (RAA) reported that a man asked the Prophet (SAW):  "Messenger of Allah, what action is dearest to Allah Most High?" The Prophet (SAW) replied: "Prayer at its proper time. The one who does not pray has no religion. Prayer is the main pillar of the religion (of Islam)." (Baihaqi)

After Umar had been stabbed, it was said to him, "As-salat, ya amir ul-mumineen!" (The prayer, Leader of the Believers!) He said, "Yes, truly!  If anyone misses the prayer, he has no share in Islam," and thereupon he prayed while his wounds were bleeding.

Abdullah bin Shaqiq (RAA) said that the Companions of the Prophet (SAW) did not consider the abandonment of any good deeds to be unbelief excepting the abandonment of prayer.

When Ali (RAA) was asked about a woman who did not pray, he replied:  "The one who does not pray is an unbeliever." Ibn Masood said that the one who does not pray has no religion, and Ibn Abbas said: "The one who leaves off a single prayer deliberately will find, when he meets Allah Most High, that He is angry with him."

Allah's Messenger (SAW) said: "When a person meets Allah after neglecting his prayer, he will meet him as if he had not worshipped him at all, nor done any good deeds." And Ibn Hazm said: "There is no greater sin after polytheism than missing a prayer [That is to say, if one deliberately delays a prayer until its time has passed, although he was able to pray it on time, without any valid reason which permits delaying it before its time.] and killing a believer without just cause."

Ibrahim Nakahi said: "The one who has left off the prayer has become a disbeliever."

 Awn bin Abdullah said: "When a person is buried, the first question which will be asked him will be concerning the prayer. If that is accepted from him, his other deeds will be looked at, but if that is not accepted form him, no other deeds will be looked at."

Abdullah bin al-As reported Allah's Messenger (SAW) as saying: "A person who combines two prayers without excuse, enters a gate among the major sins."

We seek Allah's guidance and help, for indeed. He is the Generous, the Magnanimous, and the Most Merciful of all merciful ones!