Three people, all in their 20s, were injured in the collapse.
Ambulance Victoria said a 28-year-old man suffered serious upper-body injuries.
Victoria Police said the man died as he was being airlifted to a Melbourne hospital.
A man and a woman sustained non-life-threatening injuries and were taken to hospital.
Victoria Police believe a cliff face upwards of 30 metres high gave way and collapsed on the group.
Six people were sitting under the cliff face when rocks fell and struck three of them.