Once , Ravana , Kumbharkarna , Vibhishana were performing penance to please Lord Brahma . Brahma appeared and asked them for their boons. Ravana and Kumbharkarna asked for powers and siddhis etc.( You might be knowing Ravana's boon ). But , Vibhishana asked for , that unlike his brothers he does not want any Shakti but something else. He wanted only to serve Prabhu ( Hari ) and do Bhakti. Brahma granted them their boons and disappeared .
So , you can understand now what I am trying to say. When Ravana and Kumbharkarna used their boons and enjoyed it completely then why did they question Vibhishana ? Won't he get his boon fulfilled to serve Rama ( Hari ). This was my first point .
And as for Meghnath , he realised the harsh truth in the last moment of the war. When the three most destructive weapons did not do Laxmana any harm , he then realized who Laxmana really was . But , it was too late and as a warrior , he should not switch sides after the way he had started. And Vibhishana went to Rama's side before the war started . ( You might know that before Kurukshetra Krishna asked that those wanted to switch sides could do it , it means it was Dharma ) .
And the last point is that Vibhishana was on the path of Dharma . Many say that Vibhishana became a bhakta of Lord Rama after he joined Him , but NO ! When Hanuman was searching for Sita in Lanka he heard the chanting of Rama . It was Vibhishana . He directed Hanumana to Ashoka Vatika .
So , we should not accuse Vibhishana that he left his brother in tough times. He was just doing Dharma as I have mentioned in the above points .