Three of them are undergoing treatment at Dhaka's Sheikh Hasina National Institute of Burn and Plastic Surgery, reports bdnews24.
The blast triggered a fire in the fifth-floor home.
Authorities are yet to understand the exact cause of the blast.
The victims are Rizvi Ahmed Russell, 42, his wife Rozina Begum, 35, their child Rayan Ahmed, 3, and Russell's mother Shahida Khatun, 65, all natives of Sirajganj, the Bangladeshi news portal reported.