The magnetic flotation hybrid separation process, which combines column flotation and magnetic separation, was anticipated to beneficiate a representative coal sample acquired from the El-Maghara coal mine with a relatively high ash percentage of 27.21%. The system was designed to introduce a high gradient magnetic field to a flotation column to hinder the flotation of magnetic particles even if attached to flotation bubbles, improving the coal demineralization process and producing high quality coal. The Box-Behnken design and response surface methodology were employed to investigate and optimize the combined influence of various operating parameters on the process performance. The effects of the main operating parameters such as magnetic field strength, slurry circulation flow rate, air flow rate, collector dosage, and frother concentration were investigated. Ash percentage, combustible recovery, and separation efficiency were defined as process responses. Upgrading the coal sample under the optimum conditions of 1.2 T magnetic field strength, 1500 ml/min circulation flow rate, 11.64 ml/min air flow rate, 0.87 kg/ton diesel collector, and 42 ppm methyl isobutyl carbinol frother, the predicted and two confirmation experiments data were 7.78%, 7.89%, 7.71% ash percentages with maximum recoveries of 75.86%, 76.37%, 75.94%, and maximum separation efficiencies of 77.17%, 78.51%, 78.71% respectively.