Advancements in computer technology introduce many new hardware infrastructures with high-performance processing powers. In recent years, the graphics processing unit (GPU) has been one of the popular choices that have been being used in computational engineering fields because of its massively parallel processing capacity and its easy coding structure compatible with new programming systems. The full-wave solution of large-scale electromagnetic (EM) scattering problems with traditional methods has very dense computational operations, and thus additional hardware accelerations become an indispensable demand, especially for practical and industrial applications. In this context, the GPU implementation of full-wave electromagnetic solvers such as multi-level fast multiple algorithm (MLFMA) has shown a trend in the literature for the last decade. However, the GPUs also have many restrictions and bottlenecks when implementing large-scale EM scattering problems with full-wave solvers. Limited random-access-memory (RAM) capacities and data transmission delays are the major bottlenecks. In this study, we propose a matrix partitioning scheme to overcome the RAM restriction of GPUs to be able to solve electrically large size problems in single-GPU systems with MLFMA while acquiring reasonable accelerations by considering different implementation approaches. For this purpose, the Single-Instruction-Multiple-Data (SIMD) structure of GPU is considered for each stage of the MLFMA to check its compatibility. In addition, different operators of MLFMA are fine-tuned on the GPU scale to minimize the overall effect of data transfer and device latency. The preliminary analyses show that significant time efficiencies can be obtained for the different parts of MLFMA as well as eliminating the RAM restriction. The numerical results demonstrate the overall efficiencies of our proposed solution for the bottlenecks of GPU and also validate the expected accelerations for the solution of large-scale EM problems involving electrically large canonical geometries and real-life targets such as an aircraft and a missile geometry.