Many current research efforts are based on better exploitation of sparsity— common in most large scaled problems—for computational efEciency. This work proposes different methods for permuting sparse matrices to block angular form with specified number of equal sized blocks for efficient parallelism. The problem has applications in linear programming, where there is a lot of work on the solution of problems with existing block angular structure. However, these works depend on the existing block angular structure of the matrix, and hence suffer from unscalability. We propose two hypergraph models for decomposition, and these models reduce the problem to the well-known hypergraph partitioning problem. We also propose a graph model, which reduces the problem to the graph partitioning by node separator problem. We were able to decompose very large problems, the results are quite attractive both in terms solution quality and running times.