Electrochemical noise is invaluable in the investigations of stochastic electrochemical processes. Though historically it has been applied almost exclusively to crevice and pitting corrosion studies, application to batteries is interesting for basic science and shows a huge potential for non-perturbing real-time sensor development. Given this promise, noise studies on batteries are starting to appear in the literature. In this manuscript, we are going to critically evaluate the small but emerging body of literature and investigate the mathematical methods employed. We will discuss the intricacies of electrochemical noise applied to batteries both in measurements and in analysis in order to initiate a deeper discussion. In conclusion, we argue that reliance on mathematical methods alone is not the answer, as all mathematical methods use require parameters to guide the analysis. Instead, post-mortem characterizations of batteries can guide the understanding of noise signals measured in batteries, guiding potential mathematical analysis methods and their steps.