Simultaneous Localization and Mapping (SLAM) and Structure from Motion (SFM) are important and closely related problems in robotics and vision. Not surprisingly, there is a large literature describing solutions to each problem, and more and more connections are established between the two fields. At the same time, robotics and vision researchers alike are becoming increasingly familiar with the power of graphical models as a language in which to represent inference problems. In this talk I will show how SLAM and SFM can be posed in terms of factor graphs, and how inference in them can be explained in a purely graphical manner via the concept of variable elimination. I will also show that, when applied to Gaussian problems, the algorithm yields the familiar QR and Cholesky factorization algorithms, and that this connection with linear algebra leads to strategies for very fast inference in arbitrary graphs. I will then present the Bayes tree as a novel data structure for representing the inferred posteriors. The Bayes tree is similar to a junction tree, yet better embodies the connection with sparse linear algebra. I will conclude by showing some published and preliminary work that exploits this connection to the fullest.