In applications ranging from topology optimization to pattern recognition, one needs to differentiate through the factorization of a matrix that depends on many parameters. When the matrix is data-sparse, differentiating the factors can be much more expensive than computing the factorization itself. This talk explores data-sparse solutions to the adjoint problem associated with backwards differentiation of matrix functionals and introduces a new class of fast algorithms to differentiate through data-sparse matrix factors.