> Although neural networks struggle on simple arithmetic tasks such as addition and multiplication, we show that transformers perform surprisingly well on difficult mathematical problems such as function integration and differential equations. > We define a general framework to adapt seq2seq models to various mathematical problems, and present different techniques to generate arbitrarily large datasets of functions with their integrals, and differential equations with their solutions. > On samples of randomly generated functions, we show that transformers achieve state-of-the-art performance and outperform computer algebra systems such as Mathematica. > We show that beam search can generate alternative solutions for a differential equation, all equivalent, but written in very different ways. The model was never trained to do this, but managed to figure out that different expressions correspond to the same mathematical object > We also observe that a transformer trained on functions that SymPy can integrate, is able at test time to integrate functions that SymPy is not able to integrate, i.e. the model was able to generalize beyond the set of functions integrable by SymPy. > A purely neural approach is not sufficient, since it still requires a symbolic framework to check generated hypotheses. Yet, our models perform best on very long inputs, where computer algebra systems struggle. Symbolic computation may benefit from hybrid approaches.