Recent research on Graph Neural Networks (GNNs) has shown that these algorithms can exceed state-of-the-art performance on applications with graph-structured inputs. These models present a new challenge for current machine learning accelerators because they combine dense, compute intensive operations with sparse, memory intensive operations. Furthermore, certain applications need to scale to graphs with up to billions of edges and unbalanced connections that follow power-law distributions. We present these requirements and describe the implications for machine learning accelerators. We describe Graphcore’s Colossus MK2 GC200 IPU Intelligent Processing Unit (IPU) and multi-processor systems based on the M2000 platform. The IPU adopts a radical approach towards accessing local memory at a fixed cost, independent of patterns by distributing its large SRAM and MIMD compute cores. We explain how systems based on the M2000 server leverage multi-phase execution paradigms to scale to thousands of IPUs and give them direct access to TB of DRAM, providing an effective scale up solution.