In recent years, secure collaborative machine learning paradigms have emerged as a viable option for sensitive applications. By eliminating the need to centralize data, these paradigms protect data sovereignty and reduce risks associated with large-scale data collection. However, they also expose the learning process to active attackers, amplifying robustness issues. In this talk, I'll discuss the security and robustness challenges of secure collaborative learning systems, present our efforts to mitigate some of these issues and highlight why a definitive solution to robustness in these systems is challenging.