HARMONI is the first light visible and near-IR integral field spectrograph for the ELT assisted with two adaptive optics mode (SCAO and LTAO) that will be on sky in 2029. The project is preparing for its final design review and in particular the Assembly, Integration and Test period. HARMONI adaptive optics will be tested in Marseille, whilst the IFS will be tested separately in Edinburgh, but these two systems will only be integrated together at the ELT, when the ELT M4 mirror will also be seen by HARMONI for the first time. This changes a lot our approach to test the adaptive optics systems. We propose showing how we will answer this challenging situation for HARMONI. The impact is different for the SCAO and the LTAO modes. While the SCAO mode will be mostly validated at sub-system level with a small telescope simulator, the LTAO mode requires the use of a large telescope simulator with various object as sources (both laser and natural guide stars). The development of this telescope simulator requires a lot of effort and its goal and main capabilities need to be carefully defined. We ask the question of what are the tests that are essential to be done in the laboratory, even without a perfect representation of the M4 mirror, and what are the tests we can can be postponed until the instrument is integrated at the telescope. We will also show the current design of the telescope simulator, which includes the simulation of 6 laser guide stars, 2 natural guide stars, an optical relay between the sources modules and HARMONI entrance and the used of Spatial Light Modulator (SLM) to simulate both the M4 mirror and the turbulence and to compensate for the optical relay aberrations.