it is best to test the latency from the VPS using the following command : ping server_ip_address
You can also doa trace route for the broker server and see where it is located. then try to get a vps in the same network with (using the same network provider if possible)
that would be really fast
i hope it helps