Searching a rust server in the server list can be quite long but it is possible to connect to a server directly if you know its address (which is the case when it comes to your server).
For this once the game started press the F1 key to open the console.
Then type the following command:
client.connect ip: port
Replace IP with the IP address of your server and port the port on your server.
For example, if the IP of your server is 87.98.111.222 and port 28015 on your server then simply type:
client.connect 87.98.111.222:28015
Then send the command by pressing Enter.