In the following, we assume a Site-to-Site or a Multi-Site connection. Please check the About VPN Gateways document for an explanation and more options.
Setup on Azure is described very well in the VPN site-to-site tutorial. In short, you have to create a virtual network gateway (including a local subnet and a public IP address), a local network gateway, and a VPN connection between them. Make sure to select the correct VPN type (route-based or policy-based) depending on your needs and/or your VPN gateway requirements. If your only intention is to connect one or more on-premise HSMs to Azure, you can rely on static routing and you do not need to enable BGP in the local network gateway. Instead, you should specify the IP range of your local internal network where the HSMs are located.