There is no direct bus from the Victoria Airport (YYJ) to Downtown Victoria.
If you take the bus just outside the airport (bus 88) you'll have to transfer once or twice depending on the route you take. One journey (no transfer) will cost you $2.50 and a day pass $5.
So if you want to save some money and are willing to walk a little, you can go from the Victoria Airport to Downtown using only one bus (total cost of $2.50 per person). You can easily walk on the side of the road, on the cyclist lane (there is no real sidewalk, so just be careful of the traffic and the cyclists). The road is in good condition so you can roll your luggage if needed. Expect to walk around 25 minutes.
Once arrived at the McTavish Exchange Bay B ("bus stop" on the map), wait for the bus 70 or 72. The stop is on the right side of the road from where you arrive. Pay the fare directly to the driver. Note that no change is given back, you need the exact amount. The bus takes around 50 minutes to reach Downtown.
You can check the official bus fares on the BCTransit fares page.
Victoria Harbour