Victoria is a very beautiful city with some old English charm left over from the good old days. The inner harbour which adjoins the down town and the Legislature grounds is a destination in itself. Take the shuttle from the dock return for 6-7 $ and do that to start.
As Mr. UT suggests th Gardens are a highlight. The are lighted at night as great in all cruise seasons. Depending on your time in Port as you point out a ships tour might be recommended.
All tour tickets for the city can be purchased at the tourist info center right on the inner harbour kitty corner from the Empress Hotel which is itself a draw for tourists.
There is also a great Museum and the Legislature buildings are open depending on your visit times.
Enjoy your planning.
It's a 20-30 min stroll to downtown from the ship. All flat and mostly residential.