Under Canadian law, both of you as parents have equal right to have access to a child, unless: 1) you and the other parent have expressly negotiated to curtail your individual custody or access rights; and 2) a Court has expressly ordered otherwise.
You are not entitled to single-handedly decide that your boyfriend’s access rights to the baby should be terminated. However, a court can certainly make an order to this effect, if circumstances warrant.
Access orders are usually made as an adjunct to a hearing for custody, which would determine how decision-making responsibility for your baby should be split between you and your boyfriend. This determination hinges on many factors, the key one being what is in your baby’s interests.
Judging the circumstance you describe — without more — it is unlikely that a court would grant you sole custody, and block your boyfriend’s access entirely. Courts generally make their custody/access rulings on the basis that it benefits every child to have exposure to both of his or her parents.