You'll be giving your competitors an opportunity to steal your market. The biggest problem with pursuing other opportunities is that it distracts from your efforts to effectively and efficiently manage your core business. Losing focus of your core business provides the competition with a tremendous opportunity to step in.
A leading manufacturer of high-end western leather goods pursued an opportunity to manufacturer designer clothing. They were going to be the next "Guess jeans." While they pursued this opportunity, their key competitor stole half their market share. As a CEO lamented, "We took our eye off our core business for two years and the competition came in and ate our lunch." Don't give your competitors a free lunch.
Your rowboat has become a battleship. The bigger a company gets, the harder it is to change direction. A small company with few employees can change direction much more readily than a large company. Just think how easy it was to change your logo or stationary when you first started, and how easy it was to move your operations the first time. Your actions only impacted a few people, a few customers, and a few suppliers. As you grow these relationship expand geometrically, locking you into an expanding web of interdependencies. Change is never easy but it often becomes impossible in large companies.