You can, just only for formations that are already in the scheme of the coach you pick on the Coaching Shemes screen.
For example, if you set the scheme to a default playbook that has I Form Pro, and you also have I Form Pro in your custom, you can set formation subs for that formation, and they'll work in game. You just won't have the options to set subs for additional formations that aren't in that default playbook from that screen...you can still set form subs from the playcalling screen in-game, and they'll persist for the rest of that game.
The general advice if you're interested in using form subs with a custom playbook is to start with a default book that contains as many of the formations you plan to use as possible, rather than tearing a book down to the studs and rebuilding from scratch.
Been like this for years.