Hello Justin,
Hard to tell if will start or not. All row store tables will need to be fully loaded in the DB during BW start and operation. If the amount of memory necessary to maintain these up in RAM is bigger than what you have then it won't start.
If it's close to the limit and it does not sum up with other mandatory parts of the DB/BW, you might still see lots of out of memory dumps/ unloads occurring in the system for different operations. That might appear 'ok' for a non-prod env but honestly, whenever you try to make a real test, there's a big chance things will fail miserably (or just be boring slow).
Historically, NW and BW devs are always pushing the tables to be column store (see this nice blog post from Lars), and you might think that this would allow you to have smaller boxes for non-prod systems. Theorically, yes. Never been there though. However, having nw / bw core tables being unloaded/reloaded does not sound good to me.
One exercise that can be done is to run the BW on HANA sizing ABAP report on production to get a rough idea of the required amount of memory for each store and possibility to have BW non-prod starting or not.
I've seen a few happy customers implementing and maintaining non-prod systems in a disaster recovery site. That requires another production clone appliance (for replication) and additional storages where you can have QA and Test. This way you'll be able to get much closer to the real production environment (thus, much better testing) and will have a nice disaster recovery site. Of course, if things go bad and you need your replication site to takeover production, your non-prod systems will have to go for a nap (at least until you fix things up 
).
I hope this adds a bit to the discussion.
[]'s,
Lucas de Oliveira