Based on my knowledge (which I'm still a novice, so take that for what it's worth), GPL v3.0 is fairly protective license. Although it has limits when it comes to liability, it has clauses that protect you from misappropriation and anyone who uses your rendering engine must include the very same license, and also acknowledge you as the author as per its Copyright Notice requirement. This license also has a Source Code Availability requirement, which requires that the source code for your engine be made available openly.
My concern is, are GPL-family licenses good enough to let contributors involve with this project? Because I heard that GPL licenses are like viruses and someone hates them.
Or maybe dual licenses are good enough if one of them is Apache License (for personal use only)? This is what VGG License 2.0 does.
I can see your concern with GPL-family being like a virus. Indeed, it is very restrictive and contributors may not want to get involved due to the logistical overhead of trying to add features to/improve it.
Addressing your concern about not being paid for your work, this is an age-old problem with open source. It's hard to make money off a product that is freely available to download. A good rule of thumb is to make users feel like they _want_ to give you money, instead of making them feel like they _have_ to give you money.
Some options you have if you want to keep it open source but still make money:
1. Do the dual licensing approach, and use one open source license of your choice, and one commercial license to allow businesses to use your engine in their proprietary software.
2. Donations link in the open source software
3. Paid technical support/Freemium model
4. Paid training programs to train users to become an expert with your engine
5. Early access subscription model: create a system where users can pay a monthly subscription fee in order to get new features earlier than official release.
