It will work, I’ve done it. You might try using nut oil for around half the total oil to see how strong the flavor is, then adjust accordingly. I made it once with unrefined peanut oil, which had a fairly strong flavor, and half turned out to be too much. I do think the peanut oil was more strongly flavored than, for instance, La Tourangelle almond, walnut or hazelnut oil.
Please post back on your results, would love to hear about it!