I did play on core difficulty (“intended PF experience” or something), but as far as I can tell nothing affects the enemy HP, just how much damage you take (maybe there’s something in the custom modifiers). I guess even with a weak character you can get through on a lower difficulty, but it’ll take ages.
However, even if I used proper builds, I still used my very limited knowledge of these games and systems. If you know what you’re doing, you can probably use better spells or abilities to kill things faster.
I didn’t realize this was set in the Pillars of Eternity universe. I might have to check it out, after I’ve played Pillars 2.