I doubt that the automatic system setup will be able to configure this for you.
Essentially, this is a dual feedback stage. Halls are used for commutation, and velocity loop. Linear encoder used for position loop.
This can be set up manually following procedures described in the Power Brick LV (ARM or prior) manual.
First, you want to set it up for commutation and velocity using the halls as if they were a quadrature encoder that yields 12 counts per revolution (because of x4). In this step, you would verify current loop, phasing, and open loop functionality.
Next, you would point Motor[x].pEnc to the linear encoder conversion table entry, perform an open loop test to make sure the motor moves in the correct direction of the encoder counting. If not, switch EncCtrl from 7 to 3 or vice versa and try again. Lastly, try to tune the position loop.