Roughly from what I know, custom cap just like normal cap, which has 6 slot. 3 slots must be fill with C rank. The other 3 free slot you can decide what you want to put in, SR, AR or BR or B or C. Default price is 2000 g per draw. 1 additional R rank will increase price by 500g or 300g which I not too sure. There is also extended card slot where you can increase your cap slot, to max 8 I think.
Also ms from pure A and B capsule cant be select into custom capsule machine.