The custom item selection menu is the menu from which you can add custom items as ingredients (or final product) to crafting recipes, container recipes, and a couple of other things. It should look like this:

The menu consists out of the item buttons in the middle and a cancel button on the left.

The 'Cancel' button will take you back to the previous menu.

The item buttons direct you back to the previous menu, and select that custom item as the ingredient or result. In this example, pressing the 'ruby' button will select a 'ruby'.