To create a custom checkout pane, you'll first need to tell Drupal Commerce about your pane using hook_commerce_checkout_pane_info() in your module file:

/**
* Implements hook_commerce_checkout_pane_info()
*/
function my_module_commerce_checkout_pane_info() {
  $panes['my_module'] = array(
    'title' => t('My Module Pane'),
    'page' => 'checkout',
    'weight' => 10,
    'file' => 'includes/my_module.checkout_pane.inc',
    'base' => 'my_module_pane',
  );
  return $panes;
}

Now that Drupal Commerce knows about your pane, you'll need to create the pane in the file specified includes/my_module.checkout_pane.inc.
Let's start with a settings form first.

/**
 * Implements base_settings_form()
 */
function my_module_pane_settings_form($checkout_pane) {
  $form['my_module_pane_field'] = array(
    '#type' => 'textfield',
    '#title' => t('Pane Field'),
    '#default_value' => variable_get('my_module_pane_field', ''),
  );
  return $form;
}

You do not need to add a submit button, since Drupal Commerce will add one automatically for you. Also, you do not need to add a submit callback, since Drupal Commerce will automatically store your form values using the Drupal variable system. W have the backend configuration handled, so it's time to configure the customer facing part of the pane using base_checkout_form():

/**
 * Implements base_checkout_form()
 */
function my_module_pane_checkout_form($form, $form_state, $checkout_pane, $order) {
  $checkout_form['my_module_pane_field_display'] = array(
    '#markup' => variable_get('my_module_pane_field', ''),
  );
  $checkout_form['my_module_pane_field2'] = array(
    '#type' => 'textfield',
    '#title' => t('Field 2'),
  );
  return $checkout_form;
}

The values are not automatically saved by the Drupal variable system, because it's most likely that you'll want to do something more than simply saving the submitted values. To save the values, we will use base_checkout_form_submit().

/**
 * Implements base_checkout_form_submit()
 */
function my_module_pane_checkout_form_submit($form, &$form_state, $checkout_pane, $order) {
  // Get the value for my_module_pane_field2 
  // $form_state['values']['my_module_pane_field2']
  // Do something with it.
}

Using the checkout pane framework, we can see how easy is to add form fields to the checkout process, then use the submitted values in our custom submit handler.