Integrating with the FreeBSD CAM Subsystem

CAM offers a variety of driver types. There are peripheral drivers which generate commands (da(4) for disks, sa(4) for tape, etc.), the CAM mid-layer (xpt) which routes commands, and hardware drivers (also called SIMs). SIMs interface with xpt through sys/cam/cam_sim.h. Routines for peripheral drivers are in sys/cam/cam_periph.h.

For more information on writing a CAM initiator driver, please see the following resources:

Adding target support to a SIM is relatively simple. For this example, we'll use the isp driver (sys/dev/isp) and the targ peripheral driver (sys/cam/scsi/scsi_target*) as an example. Miscellaneous CCB definitions and other data can be found in /sys/cam. First, the driver needs to indicate that it is capable of target mode. This is done by setting the PIT_PROCESSOR flag:

2588-#ifdef ISP_TARGET_MODE
2589:       cpi->target_sprt = PIT_PROCESSOR | PIT_DISCONNECT | PIT_TERM_IO;
2590-#else
2591-       cpi->target_sprt = 0;
2592-#endif

The SIM also needs to have a handler when it receives a ccb_en_lun CCB in its action routine (i.e. isp_action()). This CCB tells the SIM which bus/target/lun it wants to enable for target mode. See sys/cam/scsi/scsi_target.c to see how the enable lun CCB is set up and what fields need to be parsed by the SIM (ahc_handle_en_lun() for aic7xxx). Once it enables target mode in its device-specific way, the SIM also needs to have a queue for incoming ATIO and INOT CCBs. The user provides these resources and the SIM just grabs one off the queue and fills it out when receiving a command or when an asynchronous event occurs (like a bus reset).

The next thing a SIM driver needs to do is figure out how its particular transport provides access to incoming commands. For instance, isp has a device-specific format called ATIO2 which the firmware uses for passing information about incoming commands. Here is its structure:

typedef struct {
    isphdr_t        at_header;
    u_int32_t       at_reserved;
    u_int8_t        at_lun;         /* lun or reserved */
    u_int8_t        at_iid;         /* initiator */
    u_int16_t       at_rxid;        /* response ID */
    u_int16_t       at_flags;
    u_int16_t       at_status;      /* firmware status */
    u_int8_t        at_reserved1;
    u_int8_t        at_taskcodes;
    u_int8_t        at_taskflags;
    u_int8_t        at_execodes;
    u_int8_t        at_cdb[ATIO2_CDBLEN];   /* received CDB */
    u_int32_t       at_datalen;             /* allocated data len */
    u_int16_t       at_scclun;              /* SCC Lun or reserved */
    u_int16_t       at_wwpn[4];             /* WWPN of initiator */
    u_int16_t       at_reserved2[6];
    u_int16_t       at_oxid;
} at2_entry_t;
Upon receiving a command in its interrupt routine (or possibly while a thread processes input previously received from an interrupt handler), the SIM needs to package the incoming CDB into the CAM ccb_accept_tio structure. Important fields to fill in include the cdb, cdb_len, and init_id. If the transport always uses tags (i.e. FC), then fill in the tag_id and tag_action as well. Common fields in the ccb_hdr that should be filled in are ccb_h.status, ccb_h.target_id, and ccb_h.target_lun.
struct ccb_accept_tio {
    struct     ccb_hdr ccb_h;
    cdb_t      cdb_io;          /* Union for CDB bytes/pointer */
    u_int8_t   cdb_len;         /* Number of bytes for the CDB */
    u_int8_t   tag_action;      /* What to do for tag queueing */
    u_int8_t   sense_len;       /* Number of bytes of Sense Data */
    u_int      tag_id;          /* tag id from initator (target mode) */
    u_int      init_id;         /* initiator id of who selected */
    struct     scsi_sense_data sense_data;
};
When the ccb_accept_tio is properly filled in, call xpt_done() on it to pass it to the xpt layer which delivers it to the appropriate peripheral instance (based on the path in the ccb_hdr). This processing is done by isp_handle_platform_atio2() in isp for FC. (SCSI processing is done by isp_handle_platform_atio() and won't be discussed here).

The peripheral then receives the completed ATIO and then fetches the data to satisfy it. It then prepares and sends back multiple CTIOs, which share a common structure with an initiator request called ccb_scsiio. XPT routes the CTIO to the proper SIM and sends it with xpt_action(). For isp, this translates into a call to isp_action().

When isp_action() is called, it should process the CTIO just as if it was a command sent in initiator mode. It should pull out the appropriate items including: data_ptr (which may be a pointer to a buffer or a list of SG entries - see sys/dev/aic7xxx/aic7xxx_osm.c), dxfer_len, and scsi_status. The latter is only valid if the header has the CAM_SEND_STATUS flag set.

struct ccb_scsiio {
    struct     ccb_hdr ccb_h;
    union      ccb *next_ccb;       /* Ptr for next CCB for action */
    u_int8_t   *req_map;            /* Ptr to mapping info */
    u_int8_t   *data_ptr;           /* Ptr to the data buf/SG list */
    u_int32_t  dxfer_len;           /* Data transfer length */
                                    /* Autosense storage */ 
    struct     scsi_sense_data sense_data;
    u_int8_t   sense_len;           /* Number of bytes to autosense */
    u_int8_t   cdb_len;             /* Number of bytes for the CDB */
    u_int16_t  sglist_cnt;          /* Number of SG list entries */
    u_int8_t   scsi_status;         /* Returned SCSI status */
    u_int8_t   sense_resid;         /* Autosense resid length: 2's comp */
    u_int32_t  resid;               /* Transfer residual length: 2's comp */
    cdb_t      cdb_io;              /* Union for CDB bytes/pointer */
    u_int8_t   *msg_ptr;            /* Pointer to the message buffer */
    u_int16_t  msg_len;             /* Number of bytes for the Message */
    u_int8_t   tag_action;          /* What to do for tag queueing */
    /*
     * The tag action should be either the define below (to send a
     * non-tagged transaction) or one of the defined scsi tag messages
     * from scsi_message.h.
     */
#define         CAM_TAG_ACTION_NONE     0x00
    u_int      tag_id;              /* tag id from initator (target mode) */
    u_int      init_id;             /* initiator id of who selected */
};
The typical sequence for a WRITE is as follows:
  1. peripheral driver sends ccb_en_lun
  2. sim driver receives it in its action routine and enables target mode
  3. periph sends multiple ATIOs and INOTs for the SIM to queue
  4. sim driver receives them in its action routine and places them on a queue
  5. After some time, a WRITE command comes in.
  6. sim driver receives CDB in its private command handler.
  7. sim driver grabs an ATIO off its queue, copies cdb into it, and sets other parameters (init_id, any autosense data, etc.)
  8. sim sends ATIO to the periph driver via xpt_done()
  9. periph parses command and realizes it is a WRITE. It sets up a buffer and prepares for an incoming transfer.
  10. periph sends one or more CTIOs to the sim, each with an appropriate pointer into the buffer as well as the amount of data to transfer for each CTIO.
  11. sim gets CTIOs in its action routine, extracts pointer and data len. sim places CTIOs on a local queue.
  12. sim initiates DIR_OUT data transfer in device-specific manner
  13. as each transfer completes, sim sets any error status in the CTIO's ccb_hdr, removes it from the queue, and calls xpt_done() on the CTIO.
  14. periph receives the completed CTIO and notes any errors. It transfers the contents of the buffer to the backing store.
  15. once periph has received success from each CTIO, it prepares and sends one final CTIO that is empty but has the CAM_SEND_STATUS flag set as well as the appropriate value in scsi_status.
  16. sim receives final CTIO and sends the status to the initiator as well as tearing down any state associated with this transfer.
  17. periph sends back ATIO for sim to use for future commands.
The READ path is simpler and the status can piggyback on the final CTIO.