From my experience it is not recommended to put any course under pop up window since there are a lot of users who do not understand how pop-up works, and even when browser reports blocked pop-ups they don’t know what to do so they end up with empty browser window. I would recommend to stick with embedded courses and to forget about exit button but if you are insisting on using it here is a simple example how to create such button.

If you need exit/close button for Captivate project that opens in pop-up window you can easily create one using AS3 code in Flash. The code is quite simple:

<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>[cc lang='actionscript3']
pbcExit_mc.addEventListener(MouseEvent.CLICK, myBtnClicked);
function myBtnClicked(e:MouseEvent):void {
navigateToURL(new URLRequest("javascript:window.opener=self; window.close()"), "_parent");
}
[/cc]<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_end"></span>

Now all you need to do is to style your flash button or you can simply use this sample:

Preview
Adobe Captivate used to have similar button included in skin player until version 4. The button was excluded since it works only for pop up windows and there were some issues depending on browser used. This as3 flash button works for all browsers as far as I know, but if your browser is blocking the script you will see error message when you click EXIT button that I have included in this sample:

Here you can download button only or you can download Captivate example with all source files.