"FancyBoxThumb"-like extensions does not work on images, selected by RandomImageByCategory. Some dirty hack had to be done to make it work (doesn't tested on security issue, perfomance, image options etc. - it just works).
in RandomImageByCategory.php find:
$thumbnail = "<a href=\"{$image_title->escapeFullURL()}\">{$thumb_image->toHtml()}</a>";
and replace to:
$thumbnail = "<a class=\"image fancybox\" href=\"{$image_title->escapeFullURL()}\">" .preg_replace("/<img/","<img class=\"thumbimage\"/",$thumb_image->toHtml()) ."</a>";