As expected, there are some security risks involved.
Major one is connected with fact that erasure of data, due to TmpUsb having no battery, is possible only on next plug-in. If attacker is fast enough, he can plug TmpUsb in another computer without triggering erasure. Mitigation strategy for this attack is using "ArmMax" command in order to trigger erasure as soon as power is lost.
Similarly, attacker can charge capacitor without actually turning TmpUsb on. On next plug-in TmpUsb will think that less time passed than it actually has. Mitigation strategy is also using "ArmMax" command.
Even using "ArmMax" you are not safe if attacker is technically savvy and has access to TmpUsb that is currently working. It can put another power source in parallel to USB and thus TmpUsb won't be aware at any time that the power was lost. Mitigation is encasing microchip and other components into epoxy. Dirty job ahead.
Internal key storage is encrypted using the security feature of microcontroller itself. In ideal world this would mean that, even if attacker has direct access to chip, nobody would be able to read data without running it first. Of course, this is intended to stop normal attackers only. Somebody with unlimited resources (e.g. NSA) will not be deterred by this.
Regardless of these attacks, TmpUsb will probably protect you from 99.9% attackers, especially if they are not aware of its presence. It is not be–all and end–all device but it surely beats keeping keys on a normal USB drive.