How can I exclude specific files from a smart folder when packing?

TexturePacker's UI doesn't support excluding files directly, but you can use the --ignore-files option on the command line. Be sure to escape wildcards if using a shell:

TexturePacker sheet.tps --ignore-files \*.psd

To store the ignored files in a .tps project file, either use --save to update the project file from the command line, or edit it manually.