How can I set a custom "maximum texture size"?

When you first launch TexturePacker, you'll see a set of basic settings on the right side, including predefined options for the maximum texture size.

To set a custom size, click on Advanced settings, then navigate to the Layout section. Here, you can manually configure the texture width and height, setting them to any value greater than 0.