Advanced search returns results from only certain types of attachments, which appears to include searching by extension type. (ref: Search for pages and posts)
For example, search attachments with the .pdf extension:
Search for .mp4 attachments:
It seems that Confluence doesn't treat the extension as part of the filename. Searching for an attachment by file extension appears to only work with the file types listed above. Limiting to search title only has the same effect.
Allow Advanced Search to search additional attachment types.