Solution:

This Telegram app just installs a .desktop file, its icon and a folder where it saves its data:

Remove the folder with the Telegram and Updater file.

Then run:

rm -Rf \
  ~/.local/share/icons/telegram.png \ 
  ~/.local/share/applications/telegramdesktop.desktop \
  ~/.local/share/TelegramDesktop

Note:

I installed Telegram by double clicking the Telegram file.

No, you didn’t install Telegram by double clicking, neither apt was involved in any way. The tar.xz package contains a pre-compiled executable for Telegram Desktop and it executes it every time you launch it. Simply delete the extracted folder to get rid of Telegram.