There is a way to use a compost to heat a terrarium. You create anaerobic conditions, which leads to a rotting process instead of decomposition and you could collect the methane and burn it to heat up your tank.
Or, you could draw a water pipe through the center of the compost (with a few coils in there to add surface area and exposure time) to transfer the heat from the compost to the water and then lead it back indoors to the terrarium.
I don't think it's worth the trouble.
Other stuff about composts:
The heat is a problem, a compost can reach temperatures of about 70 degrees Celsius at best, but most organisms that break down organic matter are dead by about 45. Likewise, their activity slows down as it gets cooler. The heat energy escapes into the environment, that's why the surface is colder than the core of a compost. And I'm afraid it does need volume to work. Otherwise it will be too cold to decompose efficiently.
One thing that would be interesting, though, would be a worm compost, but it would be more of a feeding than a heating solution. That one doesn't need volume, you can have one in a 10 liter bucket if you want to. Unfortunately, the worms usually used indoors, are
Eisenia. The toxic kind, if you remember.