Ok, from what i've found out now......
It
is possible to lose points. Sorry, i felt sure i'd read that you couldn't but i was wrong

You can lose points a number of ways.....
If you demolish a building
If you lose any troops (or dismiss any)
If your citizens are unhappy, and therefore leaving your colony
and strangely, if you build a new colony

(not sure about this, but apparently it's true) The points from this are recovered pretty quickly though
There may be a few other ways, but those are the ones that are mentioned most.
Is everyone happy in your colonies? I would think this is the most likely one.