Back to Community
How can I store the states of my algorithms when it crashes and restarted

After I start running an algorithm, sometimes it's disconnected from brokers so that I need to restart my algorithm. But in this way, I lost the states of my algorithm, is it a way that I can store the states persistently? Like write to a file and read from file.

7 responses

nope. If you're looking to see how long you've held onto something you would need to calculate when your algo would have bought it.

I feel your pain though! So I ascertain from James' answer and other posts that I've read, that an algo can read a file but not write to it. That would explain why some people here are apparently using additional automation external to Quantopian algo to do the writing to the file, so that you don't have to update the file manually. I do automation for a living, so I can possibly do something for myself that satisfies my requirements. But to somehow make it easy to implement by others would be a significant challenge. That is why it is less painful to simply write the algo code with the assumption that the data will not be long - term persistent because of the algo disconnects / stops. So James's suggestion makes sense in that context. You would have to estimate / calculate important algo state stuff upon each initiation / restart of the algo.

What I do in my algo's is making sure it can rerun without hardcoded numbers. This is possible but it will ask for some custom coding wtehre you have to zip and unzip data when you use Fetcher. I think making the algo robust to restarts is the goal as it will make your life a lot easier.

I read some posts that the reason to not allow writing to file is for security reasons and I understand it. But An easy solution is to allow users to write files that exists in quantopian only. which means I can only read it but not download it. I think quantopian is awesome but write data to file is a basic feature in my humble opinion.

My suspicion is that they do not want it to be possible for employees to snoop on stored data.

But yes, if they let us provide a security key to store encrypted data I would think that would be a secure solution. The key could live in memory the same way the Robinhood OAuth token seems to.

Of course our code is already stealable, and that's worth more than data. Not really sure the logic tbh. If we could store key value pairs with a maximum length I don't see the issue and that would help with algos that don't simply look at when certain lines cross each other... For example, I'd like to store the results of machine learning so it gets better with each iteration.

Turning the algorithm from stateless to stateful is a huge step. I really suggest to implement the persistent storage layer. Quantopian can put a lot of restrictions for security reasons. But I am happy if there is one.