I previously wrote about my cryptocurrency exchange library called Cryptofeed.
As a library it requires the end user to implement software to make use of the data (store it, feed it to an algorithm, etc). I had my own
unreleased software that used cryptofeed to store data directly to Arctic, a database library built on top of MongoDB.
Over a period of several months, many users of cryptofeed asked me questions about how one could utilize cryptofeed to store data to Arctic,
so I decided to clean up my code and release it on Github - and so Cryptostore was born!
Cryptostore abstracts away a lot of work involved with collecting data from cryptofeed - users can configure everything in a yaml file. Currently data can be collected/aggregated via kafka or redis, and from there the data can be stored into cloud storage (Google Cloud Storage or AWS S3, in parquet format), local storage, or an Arctic store. User supplied plugins are supported as well - an example is provided that backfills missing trade data. This allows anyone to quickly and easily provide extra functionality without needing to either understand Cryptostore or make significant modifications to its core. I've been running this code on my own servers for just over a year, so this is the end result of a lot of testing and fixing! Its able to handle full book data from multiple exchanges with ease and its reasonably resilient and able to recover from the hiccups many exchanges experience periodically. If you encounter any issues or have any improvements or questions feel free to open an issue or a PR!
Cryptostore abstracts away a lot of work involved with collecting data from cryptofeed - users can configure everything in a yaml file. Currently data can be collected/aggregated via kafka or redis, and from there the data can be stored into cloud storage (Google Cloud Storage or AWS S3, in parquet format), local storage, or an Arctic store. User supplied plugins are supported as well - an example is provided that backfills missing trade data. This allows anyone to quickly and easily provide extra functionality without needing to either understand Cryptostore or make significant modifications to its core. I've been running this code on my own servers for just over a year, so this is the end result of a lot of testing and fixing! Its able to handle full book data from multiple exchanges with ease and its reasonably resilient and able to recover from the hiccups many exchanges experience periodically. If you encounter any issues or have any improvements or questions feel free to open an issue or a PR!
Python
Cryptocurrency
Arctic