Q3, Sprint 5 of 6
I am sorry to report that this quarter has not gone nearly as well as we all hoped. There are a number of reasons for this, all of which are being worked on.
Development cadence on – and integration with – the C++ data-processing framework has been unexpectedly slow and problematic.
The main problem is that the C++ framework was not initially designed to be easy to diagnose issues on - i.e. if the application developer sends an invalid command in JSON, the back end will typically assert - by design - which is very difficult to work with without a complete C++ debug setup and intimate knowledge of the C++ codebase – which are exactly what we don’t want to require of the application developers.
A v2 of the C++ architecture should remedy this issue, by essentially never asserting, providing rich error logging as well as rich logging in general, and by making it impossible for there to be JSON conversion errors. It will also be simplified by a factor of x2-x3 since we’ve found that my v1 architecture was overdesigned. This is embarrassing for a number of reasons and not something (obviously) I would repeat today. All in all though, it was directionally quite good.
There are two other issues with the C++ back end - up to now our developers for it were remote - and the other issue is that C++ itself just isn’t a rapid application development language. Modern C++ is much better, but not enough. So I personally am seriously considering if we should move off C++ for everything except for, essentially, performance critical algorithms. C# (.NET Core) is attractive, as are potentially Rust and Go. In the case of Rust, there may not be a reason to use C++ at all, even for the inner loops of performance critical, SIMD code (!). The choice is really just for the framework itself anyway, since in v2 the data processing ‘Nodes’ themselves are intended to be able to be developed in any language (e.g. Python).
I’m super excited to get to the point where the back end data processing development can go just as fast as the front-end application development!