<div dir="ltr">Dear Stefan,<div><br></div><div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">1) Apache Parquet, Lucene, … are all written in Java. We don't do Java.<br></blockquote><div>Parquet is a file format that allows you to cut up JSON data structures (including lists and other hierarchical data)</div><div>into chunks of similar data (i.e., like a column database, but more liberal on what the structures are - this may be</div><div>useful for the cases where you currently have strings and hashes).</div><div>There's a C++ library at <a href="https://github.com/Parquet/parquet-cpp" target="_blank">https://github.com/Parquet/parquet-cpp</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
2) Most of the recent developments (e.g. Apache Parquet) are aimed at massively distributed databases. CWB4 should be useable on a standard desktop computer or laptop.<br></blockquote><div>Some massively distributed databases are built for a similar use case as CQP (i.e., large data with few updates) and, in the case of Parquet, contain a storage layer that can be used independently.</div><div>I wasn't trying to convince anyone to build CQP4 around a distributed architecture based on Zookeeper and HDFS. Maybe that's for CQP5 ;-)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Doesn't "columnar database" just mean that the database community finally realized that the CWB3 data model wasn't such a bad idea after all?<br></blockquote><div>Columnar databases have been around since the 70s, and they've been one of the active research topics in databases in the last 10 years. And part of what you write really reminds me of an old talk from Peter Boncz (of MonetDB fame) about their sadly-not-open-source X100 project.</div><div><br></div><div>Here's an overview on Vectorwise (the product that X100 grew into), which cites lots of DB research papers that are all concerned about</div><div>pulling compressed columns (or row/column chunks) into memory:</div><div><a href="http://sites.computer.org/debull/A12mar/vectorwise.pdf" target="_blank">http://sites.computer.org/debull/A12mar/vectorwise.pdf</a><br></div><div>especially PFOR/PFOR-delta ("patched frame of reference" - a more CPU-efficient way of doing what byte coding is meant to do)<br></div><div><a href="http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1617427" target="_blank">http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1617427</a><br></div><div><br></div><div>since these have been around for some time, I would expect at least</div><div>some open-source libraries to contain implementations of them. But</div><div>maybe it's exotic enough that it's limited to high-tech DB startups.</div><div><br></div><div>On the other hand, the SETS dependency query engine from Filip Ginter's group 'just' runs on bitsets (a bit like CQP's IDList is a sparse set representation).</div><div><a href="https://github.com/fginter/dep_search" target="_blank">https://github.com/fginter/dep_search</a><br></div><div><a href="http://anthology.aclweb.org/N/N15/N15-3011.pdf" target="_blank">http://anthology.aclweb.org/N/N15/N15-3011.pdf</a><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I think we don't want to do this in relational algebra.<br></blockquote><div>At some point if you have graph structures in your data you will get</div><div>some kind of joins, and even with the current RE-over-annotated-tokens,</div><div>one could actually do some kind of query optimization if one were careful.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I'd suggest to do it in Python if the Python interpreter weren't quite as bad as it is. Slower than R, seriously?<br></blockquote><div>Slower than R would be weird. See here for a semi-scientific comparison</div><div><a href="http://www.johndcook.com/blog/2014/06/20/benchmarking-c-python-r-etc/" target="_blank">http://www.johndcook.com/blog/2014/06/20/benchmarking-c-python-r-etc/</a> </div><div><br></div><div>For Python specifically, there is Cython (a kind of C/Python hybrid language</div><div>that allows you to get 90% of the speed of C with 90% of the convenience</div><div>of Python), as well as JIT compilers such as PyPy and Numba.</div><div><br></div><div>For my own work, I usually write some Cython code when Python and</div><div>the CQP query language take too long for some simple thing.</div><div>I have mostly come to the point where I can build PyPI-installable</div><div>packages using Cython code now; a friendlier approach would be</div><div>to use the conda packaging system that you get with the Anaconda</div><div>Python distribution for Windos/Mac/Linux (which also has Numba).</div><div><br></div><div>(Another alternative that I thought of would be Javascript. However</div><div>tempting that is, I think that both the steep learning curve for V8/NodeJS</div><div>FFI and the fact that Javascript isn't that widely taught make that less</div><div>attractive).</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Implementing the query engine as a separate library with documented API is also a clear requirement for CWB4.<br></blockquote><div>yay.</div><div><br></div><div>Another thing that came to my mind: currently it's hard/impossible to ship a single</div><div>CQP corpus in a (zipped/tarred) directory and have it run out of the box. It would</div><div>be quite neat if there was a solution for that.</div><div><br></div><div>Best wishes,</div><div>Yannick</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<br>
Thanks again for your thoughts and let's continue the discussion!<br>
<span><font color="#888888">Stefan<br>
</font></span><div><div><br>
_______________________________________________<br>
CWB mailing list<br>
<a href="mailto:CWB@sslmit.unibo.it" target="_blank">CWB@sslmit.unibo.it</a><br>
<a href="http://devel.sslmit.unibo.it/mailman/listinfo/cwb" rel="noreferrer" target="_blank">http://devel.sslmit.unibo.it/mailman/listinfo/cwb</a><br>
</div></div></blockquote></div><br></div></div></div>