Hi<div>Thanks for the reply, Paul.</div><div> </div><div>I was asking mostly out of interest, as my task was to introduces NLP students to the tool, so I do not know the type of queries that they will be running, but I guess that they will not be optimised for performance (students...)</div>
<div><br></div><div>I didn't think about cold vs saved queries. </div><div>I'll test the difference and mention it on the next exercise.</div><div><br></div><div>Thanks</div><div>NT</div><div><br><br><div class="gmail_quote">
On Wed, Nov 21, 2012 at 10:27 AM, Paul Meurer <span dir="ltr"><<a href="mailto:paul.meurer@uni.no" target="_blank">paul.meurer@uni.no</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Hi Nikola,<div><br><div><blockquote type="cite">I understand. <div><div class="im">I just wanted to know if there were any available solutions..<div><br></div></div><div>As for better hardware, I am currently running cqp on pretty high-end hardware (xeon, 12gb ram) and, exept investing in SSDs in order to speed it up some more, there is not much room for improvement…</div>
</div></blockquote><div><br></div><div>I am running the German DEWAC corpus (1.8G tokens, 1.6G words) on a different system (Korpuskel) with an architecture similar to CWB. My experience is that a fast disk is what really matters. (Of course, lots of RAM and a fast CPU are also an advantage. I wouldn't call 12GB RAM and no SSD/RAID exactly high end ;-) We have a RAID60 system with 22 disks, and that is even much faster than SSDs. The result is that cold queries (those that have to fetch a lot of index pages into RAM) have acceptable speed. When you rerun the query, it is the CPU speed that matters alone (give you have enough RAM), and for most queries that is orders of magnitude faster than the cold query. The same should be true for CWB.</div>
<div><br></div><div>Multithreading could help for some queries (typically those of the scanning type, such as searching for two adjacent equal words) if you had divided your corpus into parts (or had copies of it) that were located on independent disks, such that the threads wouldn't have to compete for disk access. (That's my guess, I haven't tested it (yet).)</div>
<div class="im"><br><blockquote type="cite">
<div>But still, I am aware that querying 1.5G words in 3-4 minutes is allready pretty cool and I thank you for making this the tool</div></blockquote><div><br></div></div><div>Query response times should depend a lot on the type of your query. What are you typically querying?</div>
<div><br></div><div>Best wishes,</div><div>Paul</div><div><div class="h5"><br><blockquote type="cite"><div><div><br></div><div>regards</div><div>NT </div><div> </div><div>
<div><br><br><div class="gmail_quote">On Sun, Nov 11, 2012 at 10:16 PM, Hardie, Andrew <span dir="ltr"><<a href="mailto:a.hardie@lancaster.ac.uk" target="_blank">a.hardie@lancaster.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1f497d">Better hardware?<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1f497d">I know this sounds glib, but re-engineering CWB to make it multithreaded or to use ancillary database indexes would be a huge undertaking. Throwing better hardware
at the problem will almost certainly cost you less than the programmer time to rewrite large chunks of CWB from the ground up.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1f497d">best<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1f497d">Andrew.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a> [mailto:<a href="mailto:cwb-bounces@sslmit.unibo.it" target="_blank">cwb-bounces@sslmit.unibo.it</a>]
<b>On Behalf Of </b>Nikola Tulechki<br>
<b>Sent:</b> 11 November 2012 07:58<br>
<b>To:</b> Open source development of the Corpus WorkBench<br>
<b>Subject:</b> [CWB] cqp and very large corpora<u></u><u></u></span></p>
</div><div><div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Hello<u></u><u></u></p>
<div><p class="MsoNormal"><u></u> <u></u></p>
</div>
<div><p class="MsoNormal">I am using cqp with the *WAC corpora (1.5G words) and, while not prohibiting, response times are still in the minutes range. <u></u><u></u></p>
</div>
<div><p class="MsoNormal">Are there any ways to further speed-up the tool?<u></u><u></u></p>
</div>
<div><p class="MsoNormal">Multithreading? Indexes stored in RAM, in DB? <u></u><u></u></p>
</div>
<div><p class="MsoNormal"><u></u> <u></u></p>
</div>
<div><p class="MsoNormal">Thanks<u></u><u></u></p>
</div>
<div><p class="MsoNormal">NT<u></u><u></u></p>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
CWB mailing list<br>
<a href="mailto:CWB@sslmit.unibo.it" target="_blank">CWB@sslmit.unibo.it</a><br>
<a href="http://devel.sslmit.unibo.it/mailman/listinfo/cwb" target="_blank">http://devel.sslmit.unibo.it/mailman/listinfo/cwb</a><br>
<br></blockquote></div><br></div></div></div>
_______________________________________________<br>CWB mailing list<br><a href="mailto:CWB@sslmit.unibo.it" target="_blank">CWB@sslmit.unibo.it</a><br><a href="http://devel.sslmit.unibo.it/mailman/listinfo/cwb" target="_blank">http://devel.sslmit.unibo.it/mailman/listinfo/cwb</a><br>
</blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><br><br><div>
<span style="border-collapse:separate;border-spacing:0px">-- </span></div><div><span style="border-collapse:separate;border-spacing:0px"><span style="font-family:monospace">Paul Meurer</span><br style="font-family:monospace">
<span style="font-family:monospace">Uni Computing</span><br style="font-family:monospace"><span style="font-family:monospace">Høyteknologisenteret</span><br style="font-family:monospace"><span style="font-family:monospace">Thormøhlensgate 55</span><br style="font-family:monospace">
<span style="font-family:monospace">N-5008 Bergen</span></span></div></font></span></div></blockquote></div><br></div>