Today I had an alert come across indicating my /var/log was becoming full... I tracked this down to a lucene log file that was becoming rather large. Reading through the log I see this "WARN" repeated over and over
13373876902 [Thread-8] WARN org.wikimedia.lsearch.frontend.HttpMonitor - Thread[Thread-203870,5,main] is waiting for 7195886678 ms on * 13373876902 [Thread-8] WARN org.wikimedia.lsearch.frontend.HttpMonitor - Thread[Thread-284796,5,main] is waiting for 4240451969 ms on bad397 13373876902 [Thread-8] WARN org.wikimedia.lsearch.frontend.HttpMonitor - Thread[Thread-128942,5,main] is waiting for 9614169952 ms on 10.146.48.5:8321 13373876902 [Thread-8] WARN org.wikimedia.lsearch.frontend.HttpMonitor - Thread[Thread-285721,5,main] is waiting for 4240449957 ms on 10.146.48.5 13373876902 [Thread-8] WARN org.wikimedia.lsearch.frontend.HttpMonitor - Thread[Thread-285716,5,main] is waiting for 4240449964 ms on 10.146.48.5:40105
What troubleshooting steps can I take as im not even sure what is causing the threads to wait.
Thanks.