Topic on Talk:Quarry

I gave a query to quarry about all the articles and their categories - it's logical it will need some time. The query status is running, but when I press explain, it says "Error: Hmm... Is the SQL actually running?! ". Which is it then?

Which query? Could you give a link?

It says running but here you go : https://quarry.wmflabs.org/query/22865

I might been doing this wrong however.

From the logs, the query most likely generated too many results for quarry to store them. You hit phab:T172086.

Where can I see the logs? How could I run a query that big?

MariaDB [enwiki_p]> SELECT COUNT(1) FROM categorylinks INNER JOIN page p1 ON (p1.page_id = cl_from AND (p1.page_namespace = 0)) LEFT JOIN page p2 ON (p2.page_namespace = '14' AND (p2.page_title = cl_to)) LEFT JOIN page_props ON ((pp_page=p2.page_id) AND pp_propname = 'hiddencat') WHERE   (pp_propname IS NULL);
+----------+
| COUNT(1) |
+----------+
| 28917820 |
+----------+
1 row in set (9 min 7.29 sec)

MariaDB [enwiki_p]> Select COUNT(1) from page;
+----------+
| COUNT(1) |
+----------+
| 43592182 |
+----------+
1 row in set (2 min 1.31 sec)

Quarry will not and should not be able to store result sets this large. As for the logs, they are as files on the runner instances, and I do not know of an easy way to expose them.

Thanks for the insights. Do you know how could obtain such large data from that query?

Also, the above shell is it quarry? Cause it doesn't look so.

For large amounts of offline data, you can use https://dumps.wikimedia.org/enwiki/. The above shell is directly querying the Wiki Replicas servers on Toolforge, but quarry will produce the same results if the same query is run at the same time, since it connects to the exact same servers.

I have in fact downloaded these sql tables from the dumps and I am writing them in a database (which is a slow process) but I have a fear that this query won't work either.

Hello again.

I am thinking of using pagination and limiting the results. How many do you think I could get away with? 1 million records? More ? Less?

500k is doable.

Even though 500k was doable with Limit 5000000 (I don't have the link because I queried from the same link) giving Limit 500k,500k in https://quarry.wmflabs.org/query/22970 is not working. Damn is this frustrating

edit : first 500k results : https://quarry.wmflabs.org/query/22969

edit2: nvm